'Haunted by the past' is a commonplace phrase because it's a commonplace experience. Even if one is not, strictly speaking, 'haunted', the past is perpetually with one in the present, and the longer it grows and the further it recedes the stronger its presence seems to become.
