Isn't the biggest pain thinking about how you caused pain to someone you care about?

I hate breaking up with people, though I've only done it a couple of times I always feel so guilty about how much they are hurting because of it.