Look if he makes you uncomfortable, get out of the relationship. If you think you both genuinely love each other and he passes in your ideal type category, except for the fact that he's an ass, keep him. Do what your heart thinks is right.
I would be honest with you. You're pretty much giving an excuse for him like he is changing and all and honey if you are feeling you deserve much the probably you really so I feel like you should learn to let go