I used to struggle with that. Think of the other person as a child who doesn't know any better and doesn't know what's good for him/her. Picture the other person as lost and confused, then maybe you'll just laugh or leave him/her alone and do you. Unless someone puts their hands on you, there's no need to act out aggressively.