Society usually values controling one's emotions and does not like people causing trouble.
That's why physical aggresion in perceived negativelly in society.
The fact that someone is resorting to violence means that he wasn't smart enough to find any other way to hurt the other person (or was not smart enough to know that he should look for more sublte way:)