I don't know if this is right.... but there was a lot of country roads (dirt roads near farms) and when people (usually rich people) would ride on those roads they would spray up dust. And that dust would get on the farmers crops. Because of that it was thought of arrogance and rudeness.
Another reason is that there wasn't much in the way of roads like there is today. So, it was difficult for people driving cars to even get anywhere. And since the car was newly invented it had a lot of technical problems so it would always break and make it even more difficult for people who are driving it.
Hope this helps you!:)