In united states culture, Pulling away from the speaker indicates disinterest in whatever the speaker is telling us.
This happen because our subconscious sent a signal to our body to create a distance between us an the source of discomfort, which make us lean our body further away from the spearker.