The men appeared in "four dusty cars" and there are entirely a few of them. because they are described as catching out "in ones and twos" to defy Atticus.
Atticus addresses one of the Walter Cunningham, so we can know that he is one of the leaders of the mob because he addresses Atticus.