Because the novel is set in the 1930’s when racism (I know may still go on today) was still current and very present between whites and colored folk, Crooks color skin was a problem for the other white men. They did not want Crook staying in the bunk house with them, simply because of his color of skin.