This might be a perception thing and that the facial hair is growing at the same rate as the hair on your head. Since you're constantly grooming your facial hair, even 1mm growth will affect the appearance whereas a 1mm growth on the hair on your head makes no difference because your hair is already long.