My boyfriend has a friend that had a baby face. No hair would grow whatsoever. But then his friend joined into the navy and had to shave their face whether they had facial hair or not. Eventually, he started growing hair and now he has a beard.
So summary of my little paragraph: Shave even if you don't have much hair. Continuously shaving results in hair growing back thicker and fuller.