The witches are androgynous, first of all. They appear to be women, yet wear beards. Macbeth and Banquo are, if anything, confused. Banquo is not afraid at all. He doesn't take the weird sisters seriously, and there's no evidence of his feeling any fear.
Macbeth, however, starts, or flinches, recoils, according to Banquo. This is significant. Banquo asks:
Good sir, why do you start and seem to fear Things that do sound so fair? Hope this helped :)