The prophecy foretold that Polyphemus would be defeated by a man called Odysseus, so obviously he was expecting someone huge and strong to fight with him. However, when Odysseus defeated the cyclops using his brains only, he was really surprised because he overestimated his power. "Brawn versus brain" refers to Polyphemus fighting Odysseus - the latter man used his wits to trick the huge, strong monster and thus defeated him.