The answer is: brillant.
"Votre frère est un homme brillant."
WHY: Because the other options are not grammatically correct. "Brillante" is the feminine form of the adjective, but since we are talking about a male subject, the "frère" (brother), that would not be correct. Secondly, the third option, "brillants" is the plural form of the adjective, but since we are talking of only ONE person, it is not correct either. This only leaves us with the male, singular form of the adjective "brillant" as correct.