The three groups were ethnic groups, and each had a republic (similar to US' state today) in the former Yugoslavia:
The Bosnians
The Serbs
The Croats
the three ethnic groups have very similar languages that can be claimed to be the same language but differ by the religion: the Bosnians are mostly Muslims, the Serbs Christian Orthodox and the Croats are mostly Catholic.
Today each of these nations has a separate, independent country.