Symbols don't have to have any connection with the thing they represent: we just agree that certain things will represent those other things.
So if we don't know about what they represent; we have nothing to use. If we didn't know that green light is for "go"; there would be nothing about "green" that could help us decode its meaning to commuters, would would have to guess.