Answer: It is important for Jonas to only have memories for this reason. The community can not let every citizen experience the pain. So, they assigned this job to the Giver. The Giver is the only person that has memories, and the only one who can see colors.
Hope this helps!