At the time of the Trail of Tears, American citizens were willing to overlook the treatment of Native Americans (like the Cherokee) because the land they were moving from would be open to American citizens.
The Trail of Tears was a result of the Indian Removal Act. This removed Native Americans from their homelands in Georgia and forced them to move to reservations in modern day Oklahoma. The territory taken by the US would then be open to American citizens, specifically white male farmers.
Many Americans probably chose not to speak out against the government because they were going to benefit from the removal of these tribes. Along with this, Americans may have been worried about the repercussions (consequences) about speaking out against the government.