The correct answer is Japanese immigrants
According to the gentlemen's agreement, the United States would not ban Japanese people from entering the United States, however, Japan would not issue passports to those who would want to go to the US, so the ban on immigration would not come from the United States but rather from Japan who didn't want its citizens emigrating.