It became an issue because the other countries wouldn't accept them and he couldn't commit a mass exodus since the others wouldn't take them. That's why he decided to place them in ghettos and start with his final solution plan that was responsible for the holocaust.