Because back then the south and north had different political views. In the north African Americans were free slaves meaning people weren’t allowed to own slaves in the North. In the south, it was a slaved owned party, meaning every african american were slaves. I believe they were hesitant to move away from the south because they believed they would receive the same treatment.
I hope this helps / makes sense :)