People tend not to believe/accept things they can't see proof of with their own eyes or things that lack scientific evidence.
Continental drift is something that spans hundreds of thousands of years. People don't live that long to be able to see the position of the continents and land changing over time. Also when Wegener proposed this theory, he couldn't explain how continental drift occurred, so lacking scientific evidence, many people did not believe him.