I can imagine the following scenario:
People were eating some good crops and threw the seeds next to their settlement, then they saw that this made the plant grow in that place and continued placing the seeds there. Often they could actually see that the plants actually grew from the seeds.