Becaus with the lack of a centralized government which would regulate how people of smaller communities would be in relation to each other (in a formal way), the ones who had the most power (in this case these were the manor/castle owners) could decide that they would rent out their land to peasants which would work for them in return for a certain percentage of food.