Dirt, as in: inorganic particles, are usually not very harmful, except if they are poisonous, such as asbestos.
Food that is for some reason hard to digest, can also cause diarrhea.
So, the biggest problem insanitary food handling are organic particles, that is bacteria, viruses and parasites. For example, unsanitary food can contain tapeworm eggs, which can develop into tapeworms, dangerous parasites which cause malnutrition and starvation.