Not true. For all diseases caused by pathogens (even viruses), there is a minimum infectious dose needed to cause disease.
It does vary between different pathogens & between the condition of the host (immunocompromised vs. healthy), but if the minimum infectious dose is not met- then the body is typically able to snuff out the intruders before they can do any real damage.
Hope that helps!