Everything is deadly at a certain amount. Sometimes you accidently incest a small amount of something that's toxic, but it doesn't really do any harm because of how little it is. The body just gets rid of it during the its normal biological processes.
There's a certain amount of the toxin that you have to reach before the body begins to really feel it's effects and begin to get rid of it (vomiting, clamminess, all that jazz).
Of course it depends on the substance in question, some things are way more dangerous in much less amounts than others (e.g. ricin vs. poinsettia flowers).
Hope this helps!