A small outlier could affect the whole data set because it could reduce the mean, or make the average of the whole data set smaller.
An example would be the data set could be 1, 8, 9, 10
To find the mean you would add everything together then divide it by the number of numbers you just added together. The mean of this set would be 28/4 which equals 7. Now, if the outlier were -8 instead of 1, then the mean would be much different.
~Hope this helped!~