The factor most likely to result in a decrease in the size of a specific population is Famine.
Famine can causes shortage of water and food that can vanish a huge population from the earth.
A famine can be a result of inflation, extreme weather, conditions and imbalance, war, and crop failure or government policies. A famine at a certain area can cause starvation, deprivation, high mortality rate and disease wide spread.