The general term for an environmental factor that causes a population to decrease is a limiting factor.
Some examples are emigration and mortality, a lack of resources, natural disasters happen which makes people scared to to live there so the people flee, and habitat loss.