Sect - a term used to describe a relatively small religious organization that has split off from a denomination due to small religious differences, i.e. Jehovah's witness.
Denomination (which a sect splits off from) is a well-established religious orgaization in which a large part of the population are members, i.e. Judaism, Catholicism.