An individual's self-esteem can be affected by genetics as well as environmental factors, and poor self-esteem often begins in childhood. A person who experiences an abusive childhood or who didn't get enough approval from authority figures, feels his or her self-worth was based on achievements or feels excluded will often have poor self-esteem.