Because occasionally they find themselves displeased with them. Sometimes mortals would outright destroy one of their statues, curse their name, make love in their temple (medusa). This angered them severely and mortals were often punished for their crimes.