Farming was hard because of the rocky soil, the long and very harsh winters and the very short growing season. The soil wasn't good for growing crops, especially near the ocean. The farmers would move the rocks out of the way before planting.
Also, the early and long lasing winters killed many crops quickly. Due to this hardship the farmers in New England colonies only farmed enough for their family, themselves and their animals.