Answer:
It's because the party that's not in office is usually more angry about what's currently going on in politics, and they want to vote to change that.
For example, the Democrats gained a bunch of seats during the midterms, because they don't like what Trump is doing in office right now, and they wanted to vote to have more representation. The same thing happened when Obama was in office from 2008-2016. The Republicans gained more seats, because they didn't approve of Obama and wanted change.