I think too often we think of love as a continued state of good times. I think this song is about the reality of loving someone unconditionally, even when out of time with one another. It's about the challenge that love presents over the course of time- for better or worse, people changing together and apart but choosing each other over and over.
It doesn't have to apply to romantic love alone, but any love between two people that lasts a life time.