This line is referring to the fact that time takes beauty away from everything. Unfair in this context is the act of taking away beauty -- instead of being fair (beautiful), that thing has become "unfair" (losing its beauty). Even things that 'excel' in beauty will become unfair in time.