We can write -4 on its own precisely because it doesn't mean subtract 4. It means the opposite of four. You can think of a number line as having three parts: a positive direction, a negative direction, and zero. Everything to the right of zero is positive and everything to the left of zero is negative.