I would imagine that this analogy stemmed from the fact that a fence normally serves as a border between two territories. With the patriots being one territory, and the loyalists being the other, one who wouldn't claim either would be left sitting on the metaphoric fence.