There are so many factors that caused their decline, but I would say the main one would be the presidency of John Adams. John Adams made some major mistakes such as enacting the Alien and Sedition Acts, which basically took away the freedom of speech. This for sure made many people upset. Since John Adams was connected to the Federalist party, their reputation would tank. Soon, events like Alexander Hamilton's death, the Louisiana Purchase, and Hartford Convention made them die out.