part of the reason Madison opposed the Bank was because of Jefferson and Madison's beliefs that the ability to control commerce did not constitute a right to create a bank. However, it is evident through on of Jefferson's famous quotes that this was not the only reason:
"I sincerely believe that banking establishments are more dangerous than standing armies." -Thomas Jefferson