Its both. Depending on the type of sentence you're using. If it is describing an action you did with your grandkids then you would use, 'my grandkids and I'. If it is describing an action someone else did for you and your grandkids then it'd be 'my grandkids and me'.