Interesting problem. When computers first came out (when dinosaurs ruled the planet), we had to deal with problems that were many brackets deep. So this has practical applications.
Start at the very right and keep on removing brackets and collecting like terms.
5a - {3b - 5[a - (b - 4a)]} I just changed the sign on the right bracket.
5a - {3b - 5[a - b + 4a]}
5a - {3b - 5[5a - b]} Now deal with []. You are still on the right.
5a - {3b - 25a + 5b}
5a - {8b - 25a} Finally remove {}
5a - 8b + 25a
30a - 8b
If you can solve another like this, you are well on your way to being good in computer science.