We need to find two numbers that multiply to 24 (last coefficient) and add to 10 (middle coefficient). Through trial and error, the two values are 6 and 4
6 + 4 = 10
6*4 = 24
So we can break up the 10ab into 6ab+4ab and then use factor by grouping
a^2 + 10ab + 24b^2
a^2 + 6ab + 4ab + 24b^2
(a^2+6ab) + (4ab+24b^2)
a(a+6b) + 4b(a+6b)
(a+4b)(a+6b)
Therefore, the original expression factors completely to (a+4b)(a+6b)