Factor by grouping
4x^5 - x^3 - 12x^2 + 3
(4x^5 - x^3) + (-12x^2 + 3)
x^3(4x^2 - 1) + (-12x^2 + 3)
x^3(4x^2 - 1) - 3(4x^2 - 1)
(x^3 - 3)(4x^2 - 1)
(4x^2 - 1)(x^3 - 3)
(2x + 1)(2x - 1)(x^3 - 3) ... use the difference of squares rule here
So the original expression completely factors to (2x+1)(2x-1)(x^3-3)