They were angry at the use of the new machines.
Luddites were enraged that they have spent years learning their craft and developing skills in weaving, sewing, etc. The new machines could do this quicker and with less mistakes and were beginning to replace people who spent all their live to learn this trade. So they destroyed the machines and burned factories.
Hope this helped. Have a great day!