Helium is not that common on earth, it does not react with other elements, and scientists didn't know that it should exist and so they weren't looking for it on earth. The element that created the lines they found in the sun's spectrum in 1868 was believed to be a new element and was named helium (from the Greek ''Helios'' meaning sun) and when they finally knew it existed they found it on earth.
Hope this helps!