Cadmium is an element that usually forms a Cd2+ ion just like Zn2+.
Silver commonly forms the univalent, Ag+ ion, so it could not directly replace Zn2+ in ores and you will have to read up on this, most heavy metal mining firms have silver, lead, zinc operations; these are very common mixtures of ores. So, the cadmium is more likely to be the contaminant in zinc than in silver.