There are many answers for this
one can be that Substance X is specific to only certain materials (it's like a puzzle)
another one is that the active site of X does not fit a different substrate
let me explain it to you so you will understand, it is like a lock and key. You see how every lock has its own key? so does this