It is the thoughts that are impure basically making us view and perceive things as impure. because on an elemental level everything is pure and reflects the perfection of the Supreme being. Our thoughts are the seat source where evil, impurity, hatred, and all things perceived bad are generated, even the perception itself.
Thoughts generate words, words qualify into actions and deeds which again generate thoughts of reaction, hence if thoughts to begin with are pure then the question itself doesnt arise. Hence it is said to be pure by ‘manasa-vacha-karmana’ in order to become close to the Supreme which itself is nothing but pure thought itself.