Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

ArXi:2605.00607v1 Announce Type: new Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features.