AI RESEARCH
Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
arXiv CS.LG
•
ArXi:2506.02132v5 Announce Type: replace-cross Large transformer-based language models dominate modern NLP, yet our understanding of how they encode linguistic information relies primarily on studies of early models like BERT and GPT-2. We systematically probe 25 models from BERT Base to Qwen2.5-7B focusing on two linguistic properties: lexical identity and inflectional features across 6 diverse languages. We find a consistent pattern: inflectional features are linearly decodable throughout the model, while lexical identity is prominent early but increasingly weakens with depth.