AI RESEARCH
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
arXiv CS.LG
•
ArXi:2605.05176v1 Announce Type: new Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its nstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting.