AI RESEARCH
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
arXiv CS.LG
•
ArXi:2605.06609v1 Announce Type: new Transformers have nstrated remarkable in-context learning (ICL) capabilities. The strong ICL performance of transformers is commonly believed to arise from their ability to implicitly execute certain algorithms on the context, thereby enhancing prediction and generation. In this work, we investigate how transformers with softmax attention perform in-context learning on linear classification data.