AI RESEARCH
Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods
arXiv CS.LG
•
ArXi:2506.10959v3 Announce Type: replace While in-context learning (ICL) has achieved remarkable success in natural language and vision domains, its theoretical understanding-particularly in the context of structured geometric data-remains unexplored. This paper initiates a theoretical study of ICL for regression of H\"older functions on manifolds. We establish a novel connection between the attention mechanism and classical kernel methods, nstrating that transformers effectively perform kernel-based prediction at a new query through its interaction with the prompt.