In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

ArXi:2503.01611v3 Announce Type: replace Instruction following is a critical ability for Large Language Models to perform downstream tasks. The standard approach to instruction tuning has relied on a specific phase of supervised fine-tuning over curated instruction datasets, optionally complemented with an alignment step over human preferences. Recent work has shown the potential of in-context learning (ICL) alternatives to guide base models towards instruction following.