In-Context Learning: Learning at Inference Time

What is in-context learning?

In-context learning (ICL) is the ability of a large language model to pick up a new task from examples or instructions given directly in the prompt, without any of the model’s weights being updated. You show the model a handful of input-output pairs, or just describe what you want, and it adapts its behavior on the fly, purely by conditioning on that text.

This capability of the model is highly non-trivial if you think about it: a model trained once, frozen, and deployed can still “learn” a brand-new task at inference time. It does not need backpropagation, gradient step, or any parameter changes ; instead, it just uses tokens flowing through a forward pass.

Why people use it

A few reasons ICL is so appealing in practice:

No training required: no labeled dataset, no training loop, no GPU hours.
One model, many tasks: the same frozen weights can classify support tickets in one prompt and write SQL in the next.
Fast: adaptation happens in the time it takes to run a forward pass, not the time a training run would take.

That combination is essentialy why prompting became the default way most people interact with LLMs.

How ICL emerged and evolved

2019–2020: zero-shot, one-shot, few-shot (GPT-2 → GPT-3)

The earliest systematic demonstration of this behavior came from GPT-2 and GPT-3. Rather than fine-tuning on each downstream task, GPT-3 was evaluated by conditioning on a natural language description of the task plus zero, one, or a few demonstrations, then asking it to complete the pattern.

In the zero-shot setting, the model gets only a task instruction, no examples.
In the one-shot setting, the model gets exactly one demonstration.
In the few-shot setting, the model gets a handful of demonstrations (often 10–100).

The striking finding, reported by Brown et al. (2020), was that few-shot performance scales with model size: larger models get disproportionately better at using in-context examples, even though nothing about the training objective explicitly rewards this. It’s often cited as one of the first clear examples of an emergent capability: a capability that appears and improves as a side effect of scale rather than something anyone trained for directly.

Larger models make increasingly efficient use of in-context examples: accuracy on a synthetic reasoning task as a function of examples in context, for three model sizes. Source: Brown et al., 2020.

2021–2022: chain-of-thought and demonstration sensitivity

As ICL became a standard interface, two important refinements emerged:

Chain-of-thought (CoT) prompting: instead of showing the model input / output pairs directly, you show it input → reasoning steps → output. Wei et al. (2022) showed that for multi-step reasoning tasks, prompting the model to produce intermediate reasoning steps substantially improves performance, and that this benefit itself only appears reliably at sufficient model scale.
Demonstration selection matters a lot: researchers found that ICL is surprisingly sensitive to which examples you choose and what order you put them in, sometimes more sensitive to formatting details than to whether the demonstration labels are even correct. This sensitivity became an active area of study in its own right, since it complicates any theory of ICL as “clean” learning from evidence.

After: instruction tuning, RLHF, and long context

Two further developments changed how ICL is used in practice, even though they don’t change the underlying mechanism:

Instruction tuning and RLHF: once models were fine-tuned on instructions and human preference data, zero-shot prompting got significantly better. You often don’t need demonstrations at all anymore, just a clear instruction, because the model has already been trained to follow instructions directly.
Long context windows: as context windows grew from thousands to hundreds of thousands of tokens, researchers started exploring “many-shot” ICL: prompting with hundreds or even thousands of demonstrations. Agarwal et al. (2024) found that performance keeps improving well past the few-shot regime for many tasks, and even proposed ways to generate synthetic demonstrations (“Reinforced ICL”) once human-labeled examples run out.

The open problem: we don’t really know why ICL works

Despite ICL being a standard way people interact with LLMs, we still don’t have a full mechanistic understanding of what the model is doing internally, or why it sometimes works so well and sometimes fails terribly.

Different explanations compete for the answer, and none of them stands out as the clear winner. Each explanation captures something real, but also runs into a wall at some point.

1. ICL as Bayesian inference

The Bayesian account, developed by Xie et al. (2021), frames ICL as a byproduct of how language models are trained.

The idea: pretraining exposes a model to an enormous mixture of different tasks, styles, and topics. To predict the next token well across such a heterogeneous corpus, the model implicitly has to learn a distribution over latent concepts (different “modes” of text it might currently be generating). A prompt’s examples then act as evidence that lets the model infer which latent concept (or task) is currently active, and it uses that inferred concept to generate the completion. In other words, in-context learning looks like the model performing Bayesian posterior inference over “what task is this?”

Pretraining teaches the conditional

\[p(\text{next token} \mid \text{context}, \theta)\]

marginalized over some prior $p(\theta)$, where $\theta$ is a latent “concept” or task variable. At inference time, given a prompt of examples $x_1, y_1, \dots, x_n, y_n$, the model implicitly computes the posterior

\[p(\theta \mid x_1, y_1, \dots, x_n, y_n)\]

and uses it (marginalizing again) to predict the label for a new query. As more examples accumulate, this posterior concentrates on the concept actually being demonstrated, which is why performance tends to improve with more correct and consistent examples.

Pretraining documents are generated from a shared latent concept; at inference time, the demonstrations act as evidence that lets the model infer which concept is active and complete the pattern accordingly. Source: Xie et al., 2021.

Limitations: this theory is a functional description. It will tell you what the computation is equivalent to, not what circuit inside the network actually implements it. It also struggles to explain how models learn genuinely novel input-output mappings that couldn’t have been anticipated as a “concept” seen during pretraining (e.g., an arbitrary made-up symbol substitution task).

2. ICL as implicit gradient descent

The second account, developed by von Oswald et al. (2023), argues that when a transformer processes few-shot demonstrationsa in it’s forward pass, the process is mathematically similar to running an optimization algorithm, using the demonstrations themselves as training data.

The idea: pretraining teaches the network’s weights to implement, inside a single forward pass, something equivalent to a learning algorithm that runs at inference time. The demonstrations become the training set for this “inner” learning process, and the query becomes the test point.

Consider ordinary gradient descent (GD) on a simple linear regression loss. Initialize $w_0 = 0$, with loss

\[L(w) = \sum_i (y_i - w x_i)^2,\]

and take one GD step with learning rate $\eta$:

\[w_1 = w_0 + \eta \sum_i (y_i - w_0 x_i)\, x_i = \eta \sum_i y_i x_i .\]

The prediction on a new query $x_{k+1}$ after this one step is:

\[\hat{y}_{k+1} = w_1 \cdot x_{k+1} = \eta \sum_i y_i (x_i \cdot x_{k+1}).\]

Notice that the prediction is a weighted sum of past outputs $y_i$, where each weight is the dot product (i.e. the similarity) between the past input $x_i$ and the new query $x_{k+1}$. In other words, we predict a mixture of previously-seen outputs, weighted by how similar their inputs were to the current query.

Now compare this to a linear attention layer. Let’s set:

\[K = \{x_i\}, \quad V = \{y_i\}, \quad Q = x_{k+1}.\]

The output of linear attention is:

\[\sum_i (\text{query} \cdot \text{key}_i)\, \text{value}_i = \sum_i (x_{k+1} \cdot x_i)\, y_i .\]

This is exactly the same formula as one step of gradient descent above (up to the constant $\eta$). Von Oswald et al. (2023) show this equivalence formally, and demonstrate empirically that transformers trained on regression tasks converge to weights that closely match this gradient-descent construction, effectively becoming what they call mesa-optimizers: optimizers that emerge inside another optimization process.

A transformer can construct weights via its forward pass that match those produced by an explicit gradient descent step on the in-context demonstrations, and trained transformers empirically track the gradient-descent loss curve almost exactly. Source: von Oswald et al., 2023.

Limitations: this equivalence is rigorously proven for linear attention on simple regression-style tasks. Real transformers use nonlinear attention (because of softmax), and while some analogous results exist for more general settings, they’re harder to prove and don’t yet cover the messy, high-dimensional tasks language models are actually asked to do in practice.

3. ICL as induction heads (pattern-matching circuits)

The third thread comes from mechanistic interpretability (reverse-engineering the actual circuits inside a trained transformer) via the work of Olsson et al. (2022) at Anthropic on induction heads.

The idea: an induction head is a specific two-head attention circuit pattern. Given a context like ...A B ... A, when the model encounters a repeat of token A, an induction head attends back to the token that followed the first occurrence of A, and increases the probability of predicting B next. This is pure pattern completion: last time we saw this token, what came after it?”

This behavior is implemented by (at least) two attention heads working together:

A “previous token” head, which (in an earlier layer) copies information about the token that preceded each position into that position’s representation.
The induction head itself (in a later layer), which uses that copied information to find matching prior contexts and copy forward whatever followed them.

An induction head at work: on seeing a repeat of a token (here the prefix of "struction"), the head attends back to what followed its earlier occurrence and copies it forward, boosting the logit for that continuation. Source: Olsson et al., 2022.

This finding wasn’t derived from theory, but it was discovered empirically through circuit analysis: people examined attention patterns directly, and ablated (deleting or disabling) specific heads to see which ones causally drive in-context learning performance.

During training, there is often a sharp phase transition where induction heads suddenly emerge (a rapid jump instead of a gradual improvement). This jump correlates closely with a simultaneous boost in the model’s in-context learning ability, suggesting induction heads aren’t just correlated with ICL but may be causally responsible for a substantial part of it.

In-context learning score over training for attention-only models with one, two, and three layers. Models with more than one layer show a sharp "phase change," absent in the one-layer model, right where induction heads emerge. Source: Olsson et al., 2022.

Limitations: most of the direct evidence for induction heads comes from small models, which is true of circuit-level analysis in general ; but it’s much harder to trace circuits in models with billions of parameters and many more layers and heads. Induction heads are probably a good explanation for one component of ICL (pattern matching / completion) but likely not the whole story for more abstract or semantic forms of in-context adaptation.

Putting the three accounts together

None of these three explanations is “wrong” so much as incomplete on its own:

A reasonable current view is that pretraining shapes the weights into something that functions like a Bayesian task-identifier (the “why”), implemented via mechanisms that resemble in-context optimization steps (the “how, mathematically”), and at least partly built out of concrete circuits like induction heads (the “how, physically”). These are different zoom levels on the same phenomenon rather than mutually exclusive theories. But nobody has yet unified them into one account that holds up from toy models all the way to today’s frontier LLMs.

That gap between having a plausible story and having a mechanistic proof at scale is exactly why in-context learning remains one of the more active open problems in understanding how large language models actually work.

References

Brown, T. et al. (2020). Language Models are Few-Shot Learners. NeurIPS.
Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS.
Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An Explanation of In-context Learning as Implicit Bayesian Inference. arXiv:2111.02080.
von Oswald, J. et al. (2023). Transformers Learn In-Context by Gradient Descent. ICML.
Olsson, C. et al. (2022). In-context Learning and Induction Heads. Transformer Circuits Thread, Anthropic.
Agarwal, R. et al. (2024). Many-Shot In-Context Learning. NeurIPS.