Department of Mathematics - Seminar on Applied Mathematics - In-Context Learning in Scientific Computing

Transformer-based foundation models, pre-trained on large datasets spanning a wide range of tasks, have shown remarkable adaptability to diverse downstream applications—even in low-data regimes. A particularly striking capability is in-context learning (ICL): when given a prompt containing a few examples from a new task alongside a query, these models can produce accurate predictions without any parameter updates. This emergent behavior is often viewed as a paradigm shift for transformers, yet its theoretical foundations remain only partially understood. In this talk, I will present recent theoretical progress toward understanding ICL in scientific computing. I will focus on understanding how transformer architectures can implicitly perform task adaptation in three representative problem classes: learning solution operators of PDEs, dynamical system prediction and generative modeling.