Large Language Models LLMs and Natural Language Processing (NLP)

The paper “Nested Learning: The Illusion of Deep Learning Architectures” provides a substantially richer picture than the popular summaries around HOPE and catastrophic forgetting. It reframes much of modern deep learning—architectures and optimisers—as a hierarchy of associative memory systems operating at different time-scales. 

1. Nested Learning as a unifying paradigm

The central claim is that a neural network plus its optimiser is not a single learning loop, but a set of nested optimisation problems, each with its own “context flow” and update frequency. Components are ordered by how often they update:

• Fast components: e.g. attention states or internal memories updated at every token or step.

• Medium components: e.g. optimiser states (momentum, Adam’s first and second moments).

• Slow components: e.g. model weights updated only during pre-training or occasionally in continual learning. 

This hierarchy mirrors the brain’s multi-time-scale processing (working memory, synaptic consolidation, systems consolidation), and the authors argue that most current LLMs are effectively “anterograde amnesiacs”: they can process the immediate context and rely on a frozen long-term memory, but cannot reliably integrate new experience into that long-term store once pre-training ends. 

Nested Learning (NL) therefore adds a new design axis besides depth/width: the number and structure of learning levels. Deeper in this sense means more nested optimisation processes, not just more layers.

Optimisers reinterpreted as associative memory

A key insight is that classical optimisers (SGD, momentum, Adam) can be viewed as associative memories that compress gradients over time. 

When you add momentum, you introduce a second memory:

• The main weights Wt: slow memory of task structure.

• The momentum term Mt: a fast, key-less memory storing recent gradients.

This is already a two-level nested optimiser. Adam extends this with separate memories for the first and second moments of gradients, creating parallel memories with the same frequency but different roles. 

From this standpoint, they propose three directions for “deeper” optimisers:

1. More expressive associations – treat momentum as a proper key–value memory, mapping gradient features (or preconditioned variants) to update directions, instead of a single aggregated vector.

2. More expressive objectives – replace simple dot-product style internal losses with ℓ² regression objectives that better manage capacity and avoid pure Hebbian behaviour, giving more stable memorisation of gradient histories.

3. More expressive memory modules – replace the linear momentum accumulator with an MLP or other non-linear module (their “Deep Momentum GD”), turning the optimiser state into a small neural network that learns how to integrate gradients. 

This connects directly to current interest in learned optimisers and test-time training: optimisation itself is an in-context learning process over gradients.

Continuum Memory System and HOPE

The Continuum Memory System (CMS) generalises the classic “short-term vs long-term” distinction. Instead of two buckets, it proposes a chain of memory modules MLP1…MLPn, each updated at its own frequency f. 

Given a stream of inputs, each module:

• Processes tokens at every step for inference.

• Updates its parameters only every X steps, where X is chosen so that slower modules integrate more stable patterns over longer horizons.

Transformers, under this lens, are just the trivial case with k = 1: a single persistent MLP plus a working-memory attention block. Nested Learning instead suggests multiple persistent MLPs, each responsible for compressing its own slice of temporal context into its parameters.

HOPE (“self-referential learning module with continuum memory”) combines:

• A Titans-style self-modifying sequence model (weights that learn to update themselves based on context).

• The Continuum Memory System, so that different parts of the model adapt at different speeds. 

Empirically, HOPE is evaluated against strong baselines (Transformer++, RetNet, DeltaNet, Titans, Samba, etc.) at 340M, 760M, and 1.3B parameters on language modelling (WikiText, LAMBADA) and common-sense reasoning benchmarks (PIQA, HellaSwag, WinoGrande, ARC-E/C, Social IQa, BoolQ). HOPE consistently:

• Matches or slightly trails the very best recurrent hybrids in pure perplexity at small scale.

• Becomes competitive or superior at larger scale, slightly outperforming Titans on the aggregate accuracy metric at 1.3B parameters (57.23 vs 56.82 average). 

Additional experiments (described mainly in the appendix) indicate:

• Stronger long-context reasoning, thanks to the dynamic projection layers and deep memory.

• Improved continual learning, with reduced catastrophic forgetting when trained incrementally.

• Clear evidence of in-context learning emergence shaped by the nested optimisation design, rather than appearing as a mysteriously emergent property of scale alone. 

Implications for AI agents and system design

From this more detailed reading, several research-level implications emerge:

1. Architectural depth vs learning depth

   The paper argues that adding more layers does not necessarily increase computational depth or algorithmic expressivity; what matters is how many learning levels exist and how they interact. For agent systems, this suggests new architectures where planner, memory, and optimiser are explicitly distinct nested learners.

2. Unifying RNNs, attention, and optimisers

   Linear attention, state-space models, and fast-weight programmes can all be written as different associative memories with different internal objectives. This gives a common language for comparing new recurrent architectures (e.g. RWKV, Mamba, Titans, HOPE) and may guide principled hybrids rather than ad-hoc designs. 

3. Designing learning-aware agents

   If HOPE-style modules are plugged into agents, one can endow different subsystems with different “plasticities”:

   • Fast components for session-level adaptation (tasks, users, tools).

   • Medium components for skill-level refinement across tasks.

   • Slow components that encode robust, safety-critical knowledge with strict update controls.

4. Safety and interpretability considerations

   Self-modifying weights plus continuum memories are powerful but harder to reason about. The paper frames them as mathematically “white-box” in terms of optimisation structure, yet real-world deployment will require additional monitoring tools to track how each memory level changes over time.

5. Shift in research questions

   Instead of only asking, “How big should the model be?” or “How long a context can it handle?”, NL suggests questions such as:

   • How many learning levels should this system have?

   • Which signals feed each level’s context flow?

   • How do we constrain or regularise slower memories to remain stable yet adaptable?

These directions position Nested Learning not as a single architecture, but as a design framework for future AI systems that must be continuously learning, yet controllable—exactly the regime where agentic workflows, test-time training, and long-running tools are heading.