La solita chiacchiera?
When you dig into the guts of agentic AI, sooner or later you hit the hard mathematics: stochastic processes, decision theory, and the formalism that keeps everything together. This is not la solita chiacchiera, but the real deal—how AI systems learn, decide, and act with mathematical rigour. Let’s start at the beginning: the humble Markov Chain.
Markov Chains: stochastic transitions, niente fronzoli
A Markov Chain is the purest expression of the Markov property: the future depends only on the present, not the past. If
Applications? Weather prediction, linguistic models, genetics. But the limitation is obvious: no azioni, no ricompense. It’s descriptive, not prescriptive. Enter the MDP.
Markov Decision Process (MDP): the control layer
An MDP extends a Markov Chain by adding controllability. Formally, an MDP is a tuple
• S: states,
• A: actions,
•
•
• gamma: discount factor.
The agent’s goal is to find a policy that maximises the expected return:
This is the core of reinforcement learning: from robotics to healthcare, from e-commerce recommender systems to logistics optimisation. Yet, MDPs assume full observability. That’s a luxury real-world AI rarely has.
The Partial Observability Problem
Most real systems are messy. Observations are incomplete, noisy, or deceptive. A self-driving car doesn’t “know” the full state of the road; it only gets sensor readings. A chatbot doesn’t “see” the entire user intent; it infers from partial dialogue.
Classic solutions involve POMDPs (Partially Observable MDPs), where the agent maintains a belief state (a probability distribution over states). But maintaining and updating belief states is computationally heavy, often intractable in high dimensions.
Here’s where Memory-augmentation kicks in.
Memory-Augmented MDP (M-MDP): state + memory
A Memory-augmented Markov Decision Process extends the MDP by incorporating a structured memory variable m_t. The effective decision state becomes \tilde{s}_t = (s_t, m_t).
Formally,
where M is the memory space and f the memory update function:
This converts a partially observable process into an extended Markovian process by embedding the relevant past directly into memory. Think of it as equipping the agent with a “working memory buffer”, a cheap but powerful trick.
Agentic AI systems—AI that acts proactively, plans, and coordinates tasks—cannot survive without memory. An LLM-based agent, for example, needs to recall context across thousands of interactions, remember constraints from earlier steps, and adapt strategies on the fly.
With M-MDPs, we get:
Context retention: agents recall historical signals without blowing up state dimensionality.
Adaptive policies: decisions are conditioned on both present signals and long-horizon memory
Reduced computational burden: compared to POMDP belief-state tracking.
Integration with neural architectures: memory modules (e.g., differentiable memory, RNNs, Transformers) can instantiate m_t, bridging symbolic decision theory and deep learning.
Example: M-MDP in a shopping agent
From Markov Chains to MDPs, from memoryless dynamics to memory-augmented reasoning, the evolution of mathematical frameworks is steering AI towards true agency. The M-MDP is not just a tweak, but a foundational leap—embedding structured memory into decision processes. As agentic AI scales, M-MDPs will be indispensable in bridging short-term stochasticity with long-term adaptive intelligence.
Imagine an AI shopping agent navigating dynamic e-commerce platforms.
In a vanilla MDP, the state is the current page and inventory.
In a POMDP, the agent must maintain a probability distribution over all possible hidden states (e.g., competitor stock levels).
In an M-MDP, the agent augments its state with a memory buffer: previously clicked items, abandoned carts, or inferred preferences. This allows policy optimisation without full Bayesian inference, yet still capturing sequential dependencies.
This is not fantascienza. It is already happening in agentic commerce, where memory-augmented agents reshape product discovery, recommendation pipelines, and semantic SEO strategies.
“Benvenuti nel futuro, dove la matematica incontra l’agency”