What happens when millions of different kinds of people try to use your system?
A new research framework organised by Harvard and MIT researchers proposes a radical change in how artificial intelligence systems and digital products are tested: simulate millions of different users before exposing the product to real ones. The ambition is enormous. The scientific question is whether synthetic behaviour can ever become reliable enough to influence real-world decisions.
There is a significant change taking place in artificial intelligence evaluation, and it has little to do with making models larger.
Until now, most AI benchmarking has essentially asked the machine a question: Can you complete this task?
MatrAIx asks something considerably more difficult:
What happens when millions of different kinds of people try to use the system?
A 93-author research consortium organised by Xiaomin Li of Harvard University and Yuexing Hao of the Massachusetts Institute of Technology published “MatrAIx: Simulating the World with 8.3 Billion Persona Agents” on arXiv.
The collaboration spans dozens of academic institutions, including Stanford, Berkeley, Princeton, Oxford, Carnegie Mellon and others.
At the centre of the project is Persona 8B, an extraordinarily large synthetic population containing approximately 8.3 billion persona records, described through a common schema of 1,290 categorical behavioural and demographic dimensions.
The objective is not merely to generate fictional characters.
The researchers are attempting to build an infrastructure in which AI agents can behave as heterogeneous simulated users, interact with surveys, chatbots, websites and native applications, and generate evidence about how different categories of people might react to a digital product.
In other words, MatrAIx attempts to put an artificial population between software development and real-world deployment.
From benchmark testing to population simulation
Traditional AI benchmarks typically evaluate capability.
A coding agent receives a programming problem. A language model answers a question. A browser agent attempts to purchase an item. The resulting score generally says something about whether the system succeeded.
What it does not necessarily tell us is whether the same system works equally well for different users.
An experienced software engineer may expect an AI coding assistant to make substantial architectural decisions independently. A junior developer may require explanations, smaller modifications and explicit confirmation before changes are applied.
Two users can therefore encounter the same AI system and experience completely different levels of trust, friction and usefulness.
MatrAIx attempts to model precisely this variation.
Its architecture has three principal components: the Persona 8B population, the MatrAIx Playground execution environment, and a library of application-specific evaluation tasks. The current research release describes 1,010 task specifications across more than 25 domains, including commerce, software, finance and healthcare.
Of these tasks, 621 are survey-based, 371 concern chatbot interaction, 12 operate through web environments and six involve native applications. Importantly, the authors explicitly state that 1,010 is the number of available specifications, not the number of studies already executed.
That distinction matters.
MatrAIx is primarily an evaluation infrastructure, not evidence that more than one thousand product experiments have already been scientifically validated.
What exactly is an 8.3-billion-persona population?
The number 8.3 billion is impressive, but the interesting part of MatrAIx lies in how the personas are represented.
Every record belongs to a schema containing 1,290 dimensions divided into five broad families:
- 238 background attributes;
- 210 psychological attributes;
- 331 capability attributes;
- 124 behaviour and interaction attributes;
- 387 lifestyle attributes.
These dimensions cover variables such as age, geography, language, education, professional expertise, attitudes to risk, values, technology adoption, interests, health, cultural preferences and behavioural tendencies.
Crucially, the synthetic population is not produced by sampling all 1,290 attributes independently. Doing so would generate statistically absurd people. One could easily create, for example, a persona whose age conflicts with its education history, whose language proficiency contradicts its primary language, or whose career seniority is incompatible with employment history.
MatrAIx therefore uses a directed acyclic graph — DAG — to encode dependencies between attributes.
Formally, the population model factorises a persona distribution as:
p(x) = ∏ p(xᵢ | xParents(i))
Each attribute is consequently sampled conditionally on relevant predecessor variables. Education level can depend on age; English proficiency can depend on region and primary language; other relationships are governed by source-informed conditional distributions and explicit compatibility constraints.
The framework separates two mechanisms.
The first modifies probabilities. A particular attribute becomes more or less likely given the persona’s existing characteristics.
The second applies hard compatibility rules. An impossible combination can receive probability zero and therefore disappear from the sampling space.
This is an important architectural decision because MatrAIx is not simply generating random character descriptions with an LLM. It is attempting to construct a probabilistically coherent synthetic population in which correlations between characteristics are explicitly represented. That brings the project closer to synthetic population modelling than conventional prompt-based persona generation.
Synthetic personas are only half of the story
MatrAIx also incorporates what the researchers call human-grounded personas.
These records are extracted from six sources: Wikipedia biographies, Amazon review histories, the Stack Overflow Developer Survey, the General Social Survey, PRISM Alignment profiles and 355 consented responses collected through the project’s own persona survey.
The source material is mapped onto the same 1,290-dimensional schema used for synthetic personas.
Unsupported attributes are left empty rather than inferred automatically, while direct identifiers such as names and contact information are removed from the released records.
This hybrid architecture is conceptually important.
Pure synthetic populations inherit modelling assumptions.
Pure human-derived datasets inherit sampling bias, missing data and privacy constraints.
MatrAIx attempts to combine the scalability of the former with some of the empirical grounding of the latter.
The paper reports a public coreset containing 599,847 human-grounded records and 400,000 synthetic records, or 999,847 records in total.
There is a small release-version discrepancy worth recording. The current Hugging Face search metadata describes the public release as a 1,000,000-row coreset, with 600,000 human-grounded and 400,000 synthetic records, whereas the arXiv v1 paper reports 999,847 records.
For interpretation of the experiments, the figures reported in the paper should therefore be treated as the reference numbers for that version of the study.
Then the personas become agents
A persona record by itself does nothing.
MatrAIx becomes more interesting when that record is paired with a large language model and transformed into a persona agent.
The framework can then insert that simulated user into four kinds of environment.
In a survey environment, thousands of agents might be asked whether they would still buy a product following a price increase.
In a chatbot environment, researchers could observe whether different simulated users continue interacting with an AI assistant after it produces an incorrect answer.
In a web environment, agents can browse pages, compare products and attempt transactions.
And in the application environment, computer-use agents can operate desktop or mobile software, including sandboxed Linux environments and remote macOS or iOS systems.
Each simulation is represented as an independent trial connecting a persona, task, agent interface, model and random seed. The system preserves interaction trajectories and delegates evaluation to task-specific verifiers.
This creates something increasingly important in AI engineering: reproducible synthetic user testing.
Instead of saying, “our chatbot seems difficult for inexperienced customers”, developers could theoretically construct a particular cohort, run it against version A, deploy version B, and repeat precisely the same evaluation.
The analogy with software regression testing is obvious.
MatrAIx is effectively trying to introduce behavioural regression testing at population scale.
The 91.5% result requires careful interpretation
The headline experimental result is that persona agents followed their assigned characteristics in 366 of 400 controlled trials — 91.5%.
At first sight, this appears remarkably strong. The experiment covered ten behavioural attributes across all four MatrAIx environments. Positive and negative versions of each behavioural characteristic were tested to determine whether the agent expressed — or correctly suppressed — the assigned trait. But this number requires an important qualification.
The reported 91.5% result corresponds to Claude Opus 4.8 acting as the persona agent. In an additional experiment documented in the appendix, GPT-5.6-sol achieved 317 successful trials out of 400, or 79.2% under the corresponding protocol.
That roughly twelve-percentage-point difference is one of the most technically significant observations in the entire paper. It tells us that a persona is not simply a stable data object.
Observed synthetic behaviour is a function of at least:
persona × underlying model × prompt × environment × task × evaluator.
The underlying LLM possesses its own behavioural prior. If the persona asks the model to behave in a manner strongly opposed to that prior, the model may effectively resist its assigned identity. The researchers provide a particularly revealing example. GPT-5.6-sol struggled when instructed to behave as a deliberately verbose, rambling persona. The paper attributes this to the model’s strong tendency towards concise, structured answers.
This exposes a fundamental challenge for the entire field of synthetic-user simulation.
We may believe we are measuring the preferences of a simulated consumer when, in reality, we are partially measuring the behavioural preferences embedded in the LLM used to simulate that consumer.
There is another methodological caveat
The 91.5% behavioural-adherence result was not directly judged by human evaluators.
The paper states that all 400 verdicts were produced by a Claude Opus 4.8 LLM judge. Human evaluations appear in a separate experiment examining the quality of extracted human-grounded personas.
The authors attempt to audit the automated judge by requiring evidence from interaction trajectories, but the distinction remains important.
The headline result is therefore more precisely described as:
91.5% behavioural adherence according to an audited LLM-based evaluator under the reported Claude Opus 4.8 configuration.
That is considerably more informative than saying simply that “the personas behaved correctly 91.5% of the time”.
In the separate human evaluation, six human raters evaluated a source-matched subset of 100 personas and produced an overall mean extraction-quality score of 4.135 out of 5. The researchers also compared LLM judges against these human assessments.
These experiments strengthen the methodological foundation, but they do not establish that MatrAIx can predict actual human behaviour with 91.5% accuracy.
Those are two completely different claims.
What MatrAIx could change for product development
The immediate commercial application is easy to understand.
Consider an e-commerce business preparing to increase a subscription price. A conventional workflow might involve analytics, surveys, focus groups, A/B testing and eventually exposure to real customers. A MatrAIx-style workflow introduces another stage before this process.
Product teams could specify cohorts based on income, purchasing behaviour, technological competence, risk tolerance or other characteristics, expose those simulated users to the new pricing experience, and inspect how reactions differ between groups. The framework explicitly contains tasks related to purchase intent, price sensitivity, retention, recommendation and support resolution.
The same principle extends well beyond commerce. A bank could inspect whether different simulated customers understand a financial assistant. A software company could test whether novice and expert users discover the same feature. A healthcare application could investigate accessibility problems before beginning controlled human studies. An AI company could stress-test whether particular groups abandon an assistant following hallucinations, latency or poor recovery from failure. For UX engineering, this creates the possibility of moving from a handful of conventional personas — “Sarah, 32, urban professional” — towards thousands of probabilistically generated behavioural configurations.
The difference is not simply scale.
It is the possibility of turning persona design into an executable evaluation system.
But synthetic users are not human users
The authors themselves make this limitation explicit. MatrAIx personas are simulation instruments, not people, and sampled cohorts are not probability samples of real populations. The researchers specifically caution against using them to attribute behaviour to identifiable people or communities, to construct discriminatory targeting systems, or to replace consultation with people affected by consequential decisions. This limitation becomes particularly serious in healthcare, employment, finance and other regulated environments. A model can reproduce an assigned characteristic without accurately predicting how a real human being possessing that characteristic would behave.
These are separate statistical properties.
Persona adherence asks: does the agent behave consistently with its instructions?
Human validity asks: does that behaviour correspond to what actual humans would do?
MatrAIx provides promising evidence for the former. The much harder problem of the latter remains open. Indeed, the paper reports that identical cohorts interacting through different persona-agent models can produce dramatically different aggregate outcomes. In one experiment, the proportion selecting a paid plan ranged from 23.2% to 93.9% depending on which model powered the personas. That is not a minor experimental detail. It is a warning against treating synthetic populations as artificial opinion polls. For now, MatrAIx is better understood as a system for hypothesis generation, stress testing, segmentation analysis and early product evaluation rather than as an oracle for predicting consumer behaviour.
The authors themselves use essentially this interpretation, arguing that consequential claims about real people still require human validation.
The larger significance: AI systems are beginning to test AI systems
The deeper importance of MatrAIx lies beyond its 8.3 billion records. Software engineering has progressively automated unit testing, integration testing, security testing, performance testing and deployment validation. AI introduces a more complicated problem because the quality of an intelligent system frequently depends on interaction with a human.
MatrAIx proposes that some portion of that human interaction can itself become computational.
That produces a striking new architecture:
AI models build the product.
AI agents simulate the users.
AI agents operate the product.
AI evaluators judge the interactions.
Humans increasingly supervise the experimental design rather than execute every test.
The opportunity is substantial, but so is the epistemological risk.
If the model playing the customer, the model serving the customer and the model judging the interaction share similar training distributions — or even the same underlying model family — apparently successful evaluation can become circular. The researchers explicitly warn about this case. When the persona agent and the system under evaluation share a model backbone, positive results may reflect genuine product quality, or they may reflect a model preferring outputs similar to its own. The simulation alone cannot distinguish between those explanations.
This is likely to become one of the central methodological questions in agentic AI evaluation.
Synthetic users can make experimentation dramatically cheaper and faster. But the cheaper simulation becomes, the more important external validation becomes.
MatrAIx therefore represents something more interesting than another enormous dataset.
It represents an emerging layer in the AI engineering stack: synthetic society as testing infrastructure. Before deploying an AI assistant to one million people, simulate one million different users. Before changing a price, simulate the customers who may accept it and those who may leave. Before launching a new interface, send artificial novices, experts, impatient users, cautious users and accessibility-sensitive users through it. Not because those synthetic people can tell us with certainty what humanity will do. They cannot. But because they may reveal questions, edge cases and behavioural failure modes that conventional benchmarks never knew to ask.
And that may ultimately be MatrAIx’s most important contribution: not replacing the human user, but forcing AI evaluation to acknowledge that there was never such a thing as the user in the first place.