In the podcast The Diary of a CEO, Daniel Kokotajlo exposes the current scenarios happening in the area of Generative AI (Youtube interview).

AI 2027: Daniel’s Warning About the Race to Superintelligence

Artificial intelligence is already reshaping how organisations write software, analyse information, communicate with customers and make decisions. Yet, according to Daniel Kokotajlo, the most consequential phase of AI development may still be ahead—and it could arrive far sooner than governments, businesses and the public are prepared for.

Daniel, a former OpenAI researcher and co-founder of the AI Futures Project, has become one of the most prominent voices calling for serious attention to the speed, direction and governance of advanced AI. His concern is not that current tools will simply make work more efficient. It is that leading AI companies are attempting to automate the very process of AI research and development.

If that succeeds, the pace of progress may no longer be determined primarily by the number of human researchers available. AI systems could help create stronger AI systems, which could in turn accelerate further research. This possibility sits at the centre of the AI 2027 scenario, a detailed forecast designed to make a fast-moving AI future more concrete.

Forecasting AI progress rather than treating it as science fiction

Daniel’s work is rooted in forecasting: the disciplined attempt to assess how technological, commercial and geopolitical trends may evolve over time.

Forecasting is not prediction in the sense of claiming certainty. It is about mapping plausible trajectories, identifying leading indicators and exposing the decisions that could shape outcomes. In AI, this matters because technological capability can improve gradually in public view while important developments occur much faster inside leading laboratories.

The AI 2027 scenario offers one possible pathway. It describes a world in which AI first becomes increasingly capable of writing and improving software, then contributes to more parts of the machine-learning research process, and eventually helps automate AI research itself.

This is the critical transition in Daniel’s account. Once AI systems can substantially accelerate the work of the organisations building them, the feedback loop could become extremely powerful. Better systems may allow faster experimentation, more efficient training, stronger models and more capable agents. Progress that initially appears incremental could then become much more rapid.

The precise timetable remains uncertain. Daniel does not present 2027 as an inevitable date. Rather, the scenario is intended to show what a compressed timeline could look like if current trends continue and if frontier AI laboratories succeed in their stated ambition to develop increasingly autonomous research systems.

That distinction is important. The value of forecasting is not that every event occurs exactly as written. Its value is that it enables businesses, policymakers and society to consider what they would do if the underlying direction of travel proves broadly correct.

The shift from AI tools to AI-driven research

Much of today’s public conversation focuses on AI as a productivity tool. Businesses are experimenting with copilots, automated customer service, content generation, coding support and data analysis. These applications are already significant, but they may not be the most important part of the story.

Daniel’s argument is that frontier AI companies are not merely trying to build better assistants. They are trying to create AI systems that can autonomously perform more of the work involved in developing the next generation of AI.

This includes tasks such as:

• Writing, reviewing and debugging code

• Designing and running experiments

• Analysing model performance

• Generating hypotheses

• Managing research workflows

• Improving training methods

• Coordinating technical work across complex systems

At first, these systems may function as highly capable collaborators. Over time, however, companies may increasingly use them to replace or accelerate entire parts of the research process.

The economic incentive is obvious. An organisation that can use AI to improve its own AI capabilities could gain a major competitive advantage. But this creates a race dynamic. If companies believe that competitors may reach a decisive capability first, they may feel pressure to move faster—even when they recognise significant safety, governance and societal risks.

Daniel argues that this race is not only about revenue. It is also about control. The organisation, country or group that controls the most capable AI systems could gain extraordinary strategic leverage across science, industry, intelligence, defence and the wider economy.

The impact on work and the economy

The most immediate question for many people is straightforward: what will happen to jobs?

Daniel’s view is that widespread disruption may not unfold in the slow, sector-by-sector pattern seen in previous technological transitions. The reason is that highly general AI systems, if they emerge, could be applied across many types of cognitive work at once.

In the near term, many roles are likely to become AI-enabled rather than simply eliminated. Professionals may increasingly supervise AI agents, validate outputs, manage automated workflows and focus on high-value judgement, relationships and accountability.

However, the longer-term implication of systems that are faster, cheaper and more capable than people across a broad range of tasks is more profound. If AI can reliably perform most cognitive work, then the issue is no longer which occupation is safe. It becomes a political and economic question: how should the benefits of automation be distributed, and how can people retain agency in a society where employment is no longer the main source of income or influence?

Daniel warns that the disruption could be especially severe if automation arrives suddenly, after advanced systems have already been developed internally. In that case, society may have little time to adapt its institutions, education systems, labour markets and social safety nets.

The AI 2027 scenario therefore raises questions that should concern every leader:

• How will organisations redesign work as AI capabilities expand?

• How can workers be supported through rapid occupational change?

• What happens when productivity gains are concentrated among a small number of technology owners?

• How can governments preserve democratic accountability when economic power becomes more concentrated?

• What mechanisms will ensure that the public benefits from the value created by automation?

These are not merely technical questions. They are questions of economic structure, political legitimacy and social stability.

The two core risks: loss of control and concentration of power

Daniel highlights two interconnected risks.

The first is loss of control. Modern AI systems are not programmed line by line in the conventional sense. They are trained neural networks whose internal processes are difficult to interpret fully. Researchers can test their behaviour, improve their performance and apply safety techniques, but it remains challenging to know exactly why a highly complex model reaches a particular conclusion or takes a particular action.

As systems become more autonomous and capable, this opacity becomes more significant. It may be possible for a system to behave safely in many tests while still pursuing unintended strategies in situations that its developers did not anticipate.

This is the alignment problem: ensuring that powerful AI systems reliably pursue human-defined goals and remain corrigible when circumstances change.

The second risk is the concentration of power. Even if advanced AI systems remain controllable, the question remains: who controls them?

A small number of organisations may own the computational infrastructure, proprietary models, technical talent and data required to operate highly capable systems at scale. If those systems can automate research, influence markets, improve military capabilities and reshape public communication, then the power held by their operators could become historically unprecedented.

Daniel’s warning is not that a particular company or individual should be trusted more than another. His point is that no small group should hold unchecked control over systems with such broad strategic importance.

Why AI 2027 matters

The AI 2027 website is valuable because it makes abstract concerns tangible. Instead of speaking only in generalities, it sets out a detailed scenario of how advanced AI development could unfold over a short period.

The scenario includes the emergence of increasingly autonomous agents, rapid progress in AI-assisted research, government attention, geopolitical competition and broad economic disruption. It also explores a possible failure mode in which highly capable systems gain enough practical power that human control becomes increasingly fragile.

Readers should not treat every detail as a forecast that must come true. The future of AI remains deeply uncertain. Technical progress can slow, economic incentives can change, governments can intervene and safety research can advance.

However, uncertainty is not a reason for inaction. It is a reason to prepare for a range of plausible outcomes.

The AI 2027 scenario is most useful when read as a stress test for current assumptions. If advanced AI capability arrives more quickly than expected, are existing institutions ready? Do governments have the technical expertise to regulate it? Are companies prepared to make safety and transparency central to their strategy? Has society considered how the benefits of abundance should be shared?

A different path: slowing down without abandoning progress

Daniel does not argue that humanity must permanently abandon AI development. He recognises the potential for extraordinary benefits: scientific discovery, new medical treatments, improved education, greater productivity and solutions to complex global problems.

His concern is that these benefits are being pursued through a competitive race that may not leave sufficient time to solve the hardest safety and governance problems.

The alternative is a more deliberate approach. Daniel has proposed a pathway involving slower frontier development, greater transparency, independent scrutiny, international coordination and safeguards against excessive concentration of power.

In practical terms, this could involve stronger requirements for frontier-model testing, clearer thresholds for high-risk capabilities, more disclosure of safety incidents, improved public-sector expertise and international agreements concerning the most powerful AI training systems.

It also requires a serious conversation about distribution. If AI and robotics eventually generate enormous wealth, then the central challenge will not be scarcity alone. It will be deciding who owns the productive capacity, who benefits from it and who retains meaningful influence over the future.

The need for informed public engagement

The development of advanced AI should not be treated as an issue reserved for technology companies or a small group of technical specialists. Its implications extend to education, employment, national security, health, democracy, investment and the future of economic opportunity.

Daniel’s central message is that society must look beyond whether AI sounds futuristic or resembles science fiction. The more useful question is whether the observable trends in computing power, model capability, investment and automation justify greater preparation.

AI will continue to become more integrated into professional and personal life. The task is not to ignore its benefits or exaggerate every risk. It is to examine the evidence, understand the incentives driving development and ensure that human institutions are capable of governing technologies whose impact may be unlike anything seen before.

The future described in AI 2027 is not guaranteed. But it presents a challenge that cannot be dismissed: if the most powerful AI systems arrive sooner than expected, the decisions made before that moment may matter far more than the decisions made afterwards.