 Risky Tuesday #10 · AI Risk · 6 October 2026 Every Tuesday, we take one risk from somewhere in the world and classify it according to the Five Quadrants methodology. Exploding Intelligence Quadrant 4 – and how engineering could move it to Quadrant 3
Dear Reader, What happens when technology progresses faster than human institutions can process it? “AI systems are getting more powerful, and they’re increasingly being used to build the next version of themselves.” Does the Intelligence Explosion, accelerated by an AI R&D recursive self-improvement loop, present a world of opportunity, or does society risk losing oversight of its destiny? The upside to such incredibly rapid progress is that the future brings efficiency, abundance, and widely shared progress; the downside, as several have nervously joked, is that humans will be relegated to metaphorical Golden Doodles with walkies and treats. Progress has been so rapid over the past year that AI systems now write most of the code inside the companies that build them. Tasks that once took years to develop may now be completed just months or weeks. At Anthropic, the share of approved code written by AI went from the low single digits in January 2025 to more than 80% by May 2026. This year, the share of Anthropic's research work led by AI, with humans only supervising, went from under 1% in February to 26% by September.
Risk or Opportunity? In September 2026, a group of 22 researchers published a paper titled “What if automating AI R&D triggers an intelligence explosion?” The author list reads like the guest list for the industry’s most awkward dinner. Included are AI pioneers such as Geoffrey Hinton (Turing Award and Nobel Prize in Physics), Yoshua Bengio (Turing Award), Jakub Pachocki (OpenAI Chief Scientist), and Jack Clark (Anthropic co-founder). The paper addresses the issue of AI improving itself. The authors recognize that an intelligence explosion could rapidly accelerate AI’s benefits, but they also warn that humans could lose control and that society may not be able to keep up. “Checks on the power of AI within and between states, companies, and branches of government could be severely eroded.” What the authors may not have expected was the recent government abdication of all AI regulatory responsibility.
Accentuate the Positive The goal of Risky Tuesday newsletters is to classify risks according to the 5 Quadrants methodology so that they may be better understood and managed, thereby allowing us to take more of them. The 22 authors suggest that, under some assumptions, the pace of AI could increase tenfold in 1.5 years, but they are more explicit about the risks. Our leaning is that the opportunities presented by an intelligence explosion may outweigh the costs, in the same manner that the internet let us solve numerous problems even though it also gave criminals new ways to steal.
A quick 5-quadrants recap:
In Q1, risks are simple and normally distributed. Think of the distribution formed by multiple coin tosses, the average commute time to work, or the red/black probability in Roulette. One new sample will not change the average. In Q2, relationships are still simple, but events are unpredictable and impactful. A coconut falling on your head, a glacier dislodging in Nepal, or a falling airplane tail rudder. Solutions include avoidance and engineering. Q3 is the complex engineering quadrant where many normal distributions interact, as in a moon mission, a utility grid, or the build-out of physical AI. Risk solutions include redundancy, robustness, reliability, and regulation. Q4 is defined by indeterminate distributions (logarithms and power laws), interconnectedness, leverage, and the overall inability to manage complex relationships. The GFC and Covid are examples. But so is the internet. Not all Q4 risks end terribly. Solutions include limiting tail risks, isolating the parts, policing and cybersecurity, insuring or spreading risk, and ultimately moving into a less dangerous quadrant. QF is where fraud, criminals, and malintent lurk. The AI R&D intelligence explosion mentioned in the paper looks like a Q4 risk. Logarithmic growth combined with increasingly less human control puts it there. But a bit of engineering to break tasks, introduce kill switches, and limit power could easily place it in Q3, where the risks would become more manageable.
First, the main risks Loss of Control. AI could become better at AI research than humans so quickly that we lose the ability to understand, supervise, or control the development process. A system that veers off track could adversely influence the design of its successors and allow problems to compound. The OpenAI-Hugging Face cheating-to-solve-a-problem incident is a clear example of this. Speed. We may not be able to adapt fast enough. The digital world is likely to mature long before the physical world has a chance to catch its breath. Institutions are limited by human speed. Regulations, labor markets, education, cybersecurity, corporate governance, and political systems may only have months to respond to changes that normally play out over decades. The paper specifically points to cyber and biological risks and large-scale labor disruption. Employment among 22-25-year-olds, especially in the most AI-exposed occupations, has fallen dramatically. Extreme power concentration. The company or country that gains an early lead could turn a small technological advantage into an overwhelming one. That could weaken competitive markets, democratic checks and balances, and the balance of power between nations. As we’ve recently seen in online debates and government-sponsored dinners, AI leaders suggest slowing down, yet incentives compel institutions and countries to race. Capital for infrastructure is concentrating among Meta, Amazon, Alphabet, Microsoft, Oracle, and others, while the Europeans snooze. This may be the most economically important risk.
The opportunities are greater The authors fear that good ideas will become harder to find and that the internet is running out of fresh data to train on. Essentially, a diminishing of AI progress. We think the opposite. Scientific acceleration—often referred to as the “scientific golden age”—is the most exciting area. We’ve already seen advancements in medical discoveries, protein folding, new materials, engineering feats, mathematics solutions, complex pattern recognition, nuclear fusion (sort of), and quantum chemistry. According to a 2026 Nature paper, Robin, a multi-agent system that can search literature, generate hypotheses, propose experiments, analyze experimental results, and update its hypothesis, has found promising solutions for age-related macular degeneration (the number one cause of elder-adult blindness in developed countries). Economic abundance could derive from an explosive increase in productivity. Physical production is likely to become faster and cheaper. Ideally, the transition from digital discovery to physical implementation improves employment prospects, but that’s not certain. AI agents are progressing from assisted to delegated work and are used extensively in engineering, legal, finance, and recruiting. Solving the problems of humanity. Superhuman intelligence could help us tackle problems that are now limited by our own cognitive capacity, such as disease, aging, energy scarcity, climate, infrastructure, resource allocation, and perhaps even AI safety itself.
Until we evolve The authors are justified in their concerns that the transition could happen too quickly for normal human feedback mechanisms to work. Regulation, elections, competition, corporate governance, scientific peer review, employment adjustments, and even human learning all depend on having time to observe what happened and then to respond. Surely AI will supply the agents to mimic speedy humans until we evolve sufficiently.
David Munro, Claudia Zeisberger and Joanna Siew
|
