 Risky Tuesday #8 · AI Risk, Part II · 22 September 2026 Every Tuesday, we take one risk from somewhere in the world and classify it according to the Five Quadrants methodology. When Does the Human Cease to Be in Control?
Dear Reader Last week, we asked what happens if AI works. This week, we want to ask something more uncomfortable: at what point are we no longer in control of AI DEVELOPMENTS? We draw again on the interview with our AI expert, Professor Theos Evgeniou from INSEAD. This is not a discussion about machines going rogue. There is, as Theos reminded us, “always a non-zero probability of a rogue AI.” But he worries that spending too much time on the extreme scenarios can distract us from risks that are already much closer. As he put it, “Let’s not overdo it and miss the specific issues because we’re focusing on the abstract ones.”
Question 1: When does a human cease to be in control? At what point does keeping a human “in the loop” become largely cosmetic or decorative? Have we passed that point, or are we still in control? Theos immediately challenged an assumption embedded in the question: that having a human in the loop necessarily makes the system safer. “Having a human in the loop may … make it more dangerous sometimes.” Think about autonomous driving. If a machine drives more safely than a person, insisting that a human remains ready to intervene does not necessarily improve the outcome. Theos positioned the trade-off rather starkly: “Do you want something more dangerous, but you have somebody to blame, or … something less dangerous, and then it becomes more fuzzy?” He also highlighted an alternative, what one can call “human on appeal”, related to one of his recent articles: “Sometimes one can get the best of both worlds having a human involved mainly when another human, or an AI, raises a flag about a questionable AI decision”, in addition to some very hard or high-risk decisions, of course. It’s both a corrective and a continuously improving process. Monitoring and appeal processes may therefore deserve much more emphasis. Over time, the human role may increasingly shift toward validating AI outputs in higher-risk applications, monitoring AI systems, reporting incidents, and managing appeals. Control, safety, and accountability are not necessarily the same thing. A human can provide accountability without providing much control. Equally, removing the human may improve performance while making it harder to assign responsibility when things go wrong. And that may be one of the more difficult AI risks facing companies. If an autonomous system makes a damaging decision, who owns it? The model provider? The company that deployed it? The developer? The person who approved its use? The data provider? “It’s a difficult problem of attribution of responsibility,” Theos said. “I remember we were debating this for some time a few years back at the OECD network of experts on AI, and at some point, of course, it became a legal question beyond my understanding.” Certainly a comment that will give insurance company boards some food for thought.
Question 2: Will we still have a choice? We then nudged the question further. Even if we decide that humans should remain in control, will companies and countries actually retain that choice? Theos’s answer moved quickly from technology to competition: “That’s maybe more of a competition, and even a geopolitical question than a technological question.” Imagine two competitors. One requires human approval before its AI systems can act. The other allows its systems to make and execute decisions autonomously. If removing the human makes the second company faster, cheaper, or more effective, how long can the first company maintain its additional control? The same problem becomes much more consequential between countries. “If person A elects to become more efficient by removing the human or by building better drones, say for defence, what does person B do? Do we have a choice there?” That is an uncomfortable dynamic because the decision to surrender some human control may no longer be made in isolation. Your preferred level of control will partly depend on what everyone else is prepared to give up. And that takes us back to last week’s discussion. AI adoption can create a hockey-stick effect between those moving quickly and those falling behind. The pressure to remain competitive may itself accelerate the removal of human constraints.
Question 3: So where should the human stay? This is where Theos’s experience at Tremau, a company he founded three years ago, becomes useful. Tremau deals with content moderation at an enormous scale. It would be impossible for people to review every item. So the system does not begin by asking how to keep a human involved in every decision. It asks which decisions actually need one. The AI classifier assigns a probability. Above or below certain thresholds, the system can act automatically and make decisions independently. The uncertain cases in between (let’s call it the grey zone) go to a person: a human moderator for a final decision. “The thing in between … goes to a human for review.” Those thresholds are not fixed. They are continuously adjusted against actual outcomes: complaints, appeals, false positives, regulatory exposure, reputational risk, and the cost of human review. “You have to continuously tune your policies and probabilities.” Interestingly, he highlights that the simple choice of these probabilities can even become a significant political decision. “I encourage everyone to watch again the announcement of Zuckerberg about changes in Meta’s content moderation policies back in January 2025, days after Trump assumed office. Resetting these probabilities to become more ‘relaxed in terms of permitting content online’ was one of the main changes. One can think of why and also ask who should set these thresholds, particularly for AI with potentially systemic implications like political ones. It is about AI governance more broadly. Systemic decisions may require ‘we the people’ to somehow set the thresholds, which, in a sense, is what happened after the US elections.” The starting point is revealing: “Given the volume of cases, the default is that everything is reviewed by AI first.” In other words, the human is no longer the default decision-maker. They’re the exception. And they are also there for appeals and mistakes, which the technology of his company Tremau enables. That sounds alarming until you consider what it actually means. The objective is to invoke the human when human judgement adds value, rather than simply to maintain relevance. Or, in Theos’s rather less diplomatic formulation: “You better put a human in the loop when you know that AI will do silly things, not in general necessarily. And more so, put a human at the right time and place.”
So which Quadrant are we in? This is where the Five Quadrants analysis becomes interesting. Our first instinct would not be to put all of this into Q4. The fact that an AI system acts without a human does not automatically make the risk systemic, unknowable, or fat-tailed. Tremau is almost the opposite example. Exposures are broken down into decisions, probabilities, and thresholds. Outcomes are observed. Thresholds are adjusted. And human intervention is called for only in borderline cases. That simplifies the risk to one that can be modelled and managed. It would put us into Quadrant 2 or 3. But now change the perspective. Allow autonomous systems to transact across financial markets, operate critical infrastructure, or make military decisions. Connect thousands of those systems to one another and add competitive pressure that encourages companies or countries to remove safeguards because others already have. Now the dependencies multiply and complexity increases. Decisions happen at machine speed. One system’s output becomes another system’s input, and the consequences become harder to predict. Most important, the ability of a human to intervene if and when something begins to go wrong may become largely theoretical. That starts to look much more like Q4. So perhaps “human in the loop” is not the best test. The Five Quadrants question is not simply, “Is there a human involved?” It is, “What can the machine do, how connected is it to everything else, how quickly can the consequences propagate, and can we still intervene when we need to?” That seems to me to be the real control question.
Now I need you Where is the human still in the loop in your organisation? In appeal? More importantly, why? Because the person genuinely improves the decision? Because regulation requires it? Because customers expect it? Because the system becomes better over time? Or because everyone feels slightly better knowing there is still a human sitting somewhere between the machine and the consequences?
Next Tuesday, we continue our conversation with Theos and follow the AI risk further down the stack: models, chips, compute and the enormous capital being committed to today’s infrastructure. Claudia and Dave NOTE: In the spirit of the topic, AI wrote the first draft. The humans supervised, rephrased, and tightened the verbiage. The interview was held in person.
|