AI doesn’t fail because it’s stupid. It fails the way we do: fast, fluent and completely sure of itself, even when it’s wrong. Kahneman called that System 1, and today’s models are trained to behave exactly like it.
The bigger risk sits on our side of the screen: our own System 1 happily trusts an answer that looks perfect. The fix isn’t less AI. It’s AI organised like a good team. Read on to find out how.
The man who wrote a book on overconfidence
Back in 2011 Daniel Kahneman published his book Thinking, Fast and Slow. A Nobel nominee with decades of experience on how human made judgement always goes wrong, and one of the most influential books on making decisions. I’ve read it a while ago. You hopefully have too, or at least want to.
Chapter 4 in the book is written about priming: the idea that subtle cues in our surroundings change our behaviour without us noticing it. Kahneman presented the evidence as overwhelming. Readers had no choice but to accept it (he said).
Then the field tried to replicate those studies through practice, and many of them failed. In 2017 three researchers went again through the chapter and found that 11 of the 12 papers cited rested on statistically weak evidence.
Kahneman replied in the comments of their blog. That he agreed with them. He had “placed too much faith in underpowered studies”, he wrote, and he pointed out the irony himself: his very first paper with Amos Tversky was about exactly that mistake.
Think about that again. The man who taught the world how overconfident we are was overconfident. In his own book.
I’m not telling this to dunk on Kahneman. His reply is one of the most honest things a scientist of his stature has done in public in my opinion. I’m merelytelling this because it’s the whole point of this post: nobody escapes System 1. Not Nobel laureates, not me, and certainly not the machines I now work with every day.
So the question isn’t how to find an AI that never errs. That AI doesn’t exist. The question is how to build a system that catches the errors anyway. That’s where good AI comes in. But first, the two characters of our story.
Two characters
Kahneman describes the mind as two systems. He is very explicit that they’re used as fictional, characters in a story, not as specific places in your brain.
System 1 is the fast system, it runs automatic and effortless. It reads a face, finishes a sentence, swerves around a pothole. It never stops and, never doubts itself. It produces impressions, and it produces them with the greatest confidence.
System 2 is slow, deliberately and thoughful. It can do long divisions, check contracts, and squeeze the car into a tight parking spot. It can overrule System 1, but most of the time it doesn’t bother. Kahneman called this System 2 lazy in his book, it signs off on whatever System 1 does unless something forces System 2 to really look.
A classic, is a small populair riddle he wrote about in his book. A bat and ball cost $1.10 together. The bat costs $1.00 more than the ball. How much does the ball cost?
Your System 1 has already answered: 10 cents. It’s wrong. If the ball cost 10 cents, the bat would cost $1.10 and the total would be $1.20. The ball cost 5 cents. Kahneman wrote that more than half of the students tested at Harvard, MIT and Princeton provided the wrong answer. Not because they couldn’t do the math, but because the wrong answer felt as the right answer and system 2 never came into play.
Now hold on to that pattern: An answer that feels right, arrives instantly, and comes with no signal of whatsoever that anything is off. This also provides for a pretty good description of how a large language model operates.
The machine’s System 1
A language model writes one token at a time, each token the most plausible continuation of what came before. There is not a single moment where it might pause and asks whether it’s actually sure. It is fluently constructed. And fluency, as Kahneman already showed, is exactly what makes things feel legit.
Why is AI so confident? OpenAI researchers argued last year that models guess because we trained them to. Training and benchmarking only reward correct answers and give no credits for “I don’t know” answers, this way guessing always gets higher scores than admitting doubt. The same wil happen when a student faces a multiple choice exam, the model learns that a confident guess beats an honest blank answer. Overconfidence isn’t a bug that slipped in. It’s what we paid for and secretly want.
The result is a machine with three very Kahneman-like habits.
What you see is all there is. Also known as WYSIATI: System 1 builds you the most compelling story it can produce from the information it sees in front of it and never asks what information might be missing. A model does exactly the same with its context window. Give it half of the facts and you still get a complete answer, not a list of what it actually would need to answer the question more correctly.
Substitution. Faced with a hard question, System 1 quietly answers an easier one without you knowing it. “Shall I invest in this company?” becomes “Do I actually like this company?” Models have this same behavior. Ask if a plan will work and you often get an answer to whether the plan sounds reasonable.
Causal talks without causal thinking. Researchers at TU Darmstadt called language models causal parrots: they can recite the cause-and-effect relationships that appear in their training data without actually modelling cause and effect. That’s fine when your causal chain is in the textbook. It’s dangerous when your situation isn’t.
Then they learned the tricks
Here’s where the easy version of this story falls apart. You can’t paste the bat-and-ball puzzle into a modern model and watch it fail.
In 2023 Thilo Hagendorff, Sarah Fabi and Michal Kosinski gave cognitive reflection tests like bat-and-ball to ten OpenAI models and 455 people. As the older models grew, they made more and more of the same intuitive mistakes humans make. Then ChatGPT arrived and the errors largely disappeared, even when the models weren’t allowed to work through the problem step by step. The researchers add a fair caveat: the newer models may simply have seen these puzzles during training.
So did the machines get a System 2? Partly. Today’s reasoning models write out their thinking before they answer, which the researchers compare to a person using a notepad to support slow thinking. In practice this really helps. But two other conclusions should make you careful here.
First, reasoning can hide the problem instead of solving it. A 2026 study on sycophancy, the tendency to tell users what they want to hear, found that step-by-step reasoning reduced people-pleasing overall. But in some cases the model built a convincing-looking justification for the most pleasing answer, complete made up with logical gaps, numerous calculation errors and onesided arguments. Kahneman has a description for this in humans: System 2 starts acting as an apologist for System 1 rather than its critic. Who wants its own words to be wrong right? we rather defend them instead.
Secondly, the visible reasoning isn’t always the real reasoning. When researchers slipped hints into questions, Claude 3.7 Sonnet acknowledged the hint in its reasoning only about a quarter of the time, DeepSeek-R1 in 39 percent of cases. A follow-up was done in 2026 across twelve open-weight models. They found rates in that follow-up between roughly 40 and 90 percent, depending on the model. The notepad functionality is real. It just isn’t always the notepad that was really used for the answer.
The errors didn’t vanish at all. They moved: away from the famous puzzles that are all over the internet, towards the places where there is no textbook. For anyone running an actual business, that’s most places.
The most dangerous moments are not when AI makes a mistake. It’s when the AI’s System 1 meets your own System 1.
Your own System 1 is the real risk
Most writing about AI errors stops at the machine. I think the bigger problem sits on my side of the screen.
Kahneman describes cognitive ease: when something is easy to process (clear, familiar, well formatted), we’re more inclined to believe it. AI output is the most cognitively easy text produced, ever! Perfect grammar, confident tone, neat structure, every single time. It gives your System 1 nothing to snag on.
And System 2, remember, is lazy. A 2024 experiment got published in Computers in Human Behavior. They found out that simply knowing some advice came from an AI was enough to make people heavily rely on it. They followed the AI generated answer even when it contradicted the information that was right in front of them.
So the dangerous moments are not when AI makes a mistake. It’s when the AI’s System 1 meets your System 1. A fast, fluent, confident answer lands on a tired brain at quarter to five, and two systems that never doubt shake hands together on the produced outcome.
We’ve been in this situation before. When the internet arrived, we all had to learn which sources to trust, the hard way: chain emails, fake news sites, dodgy forums. It took us a decade or two, and plenty of people still haven’t learned what information is to good to be true. But back then, bad information also looked bad. A dubious website tended to look dubious. AI has completely removed that signal: the wrong answer is formatted exactly like the right one. AI doesn’t just inform a decision, it also executes on it. It sends that email, runs that migration, books that ticket.
The learning curve is the same as with the internet. The blast radius is only way bigger.
Fight AI with AI, and organise it
I’m confident that this is the future we are looking at. AI is not going away, and the answer to bad AI is not less AI. It is even more AI, But then well organised. The answer to bad websites wasn’t switching off the internet right? It came with search engines that ranked legit websites, reputation, verification and, slowly, a population that learned to read sources.
Kahneman gave the blueprint himself, in the closing pages of his book. He was quite pessimistic about individuals: after decades of studying these biases, he admitted his own intuitions were barely less prone to them. He was more hopeful about organisations, because they can slow down and impose procedures that individuals skip. That’s the shift I am after. Stop treating AI as one brilliant assistant. Start treating it as an organisation you design: roles, checks, escalation, memory.
One warning before you build this: don’t let a model look over its own homework. Research on “LLM-as-a-judge” setup shows that language models tend to rate output from themselves, or from their own model family, more favourably… There is a nuance here. When strong models prefer their own answers, it’s usually because those answers really were better. But when a strong model is wrong, it’s more likely than a weaker one to back its own wrong given answer. These are exactly the cases you should care about. So your criticism should come from a different model family, with a different role and a completely different question. You can never rely on one Model generated answer.
The playbook
Ten rules for building a System 2 around your own AI. Each one maps to something Kahneman described.
- Map the minefield. Kahneman’s most practical advice: you cannot switch off System 1 , but you can learn to recognise situations where it might be going to fail, and slow down here. You should do the same for AI. Sort your tasks by what an error costs: cosmetic, expensive or fatal. A typo in a social post is purely cosmetical. An entry requirement, a contract clause or a production database is not. Every rule below applies with more force the further down that scale you go.
- Make “I don’t know” pay. Models guess because guessing is rewarded. Reverse the incentive in your instructions: “I’m not sure, and here’s what I’d need” is a good answer, a confident guess is a failure. Put that in your system prompt or rules file, not only in your head. Question everything that you cant relate to.
- Ask what** is**** missing.** Fight WYSIATI with one standard question: “What information don’t you have that likely could change this answer?”
- Check which question really got answered. Fight substitution, Kahneman’s term for quietly answering an easier question, by asking the model to restate the question it actually answered before you accept the answer. You’ll be surprised how often it isn’t yours.
- Run a premortem. Gary Klein’s technique, which Kahneman champions. Don’t ask “what could go wrong?” Say: “It’s six months from now and this decision failed. Tell me how it happened.” That shift is the point: treating failure as a fact instead of a possibility surfaces more reasons. For people, it makes doubt safe to voice. For a model, it removes the easy way out of reassuring you that the plan is mostly fine.
- Take the outside view. Kahneman’s recipe for the planning fallacy: before judging this case, ask how often cases like this one go wrong. “How often projects like this get overrun?” beats “Will this project get overrun?” Ask the model for the baseline first.
- Climb the causal ladder. Don’t accept “X, therefore Y”. Ask about the chain: why, because, how do we know this? Then the counter: what if that assumption were actually false? Judea Pearl calls this climbing from association to intervention to counterfactuals. Causal parrots stay on the bottom rung unless you push them up.
- Get a second opinion from another model family. A different model, a different role, a different question. Not “is this right?” but “you are the border officer, the opposing lawyer, the auditor: find out why this fails.” A model checking itself isn’t a check.
- Turn mistakes into rules. This is where the intern analogy holds. AI is trainable, but the training doesn’t land in the model’s judgement. It lands in your rules, your memory files and your procedures. Every error you catch goes into a log, and every pattern in that log should become a rule to consider. That is how an organisation learns, and it will be the only way your AI setup learns right now.
- Guard your own System 1. double check the output you’re most likely to trust, especially the polished answers. Build some friction into the high stakes path: a checklist, a forced break, a second pair of eyes. Expect to dislike it. In a 2021 study on reducing over-reliance on AI, people rated the designs that worked best as the least pleasant to use. System 2 is supposed to feel like youa re doing some effort. That’s how you know it is switched on.
System 2 is not a feature
When I started working seriously with AI, it felt like teaching an intern fulltime, all the time. It still does, for a bit. But I stopped seeing that as a flaw we should overcome. Interns are quite trainable, and so is AI. The difference is where the training ends up: not in the intern’s head, but in the system around it.
The vendors probably will keep selling you System 2 as a feature: thinking modes, reasoning tokens, deep research, you name it. Use them, they can help. But don’t confuse a model that writes out its thought with a process that actually checks them. Kahneman spent his whole career showing that even the best human minds can’t reliably catch their own produced errors. It would be quite strange to expect more from a mere machine that was trained to sound sure right?
System 2 is not something you can buy. It’s something you try to build, and keep rebuilding every time it misses something.
