The Obedient Chooser
We are mistaking knowledge for intelligence and the systems that now "reason" haven't closed the gap.
Most of us have known the right thing to do and either done it - or not done it. We have known the sentence that would end the argument rather than extend it, and deployed it to bring peace. We knew the message could wait until morning, and sent it at midnight anyway. The knowledge was there and used badly or well.
We have a word meant to cover both of those things, the knowing and the using - we call it intelligence. A year or two ago the AI labs were racing toward "artificial general intelligence" and now they have gone shy of the term, declaring it surpassed, or misnamed. But we are still using the word intelligence, and the public still hears it in the way human beings always have.
Two halves of one word
When we call a person intelligent we mean two things at once, and rarely notice they are two.
The first is knowledge: what they know and what they know how to do. It is the half you can study for. It is also, funnily enough, the half you can write down, store, copy, and hand to someone else.
The second is judgement: the act of choosing in an uncertain state by someone who will own the outcome. Not knowing what, but deciding whether to, and being the one it happens to. It is what’s left after the knowledge has been considered; a self still has to choose what follows.
In ordinary speech, intelligence has long meant both of these qualities together. We might call a person with flawless knowledge who reliably makes bad choices clever, or, less kindly, a fool with a good memory.
“Artificial intelligence” was named in 1956, at a summer workshop at Dartmouth, and from the start it set out to build machines that could reason, abstract, even improve themselves - everything, in other words, except a self that owns the outcome. The standard textbooks set the goal as acting rationally and left aside the question of whether anyone was home inside. The choosing self was never the thing being built. And yet what has been built, at superhuman scale, is knowledge so fluent it performs the choosing too: laying out the reasons and settling on an answer. The capability may be real, but the chooser is effectively staged, or performed.
"But the models do think now"
The newest systems perform reasoning now: they lay out steps, weigh alternatives, correct themselves mid-stream….wait, that's not right, let me reconsider.
Watch a model work through a problem and it looks exactly like deliberation, like the second half of the word (possessing judgement), arriving. But I don't think the system is genuinely deliberating; the rest of this essay is my case for why.
The reasoning layer that these systems produce is applied expert knowledge of how to think. Pretraining gives it a colossal record of human reasoning to imitate. Then reinforcement shapes that reasoning from the outside: the model is rewarded for the chains of thought that reach good answers - in maths and code, where the answer can be checked automatically, and in softer domains against human preference, until it settles into the patterns that are rewarded.
This is real and useful. But it is still entirely only the first half of the word intelligence: knowledge of a procedure, applied. The reasoning layer doesn't add a chooser; it adds an extraordinarily proficient account of how choosers proceed.
Who owns the decision
So, is there really “someone” in there, or only the form of someone? It’s a deep question and the field is split on it. We can step around it because the argument doesn't need it answered. Give the system whatever inner life you like and the thing that matters is still missing.
Take whether to start a company.
The tools are intriguing here. They will model the whole venture, every strategy, every cost, and an agent will go and do some of the legwork as well. Ask outright and the system will give you a recommendation - start it, the unit economics work, the timing is good. That output looks like a decision.
But the decision is yours, because each strategy carries costs that will land on your life. The system that recommended the task is indifferent to whether you do it, or not; it’s the same after the company fails as after it succeeds. Take the example of whether to take a job, or which message to send tonight, and not a word changes - you can hand the system the reasoning every time, and handing it over is itself your choice.
You could ask a lawyer, she doesn't live with your decision either, and we still call what she does judgement. So why not the machine? The machine has consequences too, of a kind: give too much bad advice and users leave. That's a feedback loop. But it isn't the lawyer's loop. A named lawyer has her licence on the line, can be struck off, has to stand up in court. She chose the advice under uncertainty and she owns what she said. The lawyer owns the advice, you own the choice. The model owns neither. The line beneath every answer: the system can make mistakes, you should check its work, this is a precise refusal of any stake in the advice handed to you alongside the choice that was always yours. Lawyers are liable for the advice. AI labs, by design, are not.
The stakes of being embodied
In a single person, all of this is inseparable. You choose, and you are the one the choice happens to and judgement builds over a life: what's left behind by choices you’ve made and then had to live with. The model has read regretted messages and sent none of them. It holds the knowledge of judgement without ever holding the consequences for the actions taken as a result - but the consequences are what turns knowledge into judgement. The rest comes down to the body. We are born, we die, our feelings are wired into flesh, and when we are wrong, the cost reaches us as feeling before it reaches us as fact.
That is the part a machine has no version of. Grant it whatever inner life you like, it’s still never the one the cost lands on. And this state of being the one it happens to, not just having the feeling itself, is what a system can get better and better at simulating but can never truly inhabit.
What to call it, and what the confusion costs
It would feel hollow to come after the word intelligence without offering an alternative. These systems have been exposed to an enormous amount of human knowledge and then shaped, through deliberate annotation by the encoded judgement of a great many experts. When you ask one for advice, what you receive is the distilled judgement of those people, recombined and made fluent, delivered by a system that can’t tell you that is it, and what it is doing. It is manufactured expert judgement: the judgement is real, it simply isn't the machine's.
Effectively they are transformers of expert knowledge, and it might be closer to reality to simply call them this. Expert Knowledge Transformers. EKTs. It’s a little cumbersome and not as sexy as AGI, but at least it takes away the implication that someone in there is choosing.
The word choice matters - intelligence or knowledge, because of what it sets in motion. We fear these systems for concrete reasons already: for what they'll do to the truth, to our work. The use of the word intelligent adds another fear, of a different kind - we have been told they are intelligent, and intelligent, in ordinary use, means like us, knowing and choosing, with ends of its own. That fear is reasonable given the label, and the label is wrong. Fortunes are being staked on the arrival of a humanlike mind, and a hard moral question, whether anything is owed to a thing that might be choosing, gets dragged into every conversation by a word that was never meant to settle it.
And this confusion undersells what we gain by the use of EKTs. The whole worth of these tools is that they do not choose. They will never tire of you, never decide you've asked for something boring and refuse to go on, which is exactly what an intelligent chooser, sooner or later, would do. Even the agents don't change this: an agent works out its own steps, but the goal those steps serve is one you gave it, as built today, it never sets the goal itself. We want the knowledge on tap without the will, and intelligence obscures it by promising the single feature you should least want in a tool.
Put the two together: the will we don't want and the word that keeps promising it, and there it is, the thing being sold from every stage and pitch deck; and the thing some part of us desires to buy: not an intelligent chooser, but an obedient chooser. Something that judges like a mind and obeys like a tool. But those pull against each other. A chooser worth the name can choose against you; that’s what makes it a chooser and not a mechanism. So the more genuinely a system can judge, the less reliably it would do as it is told. That's the bind the word alignment is pointing at - the strain of asking for judgement and obedience in one breath. The tools we have are useful precisely because they never resolve that tension. They never had the second half at all.
The gap that doesn't close
We have spent thousands of years accumulating knowledge, and yet we still send the midnight message we know could have waited, still betray the people we love in the same old ways. Our trouble is not that we know too little. It is the gap between knowing what’s right and doing it, and that gap has never been closed by knowing more.
Intelligence, both halves of it, lives in that gap. In the act of choosing, in a state of uncertainty, by a self that will have to live in the world the choice makes.
The machines have given us the knowing, at a scale that’s astonishing, and that knowing includes methods of reasoning that are reshaping how we work. It is still just the knowing. The other half, the act of choosing, stays yours because you are the one who will live in the world made by your choice. You can hand the machine the reasons. Part of you will want to hand it the choice as well. But I urge you not to. Choosing is the most human element of the chain, because it is the thing that has lived consequences. And that’s something that no EKT will ever truly understand.