Thinking Out Loud
I have been telling people that chain-of-thought or reasoning in AI is just as likely to be bullshit.
I noticed a disturbing trend a while back of people saying that reasoning traces, from chain-of-thought or from reasoning models, could be treated as explanations for the final answer.
It’s definitely not the case. These traces can easily be hallucinations dressed up as logic.
But I didn’t realise that it could apply to how I move through life. This week, some events made me think that sometimes we need to check our own thinking. And that thinking out loud might be one way to do it.
Week 16 of life post-MAS. (Links to past weeks in my newsletter.)
Fewer meetings. But somehow more people.
The familiar. An ex-colleague who has been doing AI development for a long time. Collaborators on a paper on AI and work. A call with a fellow disciple based in the US. An experienced hand at AI for social sensemaking. A secondary school classmate and her colleague, who I met online.
New connections. A room of more than 100 IT architects at a large bank. A room full of folks in the digital asset space. A senior banker from Hong Kong wanting to explore ideas. A consultant in the fund management space who also wanted to bounce ideas.
A podcast I did on AI risk management in central banking came out and I said I like doing such podcasts as they forced me to think out loud in a permanent manner.
Which led me to realize that there was something to learn from the pitfalls in chain-of-thought.
What chain of thought actually is
In AI, chain-of-thought or reasoning is when a model shows its working. Step by step. Each step building on the last. It sounds rigorous. It looks rigorous.
But the problem is that the steps can be wrong. Each wrong step compounds the previous one.
And if the reasoning drifts early, the reasoning traces and the conclusion can be confident, fluent, sycophantic bullshit. Coherence is not correctness in the world of LLMs.
Which is also, I realised this week, a fairly accurate description of how I sometimes move through life.
Before I left
Before I tendered my resignation from MAS, I was in a chain-of-thought spiral I didn’t recognise as one.
A spiral of frustrations, each one feeling valid. Each one building on the last. Self-inflicted in some ways. Compounding quietly but not in a helpful manner. The reasoning felt sound from inside, each frustration justified by the previous one. The chain ran for a while.
The act of tendering my resignation broke it. No more spirals. An action that forced a reality check. And I have been telling many folks that I have not regretted it since. For one thing, it helped me to preserve some of the relationships I valued.
The talks
Since leaving MAS I have been talking and talking. I have always felt I should deliver insights. Real ones. Not surface observations. So each talk, I built one insight on another. The chain grew. Each addition felt justified. Each layer felt valuable. From inside the chain, it felt necessary. I thought it was great. I had reached the point where even a broken projector was not an issue.
This week a collaborator who has seen many such rooms said something gently after one of my talks. Perhaps for this audience, you were providing too much information.
I hadn’t felt it. Not once for the past 16 weeks.
That’s the problem with chain-of-thought. From inside, coherent feels like correct. Depth feels like value. And the longer the chain runs without correction, the harder it is to feel the drift.
The Karpathy Break
Andrej Karpathy released AutoResearch earlier this year. An AI agent given a training setup, left to experiment overnight. The agent modifies the code, runs a short experiment, checks if it improved, keeps or discards, repeats.
The real innovation isn’t automation. It’s the break in the chain-of-thought. Three decisions at each step .
1️⃣ Hypothesise - what to try next
2️⃣ Evaluate empirically - did it actually improve
3️⃣ Keep or discard - commit or roll back
My collaborator’s comment was my Karpathy Break moment. Not more reasoning from inside the chain. An empirical signal from outside it. Keep or discard. I kept the lesson. And I need to discard the approach.
Thinking out loud
Writing these weekly reflections is another version of the same thing. Not because the output is always right. But because putting the chain outside your own head gives others a chance to catch what you can’t catch yourself. Sixteen weeks of thinking out loud. I’ve drifted here and there. But generally the direction feels right.
Which is, of course, exactly what a chain-of-thought would conclude about itself.
The irony
I have been warning people about chain-of-thought for months. In talks. In posts. In conversations with risk teams, regulators, practitioners.
And yet. A chain of frustrations before I left, running unchecked until I tendered. A chain of insights across multiple talks, running unchecked until a collaborator said something gently.
Interesting.