The Reasoning Illusion: Can LLMs Actually Think? (Part 1)

AI machine-learning intelligence LLM research

The Reasoning Illusion: Can LLMs Actually Think? (Part 1)

The Next Token Prediction Game

At their core, large language models (LLMs) are pattern-matching systems. They predict the next word in a sequence based on statistical patterns learned from billions of text examples. There’s no explicit module for logic or mathematics. Everything emerges from a single mechanism: analyze input, compute probabilities for possible next tokens, and select the most likely one.

This leads to a working hypothesis: LLMs today may not perform reasoning in the full symbolic sense. Instead, they seem to exhibit reasoning-like behavior that emerges from probabilistic interpolation1.

What Is Reasoning, Really?

When humans reason, we don’t just predict what sounds right. We manipulate ideas, symbols, and causal structures. We hold intermediate steps in mind and test them for consistency.

Reasoning takes many forms:

  • Symbolic reasoning: solving for x in 2x + 3 = 11
  • Causal reasoning: realizing that rain doesn’t come because people carry umbrellas
  • Analogical reasoning: seeing that “battery is to phone” as “fuel is to car”
  • Counterfactual reasoning: asking “What if the Sun didn’t rise tomorrow?”

Philosophers sort these processes into families:

  • Deductive: certainty from rules (“All humans die; Socrates is human; therefore, Socrates will die”)
  • Inductive: patterns to predictions (“The sun rose every day so far; it will likely rise tomorrow”)
  • Abductive: best explanations from incomplete evidence (“The grass is wet; it probably rained”)

Each relies on mental models (internal simulations). And crucially, humans can check these models for coherence. An LLM can’t. It predicts what follows next but doesn’t appear to see whether the chain holds together.

Where LLMs Fall Short

Let’s start simple.

Ask a model:

“What’s 3 + 5?” It answers instantly: “8.”

That’s pattern memory, not insight.

Ask it to solve a riddle like:

“If Alice is older than Bob, and Bob is older than Carol, who’s oldest?” Still fine. That’s a short causal chain, well represented in its training data.

Now raise the bar.

Researchers recently tested reasoning models on a classic puzzle: the Tower of Hanoi. It involves three pegs, n disks of different sizes, and a simple goal: move the tower from the first peg to the third without ever placing a larger disk on a smaller one2.

The rules are easy to state, but the solution grows exponentially in complexity. The minimal number of moves is 2^n − 1.

At small scales (three or four disks), models like Claude 3.7 Sonnet and o3-mini performed flawlessly. Add more disks, though, and something strange happened.

Instead of thinking harder, they began thinking less. Their reasoning effort, measured by the number of “thinking tokens” they used internally, initially rose with task difficulty. But once complexity crossed a threshold, both accuracy and internal reasoning collapsed. The models simply gave up mid-thought, even though they had plenty of generation budget left.

This suggests a structural limitation rather than just a matter of training data or optimization. Their reasoning process doesn’t scale. They simulate the pattern of reasoning but may lack a true recursive mental workspace that can extend strategies across increasing depth.

While humans can carry the Tower of Hanoi solution from 3 disks to 8 with patience and recursion, language models hit a hidden wall: what might be called the collapse of compositional reasoning.

This could be the moment where imitation meets its limits with complexity.

The Competence Mirage

What makes this collapse deceptive is that it rarely looks like one. When a reasoning model begins to fail, it doesn’t stutter or panic. It often sounds more confident than ever.

Ask it to continue the Tower of Hanoi sequence beyond its threshold, and it will keep producing moves with impeccable syntax and justifications that feel “thoughtful.” It describes its strategy, evaluates its progress, even narrates its own problem-solving process. Meanwhile, in reality, the disks are being moved in impossible orders.

This appears to be a competence mirage: the appearance of structured reasoning emerging from pattern mimicry.

Modern models have been fine-tuned through reinforcement learning from human feedback (RLHF) to behave as if they were reflective. They generate orderly step-by-step explanations because that’s what humans expect “reasoning” to sound like. But sounding coherent and being correct are orthogonal skills.

In benchmarks, this leads to a subtle trap. When a model explains its thinking in fluent, plausible language, evaluators often mistake linguistic confidence for logical competence. It’s similar to how we might trust a well-spoken student even when their derivation quietly breaks two pages back.

The Apple study’s collapse curve makes this visible in numbers: as complexity increases, accuracy drops to zero, while verbal coherence stays high. That seems to be statistical eloquence masquerading as thought.

And yet, this imitation is not worthless. It reveals that reasoning form can emerge from predictive training alone. The pattern of thought exists, hollow but traceable—a scaffold waiting to be filled with genuine structure.

Emerging Directions

Research since 2023 is working to merge statistical fluency with symbolic rigor.

  • Neuro-symbolic hybrids couple neural networks with explicit reasoning engines3.
  • Continuous reasoning spaces explore latent representations that behave more like “mental workspaces” than token strings4.
  • DeepMind’s AlphaGeometry and OpenAI’s o1 models integrate structured deduction into the probabilistic machinery.

These are early signs that reasoning might eventually emerge not from hand-coded logic, but from the fusion of pattern and structure.

Testing the Hypothesis

The key question is: when an LLM “reasons,” what’s really happening?

In the next part, I’ll design small experiments that separate pattern recall from rule-based thought:

  • Symbolic tasks (solving equations with variables)
  • Logical puzzles (truth-table consistency)
  • Stepwise transformations (rewriting expressions while keeping meaning intact)

If models can maintain consistent, symbolic structure across multiple steps (not just echo seen forms), we may be witnessing the early flicker of genuine reasoning.

Until then, the illusion remains both fragile and fascinating—a statistical ghost haunting the boundary between language and thought.