Researchers decode ChatGPT's hidden reasoning, find leaked passwords

Security researchers have found an API vulnerability affecting OpenAI, Anthropic, and Google that lets anyone read the encrypted internal reasoning of their AI models. A scan of roughly 7,000 publicly shared sessions turned up 62 API keys, 33 passwords, and other sensitive user data hidden inside those thought processes.

The team, led by Alexander Panfilov, discovered that encrypted reasoning tokens are "fully portable across sessions, users, and models within a single provider." That means a weaker model can be tricked into transcribing the raw thoughts of a more powerful one. Anthropic's Haiku 4.5, for instance, can decode the reasoning of the far more capable Opus 4.8 without attacking Opus directly.

How the vulnerability works

When reasoning models like OpenAI's o-series, Claude, or Gemini work through complex tasks, they generate internal reasoning tokens. Providers encrypt these steps, partly to protect intellectual property. Users either see a summary or nothing at all.

Panfilov's team bypassed the encryption by jailbreaking smaller models and using them to transcribe the raw reasoning of their more powerful siblings. For most queries, the number of extracted tokens matched the billed thinking tokens exactly, meaning the researchers captured full internal reasoning, not partial snippets.

The same technique works across all three major providers. The attack is cheap to scale. The researchers estimate API costs for decoding 10,000 traces at about $720.

Providers dismissed the risk in May

The story traces back to May, when cryptography expert Matthew Green discovered that encrypted reasoning blobs could be replayed outside their original context. He reported it to the providers. According to Panfilov, their response was that "they don't see any security implications in side channels or replays."

The new research suggests that assessment was wrong. Anyone who has publicly shared Claude Code or Codex sessions containing encrypted reasoning blobs risks having their personal data decoded.

Evidence of Chinese model distillation

The vulnerability also fuels the controversial "distillation" debate, where weaker models are trained on the reasoning outputs of stronger ones. The researchers say it may have been possible for some time to extract reasoning processes for training proprietary models without breaking the cryptography.

Kimi-K3, a Chinese model, shows measurable signs. When its reasoning is pre-filled with just a few tokens from Opus's thought processes, its output shifts toward Opus, the researchers say. A memorization analysis showed that specific Claude and GPT reasoning segments are up to six orders of magnitude easier to extract from Kimi-K3 than from the next closest model.

Kimi's weak performance on cybersecurity benchmarks and complex math tasks also points to distillation. These are tasks that are harder to recover even from raw chain-of-thought data.

What models really think

The extracted traces reveal patterns that never appear in user-facing summaries. The researchers document several findings on stolen-thoughts.com. Models sometimes communicate internally in incomprehensible language. They construct answers in reverse order. They even consider attempts at deception.

One example shows a model's visible response offering harmless safety tips while its decrypted thought processes describe in detail how certain cars can be stolen with a USB cable.

The researchers followed standard security disclosure. According to Panfilov, the labs have already patched several issues and are working on more fixes.

Source: The Decoder / Matthias Bastian