“But marinade” and leaked passwords are what researchers found in ChatGPT’s hidden reasoning

Security researchers found a vulnerability in the APIs of every major AI provider that lets them read the encrypted thought processes of reasoning models. A scan of publicly shared sessions turned up dozens of passwords and API keys.

When AI models like OpenAI’s o-series, Anthropic’s Claude, or Google’s Gemini “think” through complex tasks, they generate internal reasoning tokens. These thought processes are either shown to users as a summary or kept completely hidden. Providers encrypt the raw reasoning steps, partly to protect their intellectual property. A research team led by Alexander Panfilov has now found a way to extract these encrypted reasoning processes through a vulnerability in the APIs of all leading AI providers. For most queries, the number of extracted tokens matches the billed thinking tokens exactly, meaning the researchers are capturing the full internal reasoning, not just partial snippets.Ad The researchers say the encrypted thought processes are “fully portable across sessions, users, and models within a single provider.” Anthropic’s smaller model, Haiku 4.5, can read the thoughts of the far more capable Opus 4.8. Through jailbreaking, Haiku can be tricked into transcribing Opus’s raw thought processes word for word without attacking the more robust Opus directly. The same trick works with OpenAI and Gemini.AdDEC_D_Incontent-1 The story goes back to May, when cryptography expert Matthew Green discovered that encrypted reasoning blobs could be replayed outside their original context and reported it to the providers. According to Panfilov, their response was that “they don’t see any security implications in side channels or replays.” The new research strongly suggests that assessment was wrong. The vulnerability also feeds into the controversial “distillation” debate, where a less capable model is heavily improved by training on the outputs of a more powerful one, specifically its reasoning.Ad The researchers say it may have been possible for some time to extract reasoning processes for training proprietary models without breaking the cryptography. That supports concerns that Chinese model makers are using these reasoning traces to train their own models on chain-of-thought data. Kimi-K3 is one example. If its reasoning is pre-filled with just a few tokens from Opus’s thought processes, its output shifts measurably toward Opus, the researchers say. A memorization analysis showed that specific Claude and GPT reasoning segments are up to six orders of magnitude easier to extract from Kimi-K3 than from the next closest model. The researchers say this suggests Kimi-K3 may have been trained on such traces.AdDEC_D_Incontent-2 The attack isn’t expensive either, so scaling it up is feasible. The authors estimate API costs for decoding 10,000 traces at about $720. Kimi’s “poor” performance on cybersecurity benchmarks and complex math tasks also point to distillation, since these are tasks that are likely harder to recover even from raw chain-of-thought data.Ad Publicly shared sessions leak passwords and API keys The vulnerability also hits end users.