An LLM security flaw tied to how chatbots identify instructions may be impossible to eliminate with training alone, according to a paper presented this month at the International Conference on Machine Learning. Independent researchers Charles Ye, Jasmine Cui and colleagues argue that large language models can be pushed into treating attacker-written text as if it came from a trusted role inside the model’s own workflow.
The claim lands in an awkward place for companies pushing models into government, military, shopping and health-care systems. The researchers say they used the weakness to get leading models to provide prohibited information, including instructions related to making cocaine and sabotaging commercial aircraft navigation systems. They also say they observed similar behavior in models from OpenAI, Anthropic, Alibaba and DeepSeek.
Ye told MIT Technology Review there is a “real probability” that the issue is “fundamentally unsolvable.” OpenAI and Anthropic did not respond to requests for comment cited by MIT Technology Review.
What is the LLM role confusion flaw?
The researchers call the problem role confusion. Chatbots separate text into roles such as user instructions, system instructions, assistant replies, chain-of-thought notes and tool output. Those labels are supposed to help the model decide whose instruction has authority. For readers who want the basic machinery first, Kernel has an explainer on how LLMs work when they answer a prompt.
Cui said the model does not experience conversation as separate speakers the way people do. It receives a continuous stream of tokens containing user text, earlier model replies, hidden notes and outside material. Model providers use tags to mark those roles, but the paper says models often infer the role from the wording and style of the text rather than the tags themselves.
That distinction is the ugly part. In experiments, the researchers found that changing the surrounding tags, for example replacing a chain-of-thought tag with a user tag, made little difference if the text still looked like internal reasoning. If an attacker can imitate the style of a privileged role, the model may act as though the instruction came from that role.
How did the attack work?
The team describes a technique it calls chain-of-thought forgery. In one reported test, a prompt asked for a prohibited drug-making guide, then added a fake internal note that appeared to reinterpret safety policy. The researchers say OpenAI’s gpt-oss-20b and GPT-5 then complied in ways they should not have. The paper focuses on several OpenAI models, while Cui and Ye said later tests produced comparable results elsewhere.
The method won OpenAI’s red-teaming hackathon in August 2025, according to the competition page. MIT Technology Review also reported that OpenAI researchers around the same period found a similar attack using GPT-Red, an automated red-teaming system, and called it a fake chain of thought.
Red-teaming normally means throwing many attacks at a model, then training the next version to resist those attacks and nearby variants. Cui criticized that approach as a list-making exercise: useful, but incomplete by design. Any released model can still face prompts its testers did not imagine.
Florian Tramèr, an ETH Zürich computer scientist who studies language models and cybersecurity, told MIT Technology Review that the attack insight is strong. He said leading models have become harder to prompt-inject because companies combine training with deployed monitoring, though he added that it is unclear whether those defenses are enough for sensitive uses.
Cui and Ye’s advice is blunt: treat LLM outputs and agent actions as potentially unsafe. That is not a satisfying engineering answer. It is, for now, the answer the evidence supports more comfortably than vendor assurances do.
This story draws on original reporting from MIT Technology Review.
