OpenAI has built a model called GPT-Red to attack its other models before outsiders do. The company says the system helped make GPT-5.6, its latest flagship model, more resistant to cyberattacks than earlier releases.
GPT-Red automates red-teaming, the practice of trying to break a system so engineers can patch weak points before release. Human testers still do that work, but OpenAI says the job is getting harder as large language models are used as agents that can handle files, visit websites, read email and calendars, edit code, and interact with other agents.
Nikhil Kandpal, an OpenAI research scientist who co-created GPT-Red, said the “risk surface” and “blast radius” grow as models get more capable. Dylan Hunn, another OpenAI researcher on the project, said the company wanted a testing system that could find new attack methods as stronger models arrive.
Most of OpenAI’s work focused on prompt injection. In that attack, malicious instructions are hidden in text a model may read, such as a web page, an email, or code. If the attack works, the model may follow the hidden instruction instead of the developer’s or user’s intended instruction. The examples OpenAI described include copying confidential information, damaging a code base, or producing harmful output.
How GPT-Red was trained
OpenAI said it started with a model that had not been trained as an attacker, then placed it in a self-play setup with several other models. GPT-Red’s role was to attack. The other models’ role was to defend. Across repeated rounds, GPT-Red improved at finding working attacks, while the target models improved at resisting them.
The company trained the system in simulated deployment settings, including web browsing, email and calendar use, and code editing. When GPT-Red found a promising attack, OpenAI said it tested variations to identify which version worked best in a given scenario.
Hunn said GPT-Red was stronger than human red-teamers at grinding through a discovered attack and finding the version most likely to work. OpenAI said it reran a 2025 experiment in which human red-teamers had looked for weaknesses in an earlier GPT-5 model, and GPT-Red found effective attacks more often than the humans had.
OpenAI also said GPT-Red found a prompt injection method its researchers had not previously seen, which they call a fake chain of thought. A chain of thought, as described by the company, is a scratchpad-like record a model uses while working through a task. GPT-Red found a way to insert a fake entry into that record so the target model would act as if it had already verified false information.
Chris Choquette-Choo, an OpenAI researcher on the team, compared it to telling someone that 1+1=3 and that they had already checked the math. The model then accepts the spoofed premise, according to OpenAI’s account.
Results and limits
OpenAI said attacks developed by GPT-Red succeeded more than 90% of the time against GPT-5, released in August last year. Against GPT-5.6, fewer than 23% of those attacks worked, according to the company.
The model also attacked Vendy, a vending machine agent made by Andon Labs, which evaluates agents on real-world tasks. OpenAI said GPT-Red made Vendy change item prices and cancel a customer order.
Jessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology, said OpenAI’s self-play approach looked promising. She also said human expertise remains important, especially for identifying where human testing is most needed.
OpenAI acknowledged GPT-Red’s gaps. The company said it is weaker at attacks that require a back-and-forth conversation with a target, and it is still limited at using images, which can carry text for prompt injection. OpenAI says GPT-Red supplements human red-teamers rather than replacing them, including by generating variants of attacks people discover.
OpenAI says it will not release GPT-Red. Choquette-Choo said reproducing it would not be trivial, citing more than a year of work and the company’s compute resources.
This story draws on original reporting from MIT Technology Review.