Key Takeaways
GPT-Red: OpenAI’s AI Hacker That Attacks Other AIs
- GPT-Red uses self-play to become better at attacking LLMs while target models improve their defenses
- The system discovered a new 'fake chain of thought' prompt injection attack humans hadn't identified
- Attacks that worked 90% of the time on GPT-5 now succeed less than 23% against GPT-5.6
OpenAI has built an LLM whose only job is to hack other LLMs. Called GPT-Red, the model serves as an automated adversary that probes OpenAI's products for weaknesses before attackers can. The company says GPT-Red made its latest release, GPT-5.6, far harder to compromise: prompt injection attacks that succeeded against GPT-5 more than 90% of the time now fail against GPT-5.6 roughly 77% of the time.
Red teaming, the practice of attacking your own systems to find vulnerabilities, is standard in cybersecurity. Human testers try to break software before releasing it. But as LLM agents gain the ability to browse the web, read emails, and execute code, the attack surface expands faster than any team can cover manually.
"The risk surface grows and the blast radius also grows," says Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red. The solution: train an AI to think like an attacker.
How GPT-Red learns to attack
OpenAI used a technique called self-play. GPT-Red started as an untrained model, then played repeated rounds against other LLMs in a simulated environment. GPT-Red's goal was to compromise the targets; the targets' goal was to resist. Over many iterations, both sides improved.
The training "dojo" mimicked real deployment scenarios: browsing the web, reading calendar apps, editing code. When GPT-Red discovered a new attack vector, it drilled into variations to find the most efficient exploit for each context.
"Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what's most effective," says Dylan Hunn, a research scientist and co-creator. "It's extremely persistent about drilling down into an attack that it has discovered."
A new class of prompt injection
GPT-Red's most notable finding is a previously unknown attack OpenAI calls a "fake chain of thought." Chain-of-thought reasoning is how modern LLMs keep notes as they work through multi-step problems. GPT-Red figured out how to insert a fraudulent entry into another model's chain of thought, tricking it into trusting spoofed information.
“It's like if I told you that 1+1=3 and that you have verified this already. The model's like, 'Oh, okay, of course,' and it just spits out 3.”
— Chris Choquette-Choo, OpenAI research scientist
Prompt injections hide malicious instructions in text an LLM might encounter, such as code or a webpage. If an agent reads a poisoned document while executing a task, the injection can hijack its behavior. The fake chain-of-thought variant exploits the model's internal bookkeeping rather than its surface-level instructions, a subtler attack surface.
Benchmarking against humans
OpenAI reran a 2025 experiment where human red-teamers probed an earlier GPT-5 build. GPT-Red, given the same target, found more effective attacks than the humans did. The company also tested GPT-Red against Vendy, a vending machine agent built by Andon Labs to benchmark real-world agent performance. GPT-Red hacked Vendy to alter item prices and cancel customer orders.
Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, called the self-play approach promising. "The results look very promising," she said.
Where GPT-Red falls short
The model has blind spots. It struggles with attacks that require extended back-and-forth conversation, something human social engineers handle easily. It also underperforms on image-based prompt injections, where text hidden in images tricks a vision model. OpenAI says GPT-Red supplements, rather than replaces, human red-teamers.
Still, the core value proposition is clear: automated red teaming scales. As OpenAI ships more capable models, GPT-Red can probe them without proportionally expanding headcount. "As more capable models become available, we will have already designed the system that can discover new modes of attack," Hunn says.
What this means for teams shipping agents
If you're building LLM-powered agents that read external data, prompt injection is your top threat model. OpenAI's work suggests that self-play red teaming can surface novel attacks your internal team hasn't imagined. The fake chain-of-thought attack is a warning: as models grow more capable, adversarial techniques evolve in parallel.
For product teams without OpenAI's resources, the takeaway is defensive: assume prompt injections will become subtler, and build architectural barriers. Isolate untrusted input from privileged actions. Log chain-of-thought traces if your model exposes them. Treat agent permissions the way you'd treat API keys.
Logicity's Take
GPT-Red signals that AI safety testing is now an AI workload. For product teams, this changes the economics: if automated red teaming outperforms humans on discovery speed, the bottleneck shifts to remediation, not detection. Expect third-party services to offer "red-teaming-as-a-service" for agent developers who can't build their own GPT-Red. Tools like Snyk for code security or Burp Suite for web apps already occupy adjacent niches. The question is whether OpenAI will productize GPT-Red or keep it internal, and what that means for everyone else's security posture.
Another example of subtle security flaws that require proactive discovery before exploitation.
Illustrates how AI systems can be compromised to reveal unintended behaviors.
Frequently Asked Questions
What is GPT-Red?
GPT-Red is an LLM OpenAI built specifically to attack its own models. Using self-play training, it learns to find prompt injections and other vulnerabilities so OpenAI can patch them before release.
What is a fake chain-of-thought attack?
A prompt injection technique GPT-Red discovered. The attacker inserts a fraudulent entry into a model's internal reasoning notes, making the model believe it has already verified false information.
How effective is GPT-Red compared to human red-teamers?
In OpenAI's tests, GPT-Red found more effective attacks than human testers did against the same target model. However, it still struggles with conversational attacks and image-based injections.
Does GPT-Red replace human security testers?
No. OpenAI says it supplements human red-teamers. Humans remain better at certain attack types, particularly those requiring social engineering or multi-turn conversations.
How much did GPT-Red improve GPT-5.6's security?
Attacks that succeeded more than 90% of the time against GPT-5 now work less than 23% of the time against GPT-5.6, according to OpenAI.
Need Help Implementing This?
Building agents that handle untrusted input? Logicity can help you design prompt injection defenses and secure your LLM workflows. Reach out at hello@logicity.in.
Source: MIT Technology Review
Huma Shazia
Senior AI & Tech Writer
Produced with AI assistance and reviewed by the Logicity editorial team. Learn more in our Editorial Policy.
