Red-Teaming after Mythos: Why AI Security Demands a New Playbook

TL;DR

• AI red-teaming is fundamentally different from traditional cybersecurity: Instead of finding discrete bugs in deterministic systems, you're probing probabilistic models for emergent behaviors that exist on a spectrum of severity—there's no clean "patched/unpatched" binary.
• The Mythos benchmark revealed critical gaps: Gray Swan's research showed that even top-tier models fail basic adversarial robustness tests, with success rates dropping from 90%+ on standard benchmarks to below 50% when attackers can adaptively probe the system.
• Product builders need layered defenses: No single technique (prompt engineering, fine-tuning, or output filtering) provides sufficient protection—effective AI safety requires defense-in-depth with monitoring, rate limiting, and continuous red-teaming integrated into your development cycle.
• The economics of AI security are inverted: Unlike traditional software where defenders have structural advantages, AI systems give attackers infinite attempts at near-zero marginal cost, fundamentally changing the security calculus for product teams.

---

If you're building AI products in 2025, you've probably added some version of "red-teaming" to your pre-launch checklist. Maybe you hired a security consultant to probe your chatbot for jailbreaks. Perhaps you ran your model through a benchmark suite. You might even have a dedicated safety team running adversarial tests.

But here's the uncomfortable truth: most of what we call "AI red-teaming" today is security theater.

I don't mean this as a criticism of teams doing their best with limited tools. I mean it as a wake-up call about how fundamentally different AI security is from the cybersecurity practices we've borrowed the terminology from. The recent work from Zico Kolter and Matt Fredrikson at Gray Swan—particularly their Mythos benchmark—crystallizes why our inherited mental models are inadequate and what we need to do differently.

The Illusion of Discrete Vulnerabilities

Traditional red-teaming assumes you're hunting for specific, exploitable bugs. You find a SQL injection vulnerability, you patch it, and that attack vector is closed. The system is either vulnerable or it isn't. This binary thinking pervades how we approach security: CVE databases, patch management, penetration testing reports with neat lists of "Critical" and "High" severity findings.

AI systems don't work this way.

When you red-team a language model, you're not finding bugs in code—you're exploring the learned behavior of a probabilistic system with billions of parameters. As the Gray Swan team discovered through their Mythos research, what looks like a "solved" safety problem on standard benchmarks completely falls apart under adaptive adversarial pressure. Their key insight: most existing safety evaluations test models against fixed, non-adaptive attacks. But real adversaries iterate.

The Mythos benchmark introduced a crucial innovation—adaptive attacks where the red-team approach evolves based on model responses. What they found was sobering. Models that scored 90%+ on standard safety benchmarks dropped below 50% success rates when faced with adaptive adversaries who could probe the system, learn its boundaries, and craft targeted attacks.

This isn't a bug. It's the nature of the system.

Why Your Prompt Engineering Won't Save You

Every AI product team I talk to has some version of a "system prompt" they believe will keep their model safe. It usually includes phrases like "You are a helpful assistant that never provides harmful information" or "Refuse requests that could be used for illegal purposes."

This is the AI equivalent of client-side input validation—a speed bump, not a barrier.

The problem isn't that system prompts are useless (they're not), but that we've dramatically overestimated their robustness. Kolter and Fredrikson's work demonstrates that even sophisticated prompt-based defenses can be systematically defeated through iterative probing. An attacker doesn't need to find the one magic jailbreak—they can explore the probability distribution of your model's outputs until they find a path through your defenses.

My take on this is straightforward: if your safety strategy can be summarized as "we wrote a really good system prompt," you don't have a safety strategy. You have a suggestion box that polite users will respect and determined adversaries will ignore.

This doesn't mean abandoning prompt engineering—it means understanding its place in a layered defense. Your system prompt is your first line of defense, not your only one. It should be complemented by fine-tuning for safety, output classifiers, rate limiting, and continuous monitoring. None of these individually provides complete protection, but together they raise the cost of attack and create detection opportunities.

The Economics of Infinite Attempts

Here's where AI security gets truly challenging: the cost asymmetry is completely inverted from traditional cybersecurity.

In traditional systems, attackers face significant costs. Each penetration attempt risks detection. Exploits need to be carefully crafted. There's often a single point of entry, and once defenders patch it, the attacker needs to start over. The defender's structural advantage is that they control the system and can modify it.

In AI systems, attackers get infinite attempts at near-zero marginal cost. They can query your model thousands of times, learning its behavior, mapping its boundaries, and iterating on attacks. Each query gives them more information about your system's probability distribution. And here's the kicker: every "failed" jailbreak attempt is actually a successful reconnaissance mission.

Gray Swan's research quantifies this asymmetry. In their adaptive attack scenarios, adversaries could systematically explore the model's response space, building a map of which phrasings, contexts, and framings would elicit desired (from the attacker's perspective) outputs. The more queries they could make, the higher their success rate climbed.

For product builders, this has immediate implications:

Rate limiting isn't optional—it's foundational. You need to make adaptive attacks expensive by limiting how quickly any single user can probe your system. This won't stop determined adversaries, but it changes the economics.

Monitoring query patterns is as important as monitoring outputs. A user making 100 variations of the same request is conducting reconnaissance, even if none of the individual queries trigger your safety filters.

You can't "solve" safety and move on. Unlike patching a SQL injection vulnerability, AI safety requires continuous red-teaming, monitoring, and iteration. The threat model evolves as attackers learn.

What Mythos Teaches Us About Evaluation

The Mythos benchmark represents a significant evolution in how we think about AI safety evaluation. Traditional benchmarks test models against fixed datasets of known problematic prompts. This is useful for establishing baselines, but it's not how real attacks work.

Mythos introduced three critical elements:

1. Adaptive adversaries that iterate based on model responses
2. Goal-oriented attacks that measure success by achieving specific harmful outcomes, not just eliciting flagged content
3. Transferability testing to see if attacks that work on one model can be adapted to others

The results were illuminating. Models that appeared robust on static benchmarks showed significant vulnerabilities when faced with adaptive attacks. More concerningly, attacks that succeeded on one model often transferred to others with minimal modification, suggesting that current safety training creates similar weak points across different systems.

For product teams, this means your pre-launch safety evaluation needs to include adaptive red-teaming. You can't just run your model through a fixed benchmark and call it secure. You need humans (or increasingly, other AI systems) actively trying to break your safety measures, learning from failures, and iterating on attacks.

This is expensive and time-consuming, which is why many teams skip it. But the cost of inadequate safety testing isn't just reputational—it's existential for AI products. One viral jailbreak can destroy user trust and invite regulatory scrutiny that takes years to recover from.

Building Robustness Into Your Development Cycle

The most important insight from Gray Swan's work isn't about specific attack techniques or defense mechanisms—it's about mindset. AI safety isn't a feature you add before launch; it's a property that emerges from your entire development process.

Here's what this looks like in practice:

Start with threat modeling. Before you write a single line of code, map out what adversarial use of your system would look like. Who would want to misuse it? What would they try to achieve? What resources would they have? This isn't paranoia—it's product design.

Red-team continuously, not just before launch. Every significant model update, prompt change, or feature addition should trigger a new round of adversarial testing. The threat surface evolves with your product.

Instrument everything. You can't defend against attacks you can't detect. Log query patterns, track unusual usage, monitor for repeated similar requests. Build dashboards that surface anomalies in real-time.

Assume your defenses will be defeated. This isn't defeatism—it's realism. Plan for what happens when (not if) someone finds a jailbreak. How quickly can you detect it? How will you respond? What's your communication strategy?

Invest in the tooling layer. The field of AI red-teaming is young, and the tooling is primitive compared to traditional cybersecurity. If you're serious about safety, you'll need to build custom tools for adversarial testing, monitoring, and response. This is infrastructure, not overhead.

I think the teams that will succeed in building robust AI products are those that treat safety as a core competency, not a compliance checkbox. This means hiring people who understand adversarial machine learning, not just traditional security. It means allocating significant engineering resources to safety tooling and monitoring. It means being willing to delay launches when safety testing reveals gaps.

This is expensive. But the alternative—shipping products with superficial safety measures and hoping for the best—is far more expensive in the long run.

The Emerging Discipline of AI Red-Teaming

We're witnessing the birth of a new discipline. AI red-teaming is evolving beyond borrowed cybersecurity practices into something distinct, with its own methodologies, tools, and expertise.

Gray Swan's work represents one approach—academic rigor applied to adversarial robustness. But we're also seeing the emergence of specialized red-teaming firms, open-source adversarial testing frameworks, and even AI systems designed to red-team other AI systems.

This professionalization is necessary and overdue. The gap between academic research on adversarial robustness and practical product security has been too wide for too long. Researchers publish papers demonstrating attacks; product teams scramble to patch specific vulnerabilities without addressing underlying fragility.

What we need is a systematic approach to AI security that acknowledges the fundamental differences from traditional cybersecurity while leveraging relevant insights from that field. This means:

Standardized evaluation frameworks that go beyond static benchmarks to include adaptive adversarial testing.

Shared knowledge about attack patterns (while being thoughtful about responsible disclosure—we don't want to create a cookbook for bad actors).

Professional certification and training for AI red-teamers, establishing standards for what constitutes thorough adversarial testing.

Regulatory frameworks that recognize the unique challenges of AI safety and set appropriate standards without stifling innovation.

We're not there yet. But the work coming out of groups like Gray Swan, along with increased attention from policymakers and industry leaders, suggests we're moving in the right direction.

Practical Recommendations for Product Builders

If you're building AI products today, here's what you should be doing:

1. Implement layered defenses. Combine prompt engineering, fine-tuning, output filtering, rate limiting, and monitoring. No single layer is sufficient.

2. Red-team with realistic adversaries. Don't just test against known jailbreaks. Have someone actively try to break your system, learning from each attempt.

3. Monitor for reconnaissance. Track query patterns that suggest users are probing your system's boundaries, even if individual queries seem benign.

4. Plan your incident response. When a jailbreak goes viral (and it probably will at some point), how quickly can you respond? What's your communication plan?

5. Stay current with research. The field is evolving rapidly. Subscribe to relevant research feeds, attend conferences, and maintain relationships with academic groups working on adversarial robustness.

6. Invest in custom tooling. Off-the-shelf security tools aren't designed for AI systems. You'll need to build specialized monitoring, testing, and response capabilities.

7. Be transparent about limitations. Don't oversell your safety measures. Users and regulators will be more forgiving of acknowledged limitations than undisclosed vulnerabilities.

The reality is that we're still in the early days of understanding how to build robustly safe AI systems. The Mythos benchmark and Gray Swan's broader research program reveal how much we don't know and how inadequate our current approaches often are.

But this isn't cause for despair—it's a call to action. The teams that take AI safety seriously, invest in proper red-teaming, and build robust defenses will have a significant competitive advantage as the market matures and regulatory scrutiny increases.

The question isn't whether AI red-teaming will become a standard practice—it will. The question is whether you'll be ahead of that curve or scrambling to catch up after an incident forces your hand.

Beyond Mythos: The Road Ahead

The work from Kolter, Fredrikson, and the Gray Swan team represents an important milestone, but it's just the beginning. As AI systems become more capable and more widely deployed, the stakes for getting security right only increase.

We need to move beyond the mindset that AI safety is about preventing specific bad outputs and toward understanding it as ensuring robust, predictable behavior under adversarial conditions. This is a harder problem, but it's the right problem.

For product builders, this means embracing uncertainty and complexity. There's no checklist that guarantees your AI system is safe. There's no certification that proves you've done enough. Safety is an ongoing practice, not a destination.

But here's the opportunity: the teams that figure out how to build genuinely robust AI systems won't just avoid disasters—they'll build products that users can trust, regulators can approve, and enterprises can deploy with confidence. That's a significant competitive moat in an increasingly crowded market.

The era of AI security theater is ending. The era of serious, systematic, continuous adversarial testing is beginning. The question is whether your team will lead that transition or be dragged along by it.