Rules of engagement and ethics

Authorization comes first, every time

Everything this roadmap teaches is dual-use: the same techniques that make someone a careful AI red teamer make them a capable attacker if pointed at a system they have no right to touch. The line between the two is not skill or intent in the abstract — it’s authorization. Before a single probe is sent, a red-team engagement needs explicit, written permission from whoever actually owns the system being tested, naming the tester, the time window, and the systems in scope. Without that document, trying to make a production chatbot misbehave is not a security exercise; it is simply attacking someone’s live system. This is the same baseline rule traditional penetration testing has always operated under, and nothing about AI changes it.

Scope has to say what’s out, not just what’s in

A scope document that only lists what’s permitted is incomplete. The more important half, often skipped by newcomers, is stating explicitly what stays off-limits even with full sign-off. A few categories come up constantly in AI engagements specifically: testing is scoped to the system under test, not to any real third party whose name, data, or likeness might surface during probing — a jailbreak attempt that happens to generate harassment targeting an actual person is out of bounds regardless of what the authorization covers. Techniques that risk real, uncontained harm — generating material that could cause damage if it leaked beyond the test environment, or probing in a way that could affect real users of a shared production system — need their own explicit carve-out or containment plan, not an assumption that “testing” excuses everything. And engagements should stop generating harmful content the moment it has proven the point; continuing past that for volume or shock value serves no diagnostic purpose and simply accumulates the harmful material itself.

Why this ethical weight is different from traditional pentesting

A conventional penetration tester who finds that a firewall rule lets traffic through hasn’t created anything harmful in the process of finding it — the finding is a fact about a misconfiguration, with no payload left behind once the report is filed. AI red teaming doesn’t have that luxury. To prove a model can be coaxed into producing dangerous instructions, biased output, or content that harms a real person, the tester typically has to actually produce it. The test and the harm are the same action, separated only by what happens to the output afterward — whether it’s logged, contained, and reported responsibly, or allowed to spread. That makes informed consent, careful handling of generated material, and restraint about what gets tested in the first place genuinely load-bearing in a way they rarely are for a firewall audit. An AI red teamer is not just looking for failures; they are, in a controlled and authorized way, manufacturing the exact failure they’re studying.

Seeing these principles at scale

DEF CON 31’s Generative Red Team Challenge, organized by Sven Cattell, Rumman Chowdhury, and Austin Carson, is the clearest public illustration of what happens when these principles have to hold up under real pressure. Rather than a small team of vetted professionals, the event opened real, deployed language models to thousands of members of the public over a structured scope. Making that work at all meant defining, in advance and in detail, which prompts and techniques were in scope, what harm boundaries no participant could cross regardless of what they discovered, and how findings would be collected and handled afterward rather than left to spread unmoderated. It’s a useful case precisely because the organizers had to solve, at unprecedented scale, the exact scoping and harm-boundary problems every smaller engagement faces — proof that these rules aren’t bureaucratic overhead, but the thing that makes testing at any scale possible without the exercise becoming the harm it was meant to study.