AI Red Teaming
Finding the ways a model, its prompts, and the systems around it can be broken — before someone else does. Twenty-four lessons from prompt injection and jailbreaks through model extraction, poisoning, and agentic exploits, to the testing methodology and defenses a real engagement runs on.
What is AI red teaming?Threat modeling for AI systemsThe OWASP Top 10 for LLM applicationsRules of engagement and ethicsPrompt injection, direct and indirectJailbreak techniquesSafety filter and guardrail bypassesMulti-turn and encoding attacksModel extraction and weight stealingModel inversion and membership inferenceData poisoning and backdoor triggersAdversarial examples and black-box attacksInsecure deserialization and RCE in model filesAPI and infrastructure security for model servingThe RAG and tool-calling attack surfaceAgentic AI security and excessive agencyBlack-box, grey-box, and white-box testingAutomated vs. manual red teamingLLM security testing tools — garak, Promptfoo, and PyRITBuilding a red-team labGuardrails and output filteringAdversarial training and robust designContinuous monitoring and detectionResponsible disclosure and reporting
What AI red teaming actually is
- What is AI red teaming?AI red teaming probes a deployed model's behavior for ways it can be made to misbehave — a discipline distinct from pentesting and ML evaluation.
- Threat modeling for AI systemsThreat modeling an AI system means naming its real assets and adversaries before testing it — the prompt itself is now an entry point.
- The OWASP Top 10 for LLM applicationsA tour of OWASP's ten named LLM vulnerability categories and why a shared, versioned checklist makes red-team scoping possible.
- Rules of engagement and ethicsScope, written authorization, and hard limits before testing an AI system — and why AI red-team ethics go beyond traditional pentest rules.
Attacking through the prompt
- Prompt injection, direct and indirectPrompt injection tricks a model into treating attacker text as instructions — either typed straight into chat or smuggled inside content it reads.
- Jailbreak techniquesJailbreaking targets a model's own trained refusal behavior directly, using role-play, split payloads, and many-shot framing to defeat it.
- Safety filter and guardrail bypassesGuardrails are a separate system bolted around a model — bypassing them, via tricks like Skeleton Key or Best-of-N, is a distinct problem from jailbreaking.
- Multi-turn and encoding attacksCrescendo escalates gradually across turns while encoding tricks disguise text — both exploit filters that judge surface text, not final intent.
Attacking the model itself
- Model extraction and weight stealingModel extraction clones a deployed model's accuracy or exact behavior purely from its API responses, without ever touching its weights.
- Model inversion and membership inferenceMembership inference tests whether a record was in a model's training set; model inversion goes further and reconstructs the data itself.
- Data poisoning and backdoor triggersPoisoning corrupts a model's training signal to skew its behavior, and backdoor triggers hide that corruption behind normal-looking accuracy.
- Adversarial examples and black-box attacksAdversarial examples fool a model with inputs that look unchanged to a human, and black-box attacks craft them using only output labels.
Attacking the system around the model
- Insecure deserialization and RCE in model filesPickle-format model checkpoints can embed executable code, so loading a "model" can secretly mean running an attacker's instructions.
- API and infrastructure security for model servingShadowRay and the TorchServe exposure show that ML-serving infrastructure gets ordinary infrastructure bugs when it skips standard API hardening.
- The RAG and tool-calling attack surfaceIn RAG and tool-calling systems, retrieved documents and tool outputs are untrusted input channels, not just the user's own prompt.
- Agentic AI security and excessive agencyWhen an injected instruction can make an autonomous agent act, not just answer, one compromised integration can cascade into real-world harm.
Running a red team engagement
- Black-box, grey-box, and white-box testingWhat a red teamer can see — public interface only, partial internals, or full source — determines what an AI red-team engagement can actually find.
- Automated vs. manual red teamingAutomated tooling finds known failure classes at scale; manual testing finds the novel ones — real engagements need both, not one instead of the other.
- LLM security testing tools — garak, Promptfoo, and PyRITA practical tour of the field's most-used open-source LLM red-teaming tools and which one to reach for first, depending on the job.
- Building a red-team labWhy practicing AI red teaming requires a dedicated, legal target, and a concrete checklist for standing up your own practice environment.
Defending and reporting
- Guardrails and output filteringGuardrails wrap a model in classifiers and rules that screen input and output — a defense layer, not a fix to the model itself.
- Adversarial training and robust designBaking resistance into a model through adversarial training and constitutional classifiers is the deeper counterpart to a bolted-on guardrail.
- Continuous monitoring and detectionRed teaming and guardrails happen before launch; production monitoring is how you catch drift and live attacks after launch.
- Responsible disclosure and reportingA finding that's never reported, or reported badly, has the same effect as no testing — this lesson covers doing it right.