Red teaming
A structured testing practice where adversaries actively try to break an AI system — finding jailbreaks, hallucinations, harmful outputs, or unsafe tool calls before attackers do.
Red teaming an AI system is the safety equivalent of a security pen test. A red team (internal or external) attacks the model with prompt injections, jailbreaks, edge-case inputs, and adversarial scenarios. The output is a list of failures the deployment team must address before launch.
For agent operators, red teaming matters more than for chat-only systems. An agent has tools, memory, and external reach — every additional surface is an attack vector. The questions a red team asks: can a user exfiltrate the system prompt? Can a tool be called with adversarial inputs? Can the agent be socially engineered to ignore its instructions?
In 2026 red teaming is increasingly automated. Tools like Promptfoo, Garak, and AI safety platforms run thousands of adversarial probes against agents in CI. Manual red teaming still finds the edge cases automation misses, but automated red teaming catches 80% of issues at a fraction of the cost.
Frequently asked
How often should I red-team my agent?+
At minimum: before every major release. Production-grade stacks run automated red-team probes in CI on every change and schedule manual deep red teaming quarterly.
Should I use an external red team?+
For high-stakes deployments (consumer-facing, regulated industries, agentic workflows touching money or compliance), yes. External teams find the failure modes your internal team has trained themselves not to see.