AI content moderation
The classifier and policy layer that filters input to and output from an LLM agent — blocks unsafe categories (CSAM, self-harm, malware), enforces brand voice, and flags PII.
Every production agent in 2026 sits behind a moderation layer. Inbound, it classifies user input and blocks or rewrites unsafe requests. Outbound, it scans model output and blocks responses that violate policy. Categories typically include violence, sexual content, self-harm, hate, illegal activity, PII, and prompt injection.
Major options: OpenAI Moderation API, Anthropic's in-model safety, Google Perspective, Microsoft Azure Content Safety, and open-source classifiers like Llama Guard. Most production systems run two layers — a fast classifier for obvious cases and a slower LLM judge for nuance.
The hard part is calibration. Over-moderation kills product usability. Under-moderation creates risk and PR liability. Tune to your audience, your jurisdiction, and your customer commitments — and revisit quarterly.
Frequently asked
Do I need content moderation if I use a frontier model with built-in safety?+
Yes — built-in safety covers the model's training-time refusals. Production moderation also enforces YOUR policies, blocks tenant-specific PII, and catches prompt injection. They layer.
What is the cheapest content moderation that ships?+
OpenAI Moderation API is free and covers the basics. Llama Guard self-hosted is free at scale. Layer either over your own deny-list for tenant-specific terms.