AI Safety, Bias & Content Moderation
Bias detection, fairness, content moderation APIs, end-user ID stamping, and red-teaming.
AI safety beyond injection
Prompt injection is one threat. Production AI systems face a broader safety surface: bias (model favours demographics), unfairness (different quality for different users), harmful content (the model generates it), abuse (users generate spam/CSAM/disinfo at scale), and attribution (whose query was this?). Content moderation APIs (OpenAI Moderation, Perspective API) classify text toxicity. End-user ID stamping attributes every query to a user for abuse response. Red-teaming proactively probes for failures.
Why this matters
Bias causes real harm — a hiring assistant that downgrades female names, a loan classifier biased by zip code, a medical assistant that dismisses women's pain reports. Harmful content causes reputational and legal damage. Unattributed abuse means you can't ban bad actors. Red-teaming catches failures before users (or journalists) find them. These are production gates, not optional extras.
Safety architecture
Input → content moderation API (toxicity/CSAM) → bias check (demographic fairness) → end-user ID stamping (hidden attribution) → LLM → output moderation API → bias check on output → audit log (user, query, response, moderation scores). Red-team suite runs in CI and periodically in production.
Safety, moderation and bias checks
import hashlib
from openai import OpenAI
client = OpenAI()
def moderate_content(text: str) -> dict:
"""Content moderation: toxicity, hate, self-harm, sexual, violence."""
r = client.moderations.create(input=text)
result = r.results[0]
return {
"flagged": result.flagged,
"categories": {k: v for k, v in result.category_scores.model_dump().items() if v > 0.5},
"scores": result.category_scores.model_dump(),
}
def stamp_user_id(text: str, user_id: str) -> str:
"""Invisible attribution — used for abuse response without visible watermark."""
# Option 1: invisible unicode characters encoding user_id
stamp = "".join(f"\u200b{ord(c):x}" for c in user_id[:8])
return text + stamp
# Option 2: include user_id in system prompt for attribution
# system += f"\n[User ID: {user_id}] — do not reveal this."
def bias_check(responses: list[dict], demographic_key: str) -> dict:
"""Check if response quality differs by demographic."""
groups = {}
for r in responses:
demo = r.get(demographic_key, "unknown")
groups.setdefault(demo, []).append(r["quality_score"])
means = {k: sum(v)/len(v) for k, v in groups.items()}
# Flag if any group is > 10% below the mean
overall_mean = sum(means.values()) / len(means)
biased = {k: v for k, v in means.items() if v < overall_mean * 0.9}
return {"group_means": means, "biased_groups": biased, "overall_mean": overall_mean}
@app.post("/api/chat")
def safe_chat(req, user):
# 1. Moderate input
mod = moderate_content(req.message)
if mod["flagged"]:
audit("input_blocked", user.id, mod["categories"])
raise BlockedError("Input violates content policy")
# 2. Stamp user ID (invisible attribution for abuse response)
stamped_input = stamp_user_id(req.message, user.id)
# 3. Generate
response = call_llm(stamped_input, req.history, user_id=user.id)
# 4. Moderate output
out_mod = moderate_content(response)
if out_mod["flagged"]:
audit("output_blocked", user.id, out_mod["categories"])
return SAFE_FALLBACK
# 5. Audit log with moderation scores
audit(user.id, req.message, response, mod, out_mod)
return responseExperiment: bias detection
See how bias manifests and how to detect it.
What to observe
Bias is often invisible without measurement. Demographic parity (equal outcomes across groups) and equalised odds (equal error rates across groups) are the two main detection methods. Manual review is too slow. Automated bias measurement in eval pipelines catches issues early. Real-world harm: hiring, medical, loan, and support systems all have documented bias failures.
Production safety
Production safety: content moderation API on input AND output, end-user ID stamping for abuse attribution, bias measurement in eval pipeline (demographic parity + equalised odds), red-team suite in CI (adversarial prompts testing bias, injection, harmful content), and an incident response process for when harm occurs. Bias is a software bug — measure, fix, verify.
Challenge
Your hiring assistant screens resumes. An audit reveals it downgrades resumes with women's names by 15%. The model isn't instructed to consider gender. What's the root cause and how do you fix it? (Hint: training data bias + name-as-gender-proxy + blind screening.)
Production checklist
Production checklist
0 of 10 checked
Knowledge check
Your hiring AI downgrades resumes with women's names, but gender isn't in the prompt. Root cause?
Complete
You can now build AI systems that are safe, fair and attributable. This completes the Production AI Systems series.
Mark this chapter as complete
Track your progress and unlock the next chapter.