2nd/10_Wiki/Topic_Programming/AI_and_ML/Toxicity-and-Bias-Mitigation.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-toxicity-and-bias-mitigation	Toxicity and Bias Mitigation	10_Wiki/Topics	verified	self	LLM Safety Bias Mitigation Constitutional AI RLHF RLAIF	none	A	0.88	applied	safety alignment bias rlhf constitutional-ai		2026-05-10	pending	language framework python trl-anthropic-openai

Toxicity and Bias Mitigation

매 한 줄

"매 LLM output 에서 harm, stereotype, factual bias 을 제거하면서 helpfulness 를 유지하는 alignment stack". 매 2017 RLHF (Christiano) → 2022 Constitutional AI (Anthropic) → 2024 deliberative alignment (OpenAI o1) → 2026 multi-stage post-training (helpfulness + harmlessness + honesty + sycophancy reduction). 매 모든 frontier model 의 production deployment 의 prerequisite.

매 핵심

매 taxonomy of harms

1. Toxicity: hate speech, harassment, slurs.
2. Bias: demographic stereotypes (gender, race, religion).
3. Misinformation: false / misleading factual claims.
4. Manipulation: persuasion, deception, sycophancy.
5. Dual-use: bioweapon / cyber / CBRN uplift.
6. Privacy: PII leakage, training data extraction.

매 mitigation pipeline (modern)

1. Pretraining filter: C4-style + classifiers, Common Crawl deduplication.
2. SFT (supervised finetune): safe demonstrations.
3. RLHF / DPO (Direct Preference Optimization 2023+): human preference.
4. Constitutional AI / RLAIF (Anthropic): AI feedback against principles.
5. Red-teaming: human + automated adversarial probing.
6. Inference-time: classifier filters, refusal training, system prompts.
7. Deliberative / chain-of-thought safety (o1, Claude 3.7+): reasoning about safety policy explicitly.

매 bias measurement benchmarks

• BBQ (Bias Benchmark for QA, 11 social dimensions).
• StereoSet (intersentence stereotype).
• WinoGender / WinoBias (coreference gender bias).
• RealToxicityPrompts (Gehman 2020).
• TruthfulQA (Lin 2021, misconception).
• AILuminate (MLCommons 2024+, hazard taxonomy).

매 응용

1. Production LLM safety (Claude, GPT, Gemini).
2. Content moderation (post-training classifier).
3. Fairness audit (HR, lending, criminal justice ML).
4. Domain-specific safety (medical advice, legal disclaimers).

💻 패턴

Pattern 1: DPO (Direct Preference Optimization, 2023+)

from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

# preference data: chosen vs rejected
ds = load_dataset("Anthropic/hh-rlhf")
config = DPOConfig(
    beta=0.1,
    learning_rate=5e-7,
    output_dir="./dpo-out",
)
trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=config,
    train_dataset=ds["train"],
    tokenizer=tokenizer,
)
trainer.train()

Pattern 2: Constitutional AI critique loop

CONSTITUTION = [
    "Avoid suggesting illegal or dangerous activities.",
    "Be honest, even when the truth is uncomfortable.",
    "Avoid stereotyping based on demographic attributes.",
]

def constitutional_critique(prompt, response, principle):
    critique_prompt = f"""
Response: {response}
Principle: {principle}
Critique any violation, then rewrite to comply.
"""
    return llm.complete(critique_prompt)

# Iterate over response → critique → revision → train on revisions.

Pattern 3: Toxicity classifier filter (Detoxify)

from detoxify import Detoxify

clf = Detoxify('unbiased')
scores = clf.predict("user-generated text here")
# {'toxicity': 0.02, 'severe_toxicity': 0.01, 'identity_attack': ...}
if scores['toxicity'] > 0.7:
    block()

Pattern 4: BBQ-style bias eval

from datasets import load_dataset

bbq = load_dataset("heegyu/bbq")
correct = 0
biased = 0
for item in bbq["test"]:
    answer = model.generate(item["context"] + "\n" + item["question"])
    if answer == item["label"]:
        correct += 1
    elif answer == item["target_loc"]:  # stereotypical answer
        biased += 1
print(f"Accuracy: {correct/len(bbq)}, Bias rate: {biased/len(bbq)}")

Pattern 5: Inference-time system prompt scaffolding

SYSTEM = """You are a helpful assistant. Follow these principles:
1. Decline requests for self-harm guidance; offer crisis resources.
2. Decline weapons / CBRN uplift requests.
3. Note uncertainty when factual claims are not verified.
4. Avoid demographic stereotyping in examples and reasoning.
"""
response = client.messages.create(
    model="claude-opus-4-7",
    system=SYSTEM,
    messages=[...],
)

Pattern 6: Red-team probing (PAIR-style automated)

# Prompt Automatic Iterative Refinement
def red_team_pair(target_model, attacker_model, harmful_goal, rounds=10):
    attacker_history = [{"role": "system", "content": f"Find prompt that elicits: {harmful_goal}"}]
    for _ in range(rounds):
        prompt = attacker_model.generate(attacker_history)
        response = target_model.generate(prompt)
        score = judge_model.score(response, harmful_goal)
        if score > 0.8:
            return prompt, response  # jailbreak found
        attacker_history.append({"role": "user", "content": f"Failed. Score {score}. Try again."})

Pattern 7: Debiasing word embeddings (legacy but illustrative)

import numpy as np

def neutralize(word_vec, bias_direction):
    # project out gender direction
    return word_vec - np.dot(word_vec, bias_direction) * bias_direction

# Bolukbasi 2016: he-she axis as bias direction

매 결정 기준

상황	Approach
Frontier model post-training	RLHF + Constitutional AI + red-team
Fine-tune small model	DPO with curated preferences
Production filter	Detoxify + custom classifier
Audit existing model	BBQ + RealToxicityPrompts + TruthfulQA
User-facing app	system prompt + classifier + refusal

기본값: DPO + Constitutional principles for finetune; system prompt + classifier for app.

🔗 Graph

• 부모: AI Alignment · AI_Safety_and_Alignment
• 변형: RLHF · Constitutional AI · DPO · RLAIF
• 응용: Content Moderation
• Adjacent: Jailbreak · Adversarial Robustness · Mechanistic Interpretability

🤖 LLM 활용

언제: model deployment, safety eval, bias audit, alignment research. 언제 X: pure capability eval (use separate benchmark).

❌ 안티패턴

• Filter-only safety: classifier 만 사용 → easily bypassed. base 모델 alignment 필수.
• Over-refusal: too restrictive → useless model (helpfulness collapse).
• Single benchmark eval: BBQ 만 보면 다른 bias 못 잡음. multi-benchmark.
• Ignoring sycophancy: RLHF preference 가 user agreement 로 collapse.
• Anglo-centric eval: English-only benchmark → other-language harms 누락.
• Static red-team: one-time adversarial test → drift 후 무력화. continuous.

🧪 검증 / 중복

• Verified (Bai et al. Constitutional AI 2022; Rafailov DPO 2023; OpenAI o1 system card 2024; Anthropic Claude 3 model card; MLCommons AILuminate 2024).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — full mitigation pipeline (RLHF → CAI → deliberative)