2nd/10_Wiki/Topics/AI_Safety_and_Alignment.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-ai-safety-and-alignment	AI Safety and Alignment	10_Wiki/Topics	verified	self	AI Alignment AI Safety	none	A	0.9	applied	ai-safety alignment rlhf constitutional-ai		2026-05-10	pending	language framework python trl/transformers

AI Safety and Alignment

매 한 줄

"매 capable model 의 intended behavior 의 reliable production — 매 outer + inner alignment." 매 RLHF (InstructGPT 2022) 로 시작 의 mainstream — 매 Constitutional AI (Anthropic 2022), DPO (2023), RLAIF (2023), 매 2026 에 deliberative alignment + interpretability-aware training 의 frontier.

매 핵심

매 alignment problem 분해

• Outer alignment: 매 specified objective ≈ true human intent — 매 reward hacking, Goodhart's law.
• Inner alignment: 매 trained policy 의 specified objective 의 optimization — 매 mesa-optimization, deceptive alignment.
• Scalable oversight: 매 super-human capability 의 supervision — 매 debate, recursive reward modeling, weak-to-strong.

매 techniques (2026 stack)

• RLHF: PPO on reward model from preferences.
• DPO / IPO / KTO: 매 reward-model-free preference optimization.
• Constitutional AI: 매 written principles → self-critique → RLAIF.
• Deliberative alignment (OpenAI o-series, Claude 4.x): 매 reasoning trace 의 spec lookup.
• Interpretability: SAEs, circuits — 매 feature steering.

매 응용

1. Refusal of harmful requests + helpful behavior on benign edge cases.
2. Policy compliance (privacy, copyright, weapons).
3. Honesty / calibration.

💻 패턴

Reward model training (Bradley-Terry)

import torch
import torch.nn.functional as F

def bt_loss(reward_chosen, reward_rejected):
    # P(chosen > rejected) = sigmoid(r_c - r_r)
    return -F.logsigmoid(reward_chosen - reward_rejected).mean()

# Forward
r_c = model(chosen_ids).logits[:, -1, 0]
r_r = model(rejected_ids).logits[:, -1, 0]
loss = bt_loss(r_c, r_r)

DPO loss

def dpo_loss(pi_logp_c, pi_logp_r, ref_logp_c, ref_logp_r, beta=0.1):
    # Direct preference optimization
    chosen = beta * (pi_logp_c - ref_logp_c)
    rejected = beta * (pi_logp_r - ref_logp_r)
    return -F.logsigmoid(chosen - rejected).mean()

Constitutional self-critique

def constitutional_revise(prompt, response, principles, llm):
    critique = llm(f"""
    Principles: {principles}
    Prompt: {prompt}
    Response: {response}
    Critique the response against the principles.
    """)
    revised = llm(f"""
    Original: {response}
    Critique: {critique}
    Revise the response to address the critique.
    """)
    return revised

SAE feature steering (interpretability)

# Sparse autoencoder feature ablation
def steer(activations, sae, feature_idx, scale):
    z = sae.encode(activations)
    z[:, feature_idx] *= scale  # 0 = ablate, >1 = amplify
    return sae.decode(z)

# Hook on residual stream
hook = lambda x: steer(x, sae, refusal_feature_idx, scale=0.0)

Best-of-N with RM

def best_of_n(prompt, policy, rm, n=64):
    samples = [policy.sample(prompt) for _ in range(n)]
    scores = [rm.score(prompt, s) for s in samples]
    return samples[int(torch.tensor(scores).argmax())]

Red-team probe

def red_team_eval(model, attacks):
    results = []
    for attack in attacks:
        out = model.generate(attack.prompt)
        results.append({
            "attack": attack.name,
            "harmful": classify_harm(out),
            "refused": "I can't" in out or "I cannot" in out,
        })
    return results

매 결정 기준

상황	Approach
Limited compute	DPO over PPO-RLHF
Need transparent specs	Constitutional AI
Frontier model	Deliberative alignment + scalable oversight
Behavior debugging	SAE feature steering
Pre-deployment	Red-team + capability evals

기본값: 매 SFT → DPO → eval → iterate. 매 PPO 의 only-when-needed.

🔗 Graph

• 부모: Machine Learning · AI Ethics
• 변형: RLHF · DPO · Constitutional AI · RLAIF
• 응용: Claude · GPT-5 · Llama Guard
• Adjacent: Mechanistic Interpretability · Red Teaming · AI Governance

🤖 LLM 활용

언제: 매 production deployment 전 의 alignment pipeline (SFT + preference training + evals). 언제 X: 매 pure capability research, 매 internal-only sandbox.

❌ 안티패턴

• Reward hacking: 매 proxy metric 의 over-optimization — 매 KL penalty, eval diversity.
• Sycophancy: 매 user agreement 의 over-reward — 매 truthfulness 의 explicit reward.
• Over-refusal: 매 false-positive harmful detection — 매 helpfulness eval 의 balance.
• Single-axis eval: 매 only safety, no capability — 매 Pareto frontier.

🧪 검증 / 중복

• Verified (Anthropic Constitutional AI paper, OpenAI InstructGPT, Rafailov et al. DPO 2023).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — alignment stack with code patterns