[G1-Sync] Manual knowledge update

2026-05-10 22:08:15 +09:00
parent 21ac3ed255
commit 504fd5fb42
3011 changed files with 380280 additions and 206977 deletions
@@ -0,0 +1,360 @@
+---
+id: ai-rlhf-dpo-basics
+title: RLHF / DPO — alignment 기초
+category: Coding
+status: draft
+source_trust_level: B
+verification_status: conceptual
+created_at: 2026-05-09
+updated_at: 2026-05-09
+tags: [ai, alignment, rlhf, vibe-coding]
+tech_stack: { language: "Python", applicable_to: ["AI"] }
+applied_in: []
+aliases: [RLHF, DPO, alignment, reward model, preference learning, instruction tuning, RLAIF]
+---
+
+# RLHF / DPO Basics
+
+> Pretrained LLM 가 helpful + harmless 안 됨. **Instruction tuning + RLHF / DPO**. Human feedback 이 model 다듬기. Modern LLM (GPT, Claude, Llama) 의 핵심.
+
+## 📖 핵심 개념
+- Pretrain: 다음 token 예측 (raw text).
+- SFT (Supervised Fine-Tune): instruction → response.
+- Reward model: 사람 비교 → score.
+- RLHF: PPO 가 model 가 reward 높임.
+- DPO: reward model 없이 직접 (modern).
+
+## 💻 코드 패턴
+
+### Stage 1: Pretraining
+```
+Raw text (web, books, code) → predict next token.
+
+Output: Base model (GPT, Llama).
+- 큰 (7B-70B+)
+- 다음 token 잘.
+- "Helpful" 안 — completion 만.
+```
+
+### Stage 2: SFT (Supervised)
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from datasets import load_dataset
+
+model = AutoModelForCausalLM.from_pretrained('llama-7b')
+tokenizer = AutoTokenizer.from_pretrained('llama-7b')
+
+dataset = load_dataset('instruction-data')
+# {'instruction': 'Translate to French', 'input': 'Hello', 'output': 'Bonjour'}
+
+# Format
+def format(ex):
+    return f"### Instruction: {ex['instruction']}\n### Input: {ex['input']}\n### Response: {ex['output']}"
+
+# Train (가장 simple)
+trainer.train()
+```
+
+→ Model 가 "instruction → response" 형식 학습.
+
+### Stage 3: Reward model
+```python
+# Human rate: response A vs B.
+# Dataset: {prompt, response_a, response_b, preferred: 'a'}
+
+# Reward model (regression)
+reward_model = AutoModelForSequenceClassification.from_pretrained('llama-7b', num_labels=1)
+
+# Train: chosen reward > rejected reward
+def loss_fn(chosen_reward, rejected_reward):
+    return -F.logsigmoid(chosen_reward - rejected_reward).mean()
+```
+
+→ Model 가 prompt + response → score 반환.
+
+### Stage 4: PPO (RLHF)
+```python
+# Loop:
+# 1. Sample prompt
+# 2. SFT model 가 response 생성
+# 3. Reward model 가 score
+# 4. PPO 가 model 가 score 높이도록 update
+# 5. KL penalty (SFT 너무 변경 X)
+
+from trl import PPOTrainer
+
+trainer = PPOTrainer(
+    model=sft_model,
+    ref_model=sft_model_frozen,  # KL ref
+    reward_model=reward_model,
+    tokenizer=tokenizer,
+)
+trainer.train()
+```
+
+→ 매우 복잡 + 불안정. 큰 GPU.
+
+### DPO (Direct Preference Optimization)
+```python
+# RLHF 의 simpler alternative.
+# Reward model 없이 — preference data 직접.
+
+from trl import DPOTrainer
+
+trainer = DPOTrainer(
+    model=sft_model,
+    ref_model=sft_model_frozen,
+    args=TrainingArguments(...),
+    train_dataset=preference_dataset,
+    beta=0.1,
+)
+trainer.train()
+```
+
+→ DPO = SFT + KL + preference loss. Simple + 안정.
+
+### DPO loss (직관)
+```python
+# π = 새 model probability
+# π_ref = SFT (frozen) probability
+
+loss = -log(σ(β * (
+    log(π(chosen) / π_ref(chosen))
+    - log(π(rejected) / π_ref(rejected))
+)))
+
+# β = KL strength (typically 0.1 - 0.5)
+```
+
+→ 새 model 가 chosen 가 rejected 보다 더 likely.
+
+### Preference dataset
+```jsonl
+{"prompt": "Tell me a joke", "chosen": "Why did the chicken...", "rejected": "I don't know"}
+{"prompt": "Explain DPO", "chosen": "DPO is a method...", "rejected": "It's complex"}
+```
+
+→ 인간 / AI 가 pair 생성.
+
+### Constitutional AI (Anthropic)
+```
+Step 1: SFT.
+Step 2: AI critic 가 자체 review (not human).
+Step 3: AI 가 better response 생성.
+Step 4: DPO / RLHF on AI-generated preferences.
+
+→ "RLAIF" (RL from AI Feedback).
+```
+
+→ 사람 cost 줄임. Scalable.
+
+### Instruction dataset
+```
+- Alpaca (52k self-instruct)
+- Dolly (15k human)
+- OpenAssistant (200k+)
+- ShareGPT (real conversation)
+
+→ Public dataset 가 starter.
+```
+
+### LoRA (parameter-efficient)
+```python
+from peft import LoraConfig, get_peft_model
+
+config = LoraConfig(
+    r=16, lora_alpha=32,
+    target_modules=['q_proj', 'v_proj'],
+    lora_dropout=0.05,
+)
+model = get_peft_model(base_model, config)
+# → 1% parameter 만 train.
+
+trainer.train()
+```
+
+→ Full fine-tune (큰 GPU) 의 alternative.
+
+### QLoRA
+```python
+from transformers import BitsAndBytesConfig
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type='nf4',
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    'llama-70b',
+    quantization_config=bnb_config,
+)
+```
+
+→ 70B model 가 single A100 (40GB) 에서 LoRA train.
+
+### Eval
+```python
+# Hellaswag, MMLU, TruthfulQA, ARC
+import lm_eval
+lm_eval.simple_evaluate(model=model, tasks=['mmlu'])
+```
+
+→ Benchmark suite.
+
+### Human eval
+```
+Annotators 가 model A vs B 비교.
+- Helpfulness
+- Honesty
+- Harmlessness
+
+→ "Win rate" — A 가 B 이긴 %.
+```
+
+### Reward hacking
+```
+RLHF model 가 reward 높이는 "trick" 발견.
+- 매 답 가 길게 (사람 가 long = better 가정).
+- "Sure!" / "Of course!" 시작.
+- 안 답 (refuse) 가 안전.
+
+→ Reward model 가 proxy. Calibration 어려움.
+```
+
+### KL penalty
+```
+KL(π || π_ref) = SFT 와 차이.
+
+너무 큰 = SFT 잊음 (잘 한 거 잃음).
+너무 작은 = update 안 됨.
+
+β = 0.1 - 0.5 가 typical.
+```
+
+### IPO / KTO / ORPO (DPO alternatives)
+```
+IPO (Identity PO): KL divergence 더 안정.
+KTO (Kahneman-Tversky): 단일 답 만 (preference X).
+ORPO: SFT + DPO combined (1 stage).
+
+→ DPO 의 변형. 매년 새.
+```
+
+### Closed vs open model
+```
+Closed (GPT-4, Claude): RLHF + 비밀.
+Open (Llama, Mistral): SFT + DPO 가 흔한.
+
+→ Open = "instruct" version.
+Llama-3-Instruct = Llama-3 + SFT + DPO + ...
+```
+
+### Personal fine-tune
+```
+Use case:
+- Domain-specific (legal, medical)
+- Style (brand voice)
+- Format (specific JSON)
+
+Data:
+- 100-1000 example 가 충분 (LoRA).
+- 10000+ 가 strong.
+```
+
+### Cost
+```
+LoRA SFT (7B, 1k examples): $5-50.
+QLoRA (70B): $50-500.
+Full fine-tune: $1000-100k.
+DPO + LoRA: 2-3x SFT.
+
+→ LoRA 가 99% use case.
+```
+
+### When NOT fine-tune?
+```
+✓ Generic task = prompt engineering.
+✓ Few-shot 가 OK = 데이터 적음.
+✓ Domain 매우 specific = fine-tune.
+
+Rule: prompt + RAG 시도. 안 되면 fine-tune.
+```
+
+### Tools
+```
+- Hugging Face TRL (DPO, PPO)
+- Axolotl (config-based fine-tune)
+- LLaMA-Factory
+- Unsloth (빠른 LoRA)
+- OpenAI / Anthropic fine-tune API (managed)
+```
+
+### Hugging Face TRL
+```python
+from trl import SFTTrainer, DPOTrainer
+
+# SFT
+trainer = SFTTrainer(model=model, train_dataset=ds, dataset_text_field='text')
+trainer.train()
+
+# DPO 
+trainer = DPOTrainer(model=sft, ref_model=sft_frozen, train_dataset=pref_ds, beta=0.1)
+trainer.train()
+```
+
+### Axolotl (config-based)
+```yaml
+# config.yml
+base_model: meta-llama/Llama-3-8B
+adapter: lora
+lora_r: 16
+datasets:
+  - path: alpaca.jsonl
+    type: alpaca
+sequence_len: 2048
+```
+
+```bash
+axolotl train config.yml
+```
+
+### Multi-modal alignment
+```
+Vision-language model (LLaVA, Qwen-VL):
+- Image-text pair train
+- DPO / RLHF + image
+
+→ 매우 복잡. Open SoTA 가 부족.
+```
+
+## 🤔 의사결정 기준
+| 상황 | 추천 |
+|---|---|
+| Generic task | Prompt + RAG |
+| Specific domain | LoRA SFT |
+| 정밀 alignment | DPO (after SFT) |
+| 고전 RLHF | PPO (복잡) |
+| Cost-sensitive | LoRA + QLoRA |
+| Modern simple | DPO (direct) |
+| 큰 cluster | Full fine-tune |
+
+## ❌ 안티패턴
+- **SFT 없이 RLHF**: SFT 가 baseline.
+- **RLHF 가 모든 거 fix**: prompt + RAG 먼저.
+- **Reward hacking**: 매 hack signal 도.
+- **KL 무시**: SFT 잊음.
+- **데이터 적음**: overfit.
+- **Eval 없음**: improvement 모름.
+- **Public model + private data leak**: privacy.
+
+## 🤖 LLM 활용 힌트
+- DPO 가 RLHF 보다 simple + 안정.
+- LoRA + QLoRA 가 cost answer.
+- Constitutional AI / RLAIF 가 Anthropic 의 답.
+- Prompt + RAG 시도 → 안 되면 fine-tune.
+
+## 🔗 관련 문서
+- [[AI_Fine_Tuning_vs_Prompting]]
+- [[AI_LLM_Eval_Patterns]]
+- [[AI_Safety_Patterns]]