docs(10_Wiki): 위키 전체 재구성 — Topic_* 폴더를 4개 카테고리로 통합 + 대규모 중복 제거

Topic_Agent/Topic_Blog/Topics/Topics_Biz/Topics_Meeting/Topics_Rag의 마크다운 지식 문서를 Topic_General/Topic_Programming/Topic_Graphic/Topic_Business 4개 카테고리로 재분류. - 중복 제거: frontmatter의 status:duplicate/merged + duplicate_of/redirect_to 필드로 자기 자신을 중복으로 선언한 리다이렉트 stub 1032개 제거, 완전 동일 내용 파일 472개 제거, 동일 파일명·다른 내용 충돌 시 더 큰(완전한) 버전만 유지(162개 제거) — 총 1639개 중복 제거. - 분류: 폴더 단위로 명확한 항목(AI_and_ML/Coding/Architecture 등 → Programming, Comfyui/Visual_Effects → Graphic, Topics_Biz/Topics_Meeting/사업 등 → Business, Poetic_Blog_Writing/창의성/Game_Design 등 → General)은 폴더 우선순위로, 나머지 혼재 폴더(Topic_Agent/Topic_Blog/Topics 루트/Thinking & Reasoning/Other/UI_UX_Assets)는 title/tags 키워드 스코어링으로 파일 단위 분류(불명확한 경우 General로 폴백). 원본 폴더명은 "From_*" 서브폴더로 보존해 추적 가능성 유지. - 최종 배치: Programming 2784 / General 1608 / Graphic 285 / Business 249 = 4926개 문서. - 에이전트 운영 상태(.astra/.agent/.obsidian/sessions/memory/_company/docs/lessons/_shared/src)는 지식 콘텐츠가 아니므로 재분류 대상에서 제외하고 원위치 유지. - Topics/Topic_email(상위 보호 폴더 Topic_email과 파일명 100% 중복) 삭제 — 보호 폴더 자체는 미변경. - 완전히 비게 된 Topic_Agent/Topic_Blog/Topics_Biz/Topics_Rag 폴더 제거.
2026-07-05 00:33:48 +09:00
parent 1cfd3bbb56
commit 9148c358d0
6455 changed files with 1 additions and 86875 deletions
@@ -0,0 +1,281 @@
+---
+id: wiki-2026-0508-dpo
+title: DPO (Direct Preference Optimization)
+category: 10_Wiki/Topics
+status: verified
+canonical_id: self
+aliases: [DPO, ORPO, SimPO, IPO, KTO, preference learning, RLHF alternative, alignment]
+duplicate_of: none
+source_trust_level: A
+confidence_score: 0.95
+verification_status: applied
+tags: [llm, alignment, dpo, rlhf, preference-optimization, ppo, fine-tuning, trl]
+raw_sources: []
+last_reinforced: 2026-05-10
+github_commit: pending
+tech_stack:
+  language: Python
+  framework: TRL / DeepSpeed / Axolotl
+---
+
+# DPO (Direct Preference Optimization)
+
+## 매 한 줄
+> **"매 reward model 의 X — 매 preference pair 의 직접 학습"**. Rafailov et al. 2023. 매 RLHF 의 simpler + 매 stable + 매 effective alternative. 매 modern variant: ORPO, SimPO, KTO. 매 Llama-3, Tülu, 매 most open model 의 standard.
+
+## 매 핵심
+
+### 매 vs RLHF
+| 측면 | RLHF (PPO) | DPO |
+|---|---|---|
+| Reward model | Required | None |
+| Stages | 2 (RM → PPO) | 1 |
+| Stability | Hard | Stable |
+| Hyperparameter | Many | Few |
+| Compute | High | Lower |
+| Quality | Strong | Comparable |
+
+### 매 DPO loss
+$$L_{DPO} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)$$
+
+- 매 y_w = 매 winner (preferred).
+- 매 y_l = 매 loser.
+- 매 β = 매 KL coefficient.
+
+### 매 derivation insight
+- 매 PPO objective 의 closed-form: 매 reward = 매 log-probability ratio.
+- 매 reward model 의 implicitly learned by 매 policy.
+
+### 매 variant
+
+#### ORPO (Odds Ratio PO)
+- 매 reference model 의 free.
+- 매 SFT + 매 preference 의 single stage.
+
+#### SimPO (Simple PO)
+- 매 reference 의 free.
+- 매 length-normalize.
+
+#### IPO (Identity PO)
+- 매 DPO 의 deterministic preference 의 fix.
+
+#### KTO (Kahneman-Tversky Optimization)
+- 매 binary feedback (good / bad) — 매 pair 없이.
+- 매 prospect theory inspired.
+
+#### SLiC-HF
+- 매 sequence-level contrastive.
+
+#### CPO (Contrastive PO)
+- 매 reference-free + length-aware.
+
+### 매 data
+- **HH-RLHF** (Anthropic): 매 helpful + harmless.
+- **UltraFeedback**.
+- **Nectar**.
+- **PKU-Beaver**.
+- **Tülu**.
+
+### 매 modern stack
+- **TRL** (HuggingFace): 매 DPOTrainer, ORPOTrainer.
+- **Axolotl**: 매 config-driven.
+- **DeepSpeed**.
+- **Unsloth**: 매 fast LoRA.
+
+### 매 응용
+1. **LLM alignment**: 매 helpful + harmless.
+2. **Fine-tune on preference**: 매 customer service tone.
+3. **Code style**: 매 specific convention.
+4. **Refusal calibration**: 매 over-refusal 의 reduce.
+
+### 매 limitation
+- 매 reference model 의 quality 의 critical.
+- 매 length bias (longer 의 win 의 tend).
+- 매 over-conservative (mode-seeking).
+- 매 verifier-based (RLVR) 가 매 specific 의 better.
+
+## 💻 패턴
+
+### DPOTrainer (TRL)
+```python
+from trl import DPOTrainer, DPOConfig
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from datasets import load_dataset
+
+model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-v0.1')
+ref_model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-v0.1')
+tokenizer = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1')
+
+# 매 dataset format: {prompt, chosen, rejected}
+dataset = load_dataset('trl-lib/ultrafeedback_binarized')
+
+config = DPOConfig(
+    output_dir='./dpo-mistral',
+    per_device_train_batch_size=4,
+    gradient_accumulation_steps=4,
+    learning_rate=5e-7,
+    num_train_epochs=1,
+    beta=0.1,  # 매 KL coefficient
+    max_length=2048,
+    max_prompt_length=512,
+)
+
+trainer = DPOTrainer(
+    model=model,
+    ref_model=ref_model,
+    args=config,
+    train_dataset=dataset['train'],
+    tokenizer=tokenizer,
+)
+trainer.train()
+```
+
+### Manual DPO loss
+```python
+import torch
+import torch.nn.functional as F
+
+def dpo_loss(policy_logp_chosen, policy_logp_rejected,
+             ref_logp_chosen, ref_logp_rejected, beta=0.1):
+    pi_logratio = policy_logp_chosen - policy_logp_rejected
+    ref_logratio = ref_logp_chosen - ref_logp_rejected
+    return -F.logsigmoid(beta * (pi_logratio - ref_logratio)).mean()
+
+# 매 logp = log P(y | x)
+```
+
+### ORPO (no reference)
+```python
+from trl import ORPOTrainer, ORPOConfig
+
+config = ORPOConfig(
+    output_dir='./orpo-mistral',
+    learning_rate=8e-6,
+    beta=0.1,  # 매 odds ratio coefficient
+)
+
+# 매 매 SFT + preference 의 single stage
+trainer = ORPOTrainer(model=model, args=config, train_dataset=dataset, tokenizer=tokenizer)
+trainer.train()
+```
+
+### KTO (binary feedback)
+```python
+from trl import KTOTrainer, KTOConfig
+
+# 매 dataset format: {prompt, completion, label: True / False}
+config = KTOConfig(
+    output_dir='./kto',
+    beta=0.1,
+    desirable_weight=1.0,
+    undesirable_weight=1.0,
+)
+
+trainer = KTOTrainer(model=model, ref_model=ref_model, args=config, train_dataset=dataset)
+trainer.train()
+```
+
+### LoRA + DPO (efficient)
+```python
+from peft import LoraConfig
+
+peft_config = LoraConfig(
+    r=16, lora_alpha=32,
+    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
+    lora_dropout=0.05,
+    task_type='CAUSAL_LM',
+)
+
+config = DPOConfig(...)
+
+trainer = DPOTrainer(
+    model=model,
+    args=config,
+    train_dataset=dataset,
+    tokenizer=tokenizer,
+    peft_config=peft_config,  # 매 LoRA
+)
+```
+
+### Preference data generation (synthetic)
+```python
+def generate_preference_pair(prompt, model_a, model_b, judge):
+    response_a = model_a.generate(prompt)
+    response_b = model_b.generate(prompt)
+    
+    chosen, rejected = judge(prompt, response_a, response_b)
+    return {'prompt': prompt, 'chosen': chosen, 'rejected': rejected}
+
+# 매 LLM-as-judge
+def gpt4_judge(prompt, a, b):
+    judgment = gpt4.generate(f"""Which response is better?
+Prompt: {prompt}
+A: {a}
+B: {b}
+
+Reply: A or B""")
+    return (a, b) if 'A' in judgment else (b, a)
+```
+
+### Length bias mitigation
+```python
+# 매 SimPO: 매 length-normalized
+def simpo_loss(logp_chosen, logp_rejected, len_chosen, len_rejected, beta=0.5, gamma=1.0):
+    pi_chosen = logp_chosen / len_chosen  # 매 normalize
+    pi_rejected = logp_rejected / len_rejected
+    return -F.logsigmoid(beta * (pi_chosen - pi_rejected) - gamma).mean()
+```
+
+### Eval (preference accuracy)
+```python
+def eval_preference_accuracy(model, eval_set):
+    correct = 0
+    for ex in eval_set:
+        logp_chosen = model.compute_logp(ex['chosen'])
+        logp_rejected = model.compute_logp(ex['rejected'])
+        if logp_chosen > logp_rejected: correct += 1
+    return correct / len(eval_set)
+```
+
+## 매 결정 기준
+| 상황 | Method |
+|---|---|
+| Standard alignment | DPO |
+| Single-stage | ORPO |
+| Length-sensitive | SimPO |
+| Binary feedback | KTO |
+| Verifiable reward | RLHF (PPO) or RLVR |
+| Limited compute | DPO + LoRA |
+| Open dataset | UltraFeedback / HH-RLHF |
+| Tone fine-tune | DPO + custom pairs |
+
+**기본값**: DPO + LoRA (efficient) + UltraFeedback.
+
+## 🔗 Graph
+- 부모: [[Fine-Tuning]] · [[Preference-Learning]]
+- 변형: [[ORPO]] · [[SimPO]] · [[KTO]] · [[IPO]]
+- 응용: [[Axolotl]] · [[Llama]]
+- Adjacent: [[RLHF]] · [[AI_Safety_and_Alignment|Constitutional-AI]] · [[Best-of-N_Sampling]] · [[Credit Assignment Problem]] · [[Cross-Entropy Loss]]
+
+## 🤖 LLM 활용
+**언제**: 매 LLM alignment. 매 customer-specific tone. 매 RLHF alternative. 매 fine-tune at scale.
+**언제 X**: 매 verifiable task (RLVR / process reward). 매 small data (SFT 의 enough).
+
+## ❌ 안티패턴
+- **No reference model** (DPO): 매 over-fit.
+- **β too high**: 매 underutilize preference.
+- **β too low**: 매 reference 의 drift.
+- **Length bias 의 ignore**: 매 long answer 의 win.
+- **Single-pair training**: 매 noisy.
+- **No SFT first**: 매 quality drop.
+
+## 🧪 검증 / 중복
+- Verified (Rafailov et al. 2023 DPO, ORPO 2024, KTO 2024).
+- 신뢰도 A.
+- Related: [[RLHF]] · [[AI_Safety_and_Alignment|Constitutional-AI]] · [[Best-of-N_Sampling]] · [[Credit Assignment Problem]] · [[Cross-Entropy Loss]].
+
+## 🕓 Changelog
+| 날짜 | 변경 |
+|---|---|
+| 2026-05-08 | Phase 1 |
+| 2026-05-10 | Manual cleanup — DPO formula + variants + 매 TRL / ORPO / KTO / LoRA / SimPO code |