refactor(topics): 멀티 에이전트용 지식 재편 — _Common(공통 기본기) + Domain_* 구조

에이전트 8종(대화형/프로그래머 C·S/디자이너/설계자/기획자/QA/PD/PM)에게 [공통 기본 능력 + 롤별 Specialty] 2층으로 지식을 주입하기 위한 재분류. 문서 내용·포맷은 무수정, 폴더 이동만 (6,372개 문서 수 보존 확인). - Topic_Programming → Domain_Programming (내부 구조 보존) - Topic_Graphic → Domain_Design - Topic_Business → Domain_Product - Topic_General → Domain_General - _Common 신설: Math(구 Topic_Math_Specialty), Reasoning(구 General/From_Thinking & Reasoning), Reasoning_Creativity(구 General/From_창의성), Communication(Poetic_Blog_Writing + From_writing) - 타 도메인의 From_* 폴더는 유지 (출처 표기일 뿐, 이미 도메인에 맞게 분류된 문서) - 빈 폴더 정리 (memory/procedures) - 에이전트→폴더 매핑은 workspace의 .astra/agent-knowledge-map.json (9개 에이전트) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 11:05:56 +09:00
parent 6549ead309
commit c24165b8bc
6193 changed files with 1717 additions and 31 deletions
@@ -0,0 +1,239 @@
+---
+id: wiki-2026-0508-hyperparameters
+title: Hyperparameters
+category: 10_Wiki/Topics
+status: verified
+canonical_id: self
+aliases: [hyperparameters, HPO, learning rate, batch size, AutoML, Optuna, Bayesian opt]
+duplicate_of: none
+source_trust_level: A
+confidence_score: 0.96
+verification_status: applied
+tags: [machine-learning, hyperparameters, hpo, automl, optuna, bayesian-opt]
+raw_sources: []
+last_reinforced: 2026-05-10
+github_commit: pending
+tech_stack:
+  language: Python
+  framework: Optuna / Ray Tune / Hyperopt / Wandb Sweeps
+---
+
+# Hyperparameters
+
+## 매 한 줄
+> **"매 model 의 의 학습 의 의 의 의 의 외부 parameter"**. 매 learning rate, batch size, depth, regularization. 매 modern HPO: Optuna (Bayesian/TPE), Ray Tune (distributed), Wandb Sweeps. 매 cost vs payoff trade-off — 매 무한정 search 의 X.
+
+## 매 핵심
+
+### 매 type
+- **Optimizer**: lr, momentum, weight decay.
+- **Architecture**: depth, width, head count.
+- **Regularization**: dropout, label smoothing.
+- **Training**: batch size, epoch, warmup.
+- **Data**: augmentation strength.
+
+### 매 search method
+- **Grid**.
+- **Random** (Bergstra 2012 — better than grid).
+- **Bayesian** (TPE, GP).
+- **Hyperband / ASHA**: 매 early stopping.
+- **PBT** (Population-based, DeepMind).
+- **NAS** (Neural Arch Search).
+
+### 매 응용
+1. **Tabular ML**: 매 큰 영향.
+2. **DL**: 매 medium-size 의 critical.
+3. **LLM fine-tune**: 매 lr + LoRA r.
+
+### 매 modern best practice
+- **Random > grid**.
+- **Bayesian when expensive**.
+- **Hyperband** for many configs.
+- **Log-scale lr**.
+- **Track everything** (Wandb).
+- **Cap budget** time/$.
+
+## 💻 패턴
+
+### Optuna (Bayesian TPE)
+```python
+import optuna
+
+def objective(trial):
+    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
+    bs = trial.suggest_categorical('bs', [16, 32, 64, 128])
+    dropout = trial.suggest_float('dropout', 0.0, 0.5)
+    n_layers = trial.suggest_int('n_layers', 2, 8)
+    
+    model = build(n_layers, dropout)
+    val_loss = train(model, lr, bs)
+    return val_loss
+
+study = optuna.create_study(direction='minimize')
+study.optimize(objective, n_trials=100)
+print(study.best_params)
+```
+
+### Random search (sklearn)
+```python
+from sklearn.model_selection import RandomizedSearchCV
+from scipy.stats import loguniform
+
+params = {'lr': loguniform(1e-5, 1e-1), 'bs': [16, 32, 64], 'dropout': uniform(0, 0.5)}
+search = RandomizedSearchCV(model, params, n_iter=50, cv=5)
+search.fit(X, y)
+```
+
+### Hyperband (ASHA)
+```python
+from ray import tune
+from ray.tune.schedulers import ASHAScheduler
+
+def train_fn(config):
+    for epoch in range(config['epochs']):
+        loss = train_step(config)
+        tune.report(loss=loss)
+
+tune.run(
+    train_fn,
+    config={'lr': tune.loguniform(1e-5, 1e-1), 'bs': tune.choice([16, 32, 64])},
+    scheduler=ASHAScheduler(metric='loss', mode='min'),
+    num_samples=100,
+)
+```
+
+### Population-Based Training (PBT)
+```python
+from ray.tune.schedulers import PopulationBasedTraining
+pbt = PopulationBasedTraining(
+    time_attr='training_iteration',
+    metric='loss', mode='min', perturbation_interval=5,
+    hyperparam_mutations={'lr': lambda: tune.loguniform(1e-5, 1e-1)},
+)
+```
+
+### Wandb Sweeps
+```yaml
+# 매 sweep.yaml
+program: train.py
+method: bayes
+metric: { name: val_loss, goal: minimize }
+parameters:
+  lr: { min: 1e-5, max: 1e-1, distribution: log_uniform_values }
+  bs: { values: [16, 32, 64, 128] }
+```
+
+```bash
+wandb sweep sweep.yaml
+wandb agent <sweep_id>
+```
+
+### Default starting points
+```python
+DEFAULTS = {
+    'transformer': {'lr': 3e-4, 'bs': 32, 'warmup': 4000, 'wd': 0.01},
+    'cnn': {'lr': 1e-3, 'bs': 256, 'momentum': 0.9, 'wd': 1e-4},
+    'lora': {'lr': 1e-4, 'r': 16, 'alpha': 32, 'dropout': 0.05},
+}
+```
+
+### LR finder (Smith)
+```python
+def find_lr(model, train_loader, lr_min=1e-7, lr_max=1):
+    lrs = np.geomspace(lr_min, lr_max, 100)
+    losses = []
+    for lr in lrs:
+        for p in model.optimizer.param_groups: p['lr'] = lr
+        loss = train_one_batch(next(iter(train_loader)))
+        losses.append(loss)
+    # 매 plot lrs vs losses → pick before divergence
+    return lrs, losses
+```
+
+### LoRA hyperparameter
+```python
+from peft import LoraConfig
+config = LoraConfig(
+    r=trial.suggest_categorical('r', [8, 16, 32, 64]),
+    lora_alpha=trial.suggest_categorical('alpha', [16, 32, 64, 128]),
+    lora_dropout=trial.suggest_float('dropout', 0, 0.2),
+)
+```
+
+### Cost cap
+```python
+import time
+class BudgetedStudy:
+    def __init__(self, budget_hours=4):
+        self.start = time.time()
+        self.budget = budget_hours * 3600
+    
+    def should_continue(self):
+        return time.time() - self.start < self.budget
+```
+
+### Early stopping per trial
+```python
+def objective_with_pruning(trial):
+    for epoch in range(50):
+        loss = train_step()
+        trial.report(loss, epoch)
+        if trial.should_prune(): raise optuna.TrialPruned()
+    return loss
+```
+
+### Track best config (Wandb)
+```python
+import wandb
+wandb.init(project='hpo', config=trial.params)
+for epoch in range(epochs):
+    wandb.log({'loss': loss, 'lr': lr, 'epoch': epoch})
+```
+
+### NAS-Bench
+```python
+# 매 NAS-Bench-101/201 — pre-computed architectures
+import nasbench
+nasbench_model = nasbench.NASBench('nasbench_only108.tfrecord')
+arch = sample_architecture()
+metrics = nasbench_model.query(arch)
+```
+
+## 매 결정 기준
+| 상황 | Method |
+|---|---|
+| < 100 trials | Random |
+| Expensive trial | Bayesian (Optuna) |
+| Many configs | Hyperband / ASHA |
+| Long-running | PBT |
+| Default start | Architecture-specific defaults |
+| Tight budget | LR finder + few trials |
+
+**기본값**: 매 Optuna TPE + Hyperband prune + Wandb track + log-scale lr + budget cap. 매 cost-aware.
+
+## 🔗 Graph
+- 부모: [[Machine-Learning]] · [[AutoML]]
+- 변형: [[Hyperparameters|Hyperparameter-Optimization]] · [[NAS]]
+- 응용: [[Optuna]]
+- Adjacent: [[Bayesian-Optimization]] · [[Gaussian-Processes]] · [[Fine-tuning]]
+
+## 🤖 LLM 활용
+**언제**: 매 production model. 매 fine-tune. 매 architecture sweep.
+**언제 X**: 매 throwaway / quick PoC.
+
+## ❌ 안티패턴
+- **Grid for many params**: 매 exponential cost.
+- **No log-scale lr**: 매 wasteful.
+- **Ignore early stopping**: 매 budget waste.
+- **No baseline**: 매 HPO 의 worth 의 invisible.
+- **Test set leak**: 매 HPO with test.
+
+## 🧪 검증 / 중복
+- Verified (Bergstra 2012, Optuna docs, Ray Tune docs).
+- 신뢰도 A.
+
+## 🕓 Changelog
+| 날짜 | 변경 |
+|---|---|
+| 2026-05-08 | Phase 1 |
+| 2026-05-10 | Manual cleanup — methods + 매 Optuna / Ray / Wandb / LR finder code |