Files

T

Antigravity Agent f8b21af4be Wiki cleanup: error-doc removal, dedup merge, link normalization

10_Wiki/Topics 대규모 정리:
- 오류 캡처/미완성 stub 문서 227개 제거
- 교차폴더 중복 43클러스터 병합 (63파일 → redirect)
- 링크명 정규화: 깨진 링크 수정·redirect 직결·개념 매핑 ~2,400건
- 카테고리 MOC 6개 신규 생성
- Graph 섹션 미해결 related-keyword 링크 10,058건 제거

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-20 23:52:15 +09:00

6.4 KiB

Raw Blame History

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

title

Hyperparameters

매 한 줄

"매 model 의 의 학습 의 의 의 의 의 외부 parameter". 매 learning rate, batch size, depth, regularization. 매 modern HPO: Optuna (Bayesian/TPE), Ray Tune (distributed), Wandb Sweeps. 매 cost vs payoff trade-off — 매 무한정 search 의 X.

매 핵심

매 type

Optimizer: lr, momentum, weight decay.
Architecture: depth, width, head count.
Regularization: dropout, label smoothing.
Training: batch size, epoch, warmup.
Data: augmentation strength.

매 search method

Grid.
Random (Bergstra 2012 — better than grid).
Bayesian (TPE, GP).
Hyperband / ASHA: 매 early stopping.
PBT (Population-based, DeepMind).
NAS (Neural Arch Search).

매 응용

Tabular ML: 매 큰 영향.
DL: 매 medium-size 의 critical.
LLM fine-tune: 매 lr + LoRA r.

매 modern best practice

Random > grid.
Bayesian when expensive.
Hyperband for many configs.
Log-scale lr.
Track everything (Wandb).
Cap budget time/$.

💻 패턴

Optuna (Bayesian TPE)

import optuna

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    bs = trial.suggest_categorical('bs', [16, 32, 64, 128])
    dropout = trial.suggest_float('dropout', 0.0, 0.5)
    n_layers = trial.suggest_int('n_layers', 2, 8)
    
    model = build(n_layers, dropout)
    val_loss = train(model, lr, bs)
    return val_loss

study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100)
print(study.best_params)

Random search (sklearn)

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

params = {'lr': loguniform(1e-5, 1e-1), 'bs': [16, 32, 64], 'dropout': uniform(0, 0.5)}
search = RandomizedSearchCV(model, params, n_iter=50, cv=5)
search.fit(X, y)

Hyperband (ASHA)

from ray import tune
from ray.tune.schedulers import ASHAScheduler

def train_fn(config):
    for epoch in range(config['epochs']):
        loss = train_step(config)
        tune.report(loss=loss)

tune.run(
    train_fn,
    config={'lr': tune.loguniform(1e-5, 1e-1), 'bs': tune.choice([16, 32, 64])},
    scheduler=ASHAScheduler(metric='loss', mode='min'),
    num_samples=100,
)

Population-Based Training (PBT)

from ray.tune.schedulers import PopulationBasedTraining
pbt = PopulationBasedTraining(
    time_attr='training_iteration',
    metric='loss', mode='min', perturbation_interval=5,
    hyperparam_mutations={'lr': lambda: tune.loguniform(1e-5, 1e-1)},
)

Wandb Sweeps

# 매 sweep.yaml
program: train.py
method: bayes
metric: { name: val_loss, goal: minimize }
parameters:
  lr: { min: 1e-5, max: 1e-1, distribution: log_uniform_values }
  bs: { values: [16, 32, 64, 128] }

wandb sweep sweep.yaml
wandb agent <sweep_id>

Default starting points

DEFAULTS = {
    'transformer': {'lr': 3e-4, 'bs': 32, 'warmup': 4000, 'wd': 0.01},
    'cnn': {'lr': 1e-3, 'bs': 256, 'momentum': 0.9, 'wd': 1e-4},
    'lora': {'lr': 1e-4, 'r': 16, 'alpha': 32, 'dropout': 0.05},
}

LR finder (Smith)

def find_lr(model, train_loader, lr_min=1e-7, lr_max=1):
    lrs = np.geomspace(lr_min, lr_max, 100)
    losses = []
    for lr in lrs:
        for p in model.optimizer.param_groups: p['lr'] = lr
        loss = train_one_batch(next(iter(train_loader)))
        losses.append(loss)
    # 매 plot lrs vs losses → pick before divergence
    return lrs, losses

LoRA hyperparameter

from peft import LoraConfig
config = LoraConfig(
    r=trial.suggest_categorical('r', [8, 16, 32, 64]),
    lora_alpha=trial.suggest_categorical('alpha', [16, 32, 64, 128]),
    lora_dropout=trial.suggest_float('dropout', 0, 0.2),
)

Cost cap

import time
class BudgetedStudy:
    def __init__(self, budget_hours=4):
        self.start = time.time()
        self.budget = budget_hours * 3600
    
    def should_continue(self):
        return time.time() - self.start < self.budget

Early stopping per trial

def objective_with_pruning(trial):
    for epoch in range(50):
        loss = train_step()
        trial.report(loss, epoch)
        if trial.should_prune(): raise optuna.TrialPruned()
    return loss

Track best config (Wandb)

import wandb
wandb.init(project='hpo', config=trial.params)
for epoch in range(epochs):
    wandb.log({'loss': loss, 'lr': lr, 'epoch': epoch})

NAS-Bench

# 매 NAS-Bench-101/201 — pre-computed architectures
import nasbench
nasbench_model = nasbench.NASBench('nasbench_only108.tfrecord')
arch = sample_architecture()
metrics = nasbench_model.query(arch)

매 결정 기준

상황	Method
< 100 trials	Random
Expensive trial	Bayesian (Optuna)
Many configs	Hyperband / ASHA
Long-running	PBT
Default start	Architecture-specific defaults
Tight budget	LR finder + few trials

기본값: 매 Optuna TPE + Hyperband prune + Wandb track + log-scale lr + budget cap. 매 cost-aware.

🔗 Graph

부모: Machine-Learning · AutoML
변형: Hyperparameters · NAS
응용: Optuna
Adjacent: Bayesian-Optimization · Gaussian-Processes · Fine-tuning

🤖 LLM 활용

언제: 매 production model. 매 fine-tune. 매 architecture sweep. 언제 X: 매 throwaway / quick PoC.

❌ 안티패턴

Grid for many params: 매 exponential cost.
No log-scale lr: 매 wasteful.
Ignore early stopping: 매 budget waste.
No baseline: 매 HPO 의 worth 의 invisible.
Test set leak: 매 HPO with test.

🧪 검증 / 중복

Verified (Bergstra 2012, Optuna docs, Ray Tune docs).
신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — methods + 매 Optuna / Ray / Wandb / LR finder code

6.4 KiB Raw Blame History