2nd/10_Wiki/Topics/AI_and_ML/Black-Box-Optimization.md

---
id: wiki-2026-0508-black-box-optimization
title: Black-Box Optimization
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [블랙박스 최적화, derivative-free optimization, BBO, Bayesian optimization, CMA-ES, gradient-free]
duplicate_of: none
source_trust_level: A
confidence_score: 0.92
verification_status: applied
tags: [optimization, bayesian-optimization, cma-es, hyperparameter-tuning, automl, gradient-free, derivative-free]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: Optuna / scikit-optimize / Ray Tune / BoTorch / nevergrad
---

# Black-Box Optimization

## 📌 한 줄 통찰
> **"매 gradient X 의 best 의 search"**. 매 expensive function (1 trial = hour) 의 minimum sample 의 best. 매 hyperparameter / drug / robotics / circuit design 의 standard. 매 Bayesian Optimization (GP) 의 dominant.

## 📖 핵심

### 매 setting
- 매 f(x): 매 expensive (분 ~ 일).
- 매 gradient X 또는 매 noisy.
- 매 budget 매 limited (10-1000 trial).
- 매 goal: min/max f.

### 매 method

#### Random / Grid search
- 매 simple, 매 baseline.
- 매 random > grid (high-dim).

#### Bayesian Optimization (BO)
- 매 surrogate model (Gaussian Process / TPE) 의 fit.
- 매 acquisition function (EI, UCB, PI) 의 next 결정.
- ✅ 매 sample-efficient.
- ❌ 매 GP scale O(N³).

#### Evolutionary
- **CMA-ES**: 매 covariance matrix adaptation. 매 continuous.
- **GA**: 매 discrete.
- **Differential Evolution**: 매 robust.

#### Simulated Annealing
- 매 random walk + 매 cooling schedule.
- 매 escape local min.

#### Population-based
- **Particle Swarm** (PSO).
- **Population-Based Training** (PBT, DeepMind).

#### TPE (Tree-structured Parzen Estimator)
- 매 Optuna default.
- 매 conditional parameter OK.

#### NES (Natural Evolution Strategy)
- 매 OpenAI ES.
- 매 distributed-friendly.

### 매 acquisition function (BO)
- **Expected Improvement (EI)**: 매 expected gain over best.
- **UCB** (Upper Confidence Bound): 매 exploit + explore (κ).
- **PI** (Probability of Improvement): 매 simple.
- **TS** (Thompson Sampling): 매 sample posterior.
- **q-EI**: 매 batch parallel.

### 매 응용
1. **Hyperparameter tune**: 매 Optuna, 매 Ray Tune.
2. **AutoML**: 매 architecture + hyperparam.
3. **Drug discovery**: 매 molecule design.
4. **Robotics**: 매 policy parameter.
5. **A/B test**: 매 thompson sampling.
6. **Material design**: 매 alloy composition.
7. **Compiler**: 매 optimization flag.
8. **NN architecture search**: NAS.

### 매 high-dim / structured
- **Trust Region BO**: 매 local search.
- **Multi-fidelity**: 매 cheap proxy.
- **Constraint BO**: 매 feasibility constraint.
- **Multi-objective**: 매 Pareto front.
- **Categorical / mixed**: 매 SMAC, 매 TPE.

### 매 modern compute
- **Parallel batch**: 매 q-acquisition.
- **Async**: 매 worker 의 done 의 즉시 propose.
- **Warm-start**: 매 prior task 의 transfer.
- **Multi-fidelity** (Hyperband, BOHB): 매 budget allocation.

## 💻 패턴

### Optuna (TPE)
```python
import optuna

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    n_layers = trial.suggest_int('n_layers', 1, 5)
    optimizer = trial.suggest_categorical('optimizer', ['adam', 'sgd'])
    
    model = build(n_layers, lr, optimizer)
    return train_and_eval(model)

study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100, n_jobs=4)

print(study.best_params, study.best_value)
```

### scikit-optimize (GP-BO)
```python
from skopt import gp_minimize
from skopt.space import Real, Integer, Categorical

space = [
    Real(1e-5, 1e-1, prior='log-uniform', name='lr'),
    Integer(1, 10, name='depth'),
    Categorical(['relu', 'gelu'], name='activation'),
]

result = gp_minimize(
    objective,
    space,
    n_calls=50,
    acq_func='EI',
    random_state=42,
)
print(f'best: {result.x}, value: {result.fun}')
```

### CMA-ES (continuous)
```python
import cma

def objective(x):
    return sum(xi**2 for xi in x)  # 매 minimize

es = cma.CMAEvolutionStrategy(x0=[1.0]*10, sigma0=0.5)
es.optimize(objective, iterations=100)
print(es.result.xbest)
```

### BoTorch (PyTorch BO)
```python
import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_mll
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf
from gpytorch.mlls import ExactMarginalLogLikelihood

# 매 X, Y 의 train data
gp = SingleTaskGP(X, Y)
mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
fit_gpytorch_mll(mll)

ei = ExpectedImprovement(model=gp, best_f=Y.max())
candidate, _ = optimize_acqf(
    ei, bounds=bounds, q=1, num_restarts=10, raw_samples=512,
)
# 매 candidate 의 evaluate → 매 GP 의 update.
```

### Hyperband / BOHB (multi-fidelity)
```python
from ray import tune
from ray.tune.schedulers import HyperBandScheduler

scheduler = HyperBandScheduler(metric='loss', mode='min')
analysis = tune.run(
    train_fn,
    config={'lr': tune.loguniform(1e-5, 1e-1)},
    scheduler=scheduler,
    num_samples=100,
    resources_per_trial={'gpu': 1},
)
```

→ 매 cheap (low epoch) 의 explore + 매 promising 의 더 exploit.

### Multi-objective (Pareto)
```python
import optuna

def objective(trial):
    x = trial.suggest_float('x', 0, 5)
    y = trial.suggest_float('y', 0, 5)
    return x**2, (x-2)**2 + y**2  # 매 둘 다 minimize

study = optuna.create_study(directions=['minimize', 'minimize'])
study.optimize(objective, n_trials=100)
# 매 Pareto front 의 visualize.
optuna.visualization.plot_pareto_front(study).show()
```

## 🤔 결정 기준
| 상황 | Method |
|---|---|
| Hyperparam (medium budget) | Optuna (TPE) |
| Hyperparam (small budget) | GP-BO (skopt / BoTorch) |
| Continuous high-dim | CMA-ES |
| Discrete + continuous | TPE / SMAC |
| Multi-fidelity | BOHB / Hyperband |
| Distributed / async | Ray Tune |
| RL policy | CMA-ES / OpenAI ES |
| Multi-objective | NSGA-II / qNEHVI |

**기본값**: Optuna 의 baseline. 매 small budget 가 BoTorch.

## 🔗 Graph
- 부모: [[Optimization]] · [[AutoML]] · [[Hyperparameters|Hyperparameter-Tuning]]
- 변형: [[Bayesian-Optimization]] · [[CMA-ES]] · [[Genetic-Algorithm]] · [[Simulated-Annealing]]
- 응용: [[Optuna]]
- Adjacent: [[Gaussian-Process]] · [[NAS]]

## 🤖 LLM 활용
**언제**: 매 expensive function. 매 hyperparameter tune. 매 gradient 없는 system. 매 design space search.
**언제 X**: 매 cheap function (gradient 더 fast). 매 closed-form solution.

## ❌ 안티패턴
- **Grid search high-dim**: 매 curse of dimensionality.
- **Acquisition 의 always EI** (high-noise): 매 UCB 가 좋음.
- **No warm-start (related task)**: 매 sample waste.
- **GP 의 1000+ trial**: 매 cubic scale.
- **No multi-fidelity** (cheap proxy 가능): 매 budget waste.
- **Single objective (multi-criteria 의 case)**: 매 weight 의 wrong.

## 🧪 검증 / 중복
- Verified (Snoek et al. BO, Hansen CMA-ES, Optuna paper).
- 신뢰도 A.
- Related: [[Bayesian-Optimization]] · [[CMA-ES]] · [[AutoML]] · [[Optuna]].

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — methods + acquisition + 매 Optuna / BoTorch / CMA-ES code |