2nd/10_Wiki/Topics/AI_and_ML/Black-Box-Optimization.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-black-box-optimization	Black-Box Optimization	10_Wiki/Topics	verified	self	블랙박스 최적화 derivative-free optimization BBO Bayesian optimization CMA-ES gradient-free	none	A	0.92	applied	optimization bayesian-optimization cma-es hyperparameter-tuning automl gradient-free derivative-free		2026-05-10	pending	language framework Python Optuna / scikit-optimize / Ray Tune / BoTorch / nevergrad

Black-Box Optimization

📌 한 줄 통찰

"매 gradient X 의 best 의 search". 매 expensive function (1 trial = hour) 의 minimum sample 의 best. 매 hyperparameter / drug / robotics / circuit design 의 standard. 매 Bayesian Optimization (GP) 의 dominant.

📖 핵심

매 setting

• 매 f(x): 매 expensive (분 ~ 일).
• 매 gradient X 또는 매 noisy.
• 매 budget 매 limited (10-1000 trial).
• 매 goal: min/max f.

매 method

Random / Grid search

• 매 simple, 매 baseline.
• 매 random > grid (high-dim).

Bayesian Optimization (BO)

• 매 surrogate model (Gaussian Process / TPE) 의 fit.
• 매 acquisition function (EI, UCB, PI) 의 next 결정.
• ✅ 매 sample-efficient.
• ❌ 매 GP scale O(N³).

Evolutionary

• CMA-ES: 매 covariance matrix adaptation. 매 continuous.
• GA: 매 discrete.
• Differential Evolution: 매 robust.

Simulated Annealing

• 매 random walk + 매 cooling schedule.
• 매 escape local min.

Population-based

• Particle Swarm (PSO).
• Population-Based Training (PBT, DeepMind).

TPE (Tree-structured Parzen Estimator)

• 매 Optuna default.
• 매 conditional parameter OK.

NES (Natural Evolution Strategy)

• 매 OpenAI ES.
• 매 distributed-friendly.

매 acquisition function (BO)

• Expected Improvement (EI): 매 expected gain over best.
• UCB (Upper Confidence Bound): 매 exploit + explore (κ).
• PI (Probability of Improvement): 매 simple.
• TS (Thompson Sampling): 매 sample posterior.
• q-EI: 매 batch parallel.

매 응용

1. Hyperparameter tune: 매 Optuna, 매 Ray Tune.
2. AutoML: 매 architecture + hyperparam.
3. Drug discovery: 매 molecule design.
4. Robotics: 매 policy parameter.
5. A/B test: 매 thompson sampling.
6. Material design: 매 alloy composition.
7. Compiler: 매 optimization flag.
8. NN architecture search: NAS.

매 high-dim / structured

• Trust Region BO: 매 local search.
• Multi-fidelity: 매 cheap proxy.
• Constraint BO: 매 feasibility constraint.
• Multi-objective: 매 Pareto front.
• Categorical / mixed: 매 SMAC, 매 TPE.

매 modern compute

• Parallel batch: 매 q-acquisition.
• Async: 매 worker 의 done 의 즉시 propose.
• Warm-start: 매 prior task 의 transfer.
• Multi-fidelity (Hyperband, BOHB): 매 budget allocation.

💻 패턴

Optuna (TPE)

import optuna

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    n_layers = trial.suggest_int('n_layers', 1, 5)
    optimizer = trial.suggest_categorical('optimizer', ['adam', 'sgd'])
    
    model = build(n_layers, lr, optimizer)
    return train_and_eval(model)

study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100, n_jobs=4)

print(study.best_params, study.best_value)

scikit-optimize (GP-BO)

from skopt import gp_minimize
from skopt.space import Real, Integer, Categorical

space = [
    Real(1e-5, 1e-1, prior='log-uniform', name='lr'),
    Integer(1, 10, name='depth'),
    Categorical(['relu', 'gelu'], name='activation'),
]

result = gp_minimize(
    objective,
    space,
    n_calls=50,
    acq_func='EI',
    random_state=42,
)
print(f'best: {result.x}, value: {result.fun}')

CMA-ES (continuous)

import cma

def objective(x):
    return sum(xi**2 for xi in x)  # 매 minimize

es = cma.CMAEvolutionStrategy(x0=[1.0]*10, sigma0=0.5)
es.optimize(objective, iterations=100)
print(es.result.xbest)

BoTorch (PyTorch BO)

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_mll
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf
from gpytorch.mlls import ExactMarginalLogLikelihood

# 매 X, Y 의 train data
gp = SingleTaskGP(X, Y)
mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
fit_gpytorch_mll(mll)

ei = ExpectedImprovement(model=gp, best_f=Y.max())
candidate, _ = optimize_acqf(
    ei, bounds=bounds, q=1, num_restarts=10, raw_samples=512,
)
# 매 candidate 의 evaluate → 매 GP 의 update.

Hyperband / BOHB (multi-fidelity)

from ray import tune
from ray.tune.schedulers import HyperBandScheduler

scheduler = HyperBandScheduler(metric='loss', mode='min')
analysis = tune.run(
    train_fn,
    config={'lr': tune.loguniform(1e-5, 1e-1)},
    scheduler=scheduler,
    num_samples=100,
    resources_per_trial={'gpu': 1},
)

→ 매 cheap (low epoch) 의 explore + 매 promising 의 더 exploit.

Multi-objective (Pareto)

import optuna

def objective(trial):
    x = trial.suggest_float('x', 0, 5)
    y = trial.suggest_float('y', 0, 5)
    return x**2, (x-2)**2 + y**2  # 매 둘 다 minimize

study = optuna.create_study(directions=['minimize', 'minimize'])
study.optimize(objective, n_trials=100)
# 매 Pareto front 의 visualize.
optuna.visualization.plot_pareto_front(study).show()

🤔 결정 기준

상황	Method
Hyperparam (medium budget)	Optuna (TPE)
Hyperparam (small budget)	GP-BO (skopt / BoTorch)
Continuous high-dim	CMA-ES
Discrete + continuous	TPE / SMAC
Multi-fidelity	BOHB / Hyperband
Distributed / async	Ray Tune
RL policy	CMA-ES / OpenAI ES
Multi-objective	NSGA-II / qNEHVI

기본값: Optuna 의 baseline. 매 small budget 가 BoTorch.

🔗 Graph

• 부모: Optimization · AutoML · Hyperparameter-Tuning
• 변형: Bayesian-Optimization · CMA-ES · Genetic-Algorithm · Simulated-Annealing · TPE
• 응용: Optuna · Ray-Tune · BoTorch · scikit-optimize · nevergrad
• 변형 (high-dim): Trust-Region-BO · BOHB · Hyperband
• Adjacent: Gaussian-Process · Acquisition-Function · NAS

🤖 LLM 활용

언제: 매 expensive function. 매 hyperparameter tune. 매 gradient 없는 system. 매 design space search. 언제 X: 매 cheap function (gradient 더 fast). 매 closed-form solution.

❌ 안티패턴

• Grid search high-dim: 매 curse of dimensionality.
• Acquisition 의 always EI (high-noise): 매 UCB 가 좋음.
• No warm-start (related task): 매 sample waste.
• GP 의 1000+ trial: 매 cubic scale.
• No multi-fidelity (cheap proxy 가능): 매 budget waste.
• Single objective (multi-criteria 의 case): 매 weight 의 wrong.

🧪 검증 / 중복

• Verified (Snoek et al. BO, Hansen CMA-ES, Optuna paper).
• 신뢰도 A.
• Related: Bayesian-Optimization · CMA-ES · AutoML · Optuna.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — methods + acquisition + 매 Optuna / BoTorch / CMA-ES code