---
id: wiki-2026-0508-lagrange-multipliers
title: Lagrange Multipliers
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Lagrangian Method, Constrained Optimization, KKT]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [optimization, calculus, constrained, kkt, ml-foundations]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: scipy, cvxpy
---

# Lagrange Multipliers

## 매 한 줄
> **"매 constrained optimization 의 universal trick"**. Lagrange Multiplier 방법은 equality-constrained problem $\min f(x)$ s.t. $g(x)=0$ 을 $\nabla f = \lambda \nabla g$ 의 unconstrained stationary condition 으로 reduce 한다. Joseph-Louis Lagrange (1788) 이 도입했고, 2026 ML 의 SVM, regularization, RLHF reward shaping, physics-informed NN 에 매 fundamental.

## 매 핵심

### 매 Equality 형
- problem: $\min f(x)$ s.t. $g_i(x) = 0$
- Lagrangian: $\mathcal{L}(x, \lambda) = f(x) + \sum_i \lambda_i g_i(x)$
- stationarity: $\nabla_x \mathcal{L} = 0, \nabla_\lambda \mathcal{L} = 0$
- $\lambda_i$ = sensitivity of optimal $f^*$ to perturbation of $g_i$ (shadow price)

### 매 KKT (inequality 확장)
- problem: $\min f(x)$ s.t. $g_i(x) \le 0, h_j(x) = 0$
- $\mathcal{L} = f + \sum \mu_i g_i + \sum \lambda_j h_j$
- KKT conditions:
  1. **Stationarity**: $\nabla_x \mathcal{L} = 0$
  2. **Primal feasibility**: $g_i \le 0, h_j = 0$
  3. **Dual feasibility**: $\mu_i \ge 0$
  4. **Complementary slackness**: $\mu_i g_i = 0$

### 매 응용
1. **SVM**: max-margin → KKT dual.
2. **Regularization**: Lagrangian view of $\min L + \lambda \|w\|^2$.
3. **Constrained RL**: CPO, Lagrangian SAC.
4. **Physics**: Euler-Lagrange equations of motion.
5. **Economics**: utility maximization budget constraint.

## 💻 패턴

### Symbolic solve (sympy)
```python
import sympy as sp

x, y, lam = sp.symbols("x y lambda")
f = x**2 + y**2          # min x^2 + y^2
g = x + y - 1            # s.t. x + y = 1
L = f + lam * g

eqs = [sp.diff(L, v) for v in (x, y, lam)]
sol = sp.solve(eqs, [x, y, lam])
print(sol)  # {x: 1/2, y: 1/2, lambda: -1}
```

### scipy constrained
```python
from scipy.optimize import minimize

f = lambda x: x[0]**2 + x[1]**2
cons = {"type": "eq", "fun": lambda x: x[0] + x[1] - 1}
res = minimize(f, x0=[0.0, 0.0], constraints=cons)
print(res.x)  # ≈ [0.5, 0.5]
```

### CVXPY (convex)
```python
import cvxpy as cp
x = cp.Variable(2)
prob = cp.Problem(cp.Minimize(cp.sum_squares(x)),
                  [cp.sum(x) == 1, x >= 0])
prob.solve()
print(x.value, prob.constraints[0].dual_value)  # primal + lambda
```

### SVM dual derivation
```python
# Primal: min 1/2 ||w||^2 s.t. y_i(w·x_i + b) >= 1
# Dual via KKT: max sum a_i - 1/2 sum a_i a_j y_i y_j x_i·x_j
# s.t. a_i >= 0, sum a_i y_i = 0
import numpy as np
from scipy.optimize import minimize

def svm_dual(X, y):
    n = len(y)
    K = (y[:,None]*y) * (X @ X.T)
    obj = lambda a: 0.5 * a @ K @ a - a.sum()
    cons = [{"type":"eq", "fun": lambda a: a @ y}]
    bnds = [(0, None)] * n
    res = minimize(obj, np.zeros(n), bounds=bnds, constraints=cons)
    return res.x  # dual variables = lambda_i
```

### Penalty / augmented Lagrangian
```python
def augmented_lagrangian(f, g, x0, mu0=1.0, rho=1.5, iters=20):
    x, mu = x0, mu0
    lam = 0.0
    for _ in range(iters):
        # solve unconstrained
        from scipy.optimize import minimize
        L = lambda x: f(x) + lam*g(x) + (mu/2)*g(x)**2
        x = minimize(L, x).x
        lam += mu * g(x)  # multiplier update
        mu *= rho
    return x, lam
```

## 매 결정 기준
| Problem | Method |
|---|---|
| small symbolic | sympy |
| smooth nonlinear | scipy SLSQP |
| convex | CVXPY |
| large-scale | augmented Lagrangian / ADMM |
| RL constraints | Lagrangian PPO/SAC |

**기본값**: CVXPY for convex, scipy SLSQP otherwise.

## 🔗 Graph
- 부모: [[Optimization]]
- 응용: [[SVM]] · [[L1-and-L2-Regularization|Regularization]]

## 🤖 LLM 활용
**언제**: deriving SVM/RLHF math, constrained policy optimization, MLE w/ constraints.
**언제 X**: unconstrained smooth — 그냥 gradient descent.

## ❌ 안티패턴
- **Forgetting CQ**: constraint qualification (LICQ, Slater) 만족 X 면 KKT 무효.
- **Sign convention 혼동**: $\le$ 와 $\ge$ 따라 $\mu$ sign flip.
- **Non-convex KKT 의 sufficiency 가정**: KKT 는 매 necessary, sufficient 는 convex 에서만.
- **Penalty 만 사용**: ill-conditioning — augmented Lagrangian 사용.

## 🧪 검증 / 중복
- Verified (Boyd & Vandenberghe 2004 ch 5, Nocedal & Wright 2006 ch 12).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — Lagrangian, KKT, SVM dual, augmented Lagrangian |
