---
id: wiki-2026-0508-least-squares-methods
title: Least Squares Methods
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Least-Squares, OLS, 최소제곱법]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [optimization, regression, linear-algebra, statistics]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: numpy-scipy
---

# Least Squares Methods

## 매 한 줄
> **"매 residual의 제곱합을 최소화하라"**. Gauss(1795)와 Legendre(1805)가 독립 발견; 매 modern ML의 MSE loss·linear regression·Kalman filter의 직접 조상이다. 2026에도 매 LoRA fine-tuning의 closed-form initializer로 살아있다.

## 매 핵심

### 매 정식화
- **OLS**: minimize ‖y − Xβ‖²; closed-form β̂ = (XᵀX)⁻¹Xᵀy.
- **Weighted LS**: residual에 weight W; β̂ = (XᵀWX)⁻¹XᵀWy.
- **Total LS**: 매 X에도 noise — SVD 기반.
- **Regularized**: Ridge (L2), Lasso (L1), Elastic Net.

### 매 수치적 안정성
- Normal equation 직접 풀이는 매 condition number² 증폭 → QR/SVD 권장.
- 매 LAPACK `gels` 는 QR 사용.

### 매 응용
1. Linear regression (econometrics, ML baseline).
2. Curve fitting (lmfit, scipy.optimize.curve_fit).
3. Bundle adjustment (SLAM, photogrammetry).
4. Kalman update step (least-squares projection).

## 💻 패턴

### OLS via NumPy (lstsq)
```python
import numpy as np
beta, residuals, rank, sv = np.linalg.lstsq(X, y, rcond=None)
```

### Ridge regression
```python
def ridge(X, y, lam):
    n_features = X.shape[1]
    return np.linalg.solve(X.T @ X + lam * np.eye(n_features), X.T @ y)
```

### Nonlinear least squares
```python
from scipy.optimize import least_squares
def residual(params, x, y):
    a, b, c = params
    return y - (a * np.exp(-b * x) + c)
result = least_squares(residual, [1, 1, 0], args=(x, y))
```

### QR-based solve (numerically stable)
```python
Q, R = np.linalg.qr(X)
beta = np.linalg.solve(R, Q.T @ y)
```

### Weighted LS
```python
W_sqrt = np.sqrt(weights)
beta = np.linalg.lstsq(X * W_sqrt[:, None], y * W_sqrt, rcond=None)[0]
```

### Recursive LS (online update)
```python
def rls_update(P, beta, x, y, lam=0.99):
    k = P @ x / (lam + x @ P @ x)
    beta = beta + k * (y - x @ beta)
    P = (P - np.outer(k, x @ P)) / lam
    return P, beta
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Well-conditioned, small p | Normal equation |
| Ill-conditioned | QR or SVD |
| Many features, sparse | Lasso |
| Heteroscedastic noise | Weighted LS |
| Streaming data | RLS |

**기본값**: `np.linalg.lstsq` (uses SVD-based GELSD).

## 🔗 Graph
- 부모: [[Linear-Algebra-Foundations]] · [[Optimization]]
- 변형: [[Ridge-Regression]] · [[Lasso]]
- 응용: [[Kalman-Filter-and-State-Tracking]] · [[Linear-Regression]] · [[Bundle-Adjustment]]

## 🤖 LLM 활용
**언제**: Linear/affine model fit, baseline regression, calibration curves.
**언제 X**: Outliers 다수 (use Huber/RANSAC), heavy non-linearity (use NN/GP).

## ❌ 안티패턴
- **Normal equation 남용**: 매 ill-conditioned에서 매 silently 망함.
- **Outlier 무시**: L2 loss는 매 outlier 민감 — robust loss 고려.
- **Multicollinearity 방치**: VIF 점검 또는 Ridge.

## 🧪 검증 / 중복
- Verified (Trefethen & Bau "Numerical Linear Algebra").
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — OLS/Ridge/RLS patterns |