---
id: wiki-2026-0508-loss-functions-foundations
title: Loss Functions Foundations
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Loss Functions, Cost Functions, Objective Functions, Loss-Functions]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [loss, objective, training, mse, cross-entropy, focal, contrastive, dice]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack: { language: Python, framework: PyTorch }
---

# Loss Functions Foundations

## 매 한 줄
> **"매 loss는 task가 정한다"**. Regression→MSE/MAE/Huber, Classification→CE/Focal, Metric→Contrastive/Triplet, Segmentation→Dice/IoU.

## 매 핵심
### 매 회귀 (Regression)
- **MSE (L2)**: ½(y-ŷ)². 미분 깔끔, outlier에 민감.
- **MAE (L1)**: |y-ŷ|. robust, 0에서 미분 불가.
- **Huber**: |e|<δ면 MSE, 아니면 MAE. δ=1 기본.
- **Log-cosh**: smooth Huber. 자동 미분 친화.
- **Quantile**: max(τe, (τ-1)e). 중앙값/구간 예측.

### 매 분류 (Classification)
- **BCE**: -[y log p + (1-y) log(1-p)]. 이진/다중라벨.
- **CE (softmax)**: -Σ y_k log p_k. 다중클래스.
- **Focal** (Lin 2017): -α(1-p)^γ log p. easy example down-weight, γ=2 기본.
- **Label smoothing**: y → y(1-ε) + ε/K. overconfidence 방지.
- **Hinge**: max(0, 1-y·ŷ). SVM. y∈{-1,+1}.

### 매 Metric Learning
- **Contrastive** (Hadsell 2006): pair. y·d² + (1-y)·max(0, m-d)².
- **Triplet**: max(0, d(a,p) - d(a,n) + margin).
- **InfoNCE / NT-Xent** (SimCLR): -log exp(sim+/τ) / Σ exp(sim/τ).
- **Cosine embedding**: 1 - cos(a,b).

### 매 Segmentation
- **Dice**: 1 - 2|A∩B|/(|A|+|B|). class imbalance 강함.
- **IoU/Jaccard**: 1 - |A∩B|/|A∪B|.
- **Tversky**: FP/FN weight 조정.
- **Boundary loss**: 거리변환 가중.

## 💻 패턴
### Regression losses
```python
import torch, torch.nn.functional as F
mse = F.mse_loss(pred, y)
mae = F.l1_loss(pred, y)
huber = F.huber_loss(pred, y, delta=1.0)        # smooth_l1 ≈ huber(δ=1)
# Quantile
def quantile_loss(pred, y, tau=0.5):
    e = y - pred
    return torch.maximum(tau*e, (tau-1)*e).mean()
```

### Classification losses
```python
ce = F.cross_entropy(logits, y_int)               # logits, not probs
bce = F.binary_cross_entropy_with_logits(logits, y_float)
# Label smoothing (built-in)
ce_ls = F.cross_entropy(logits, y_int, label_smoothing=0.1)
```

### Focal loss (이진)
```python
def focal_bce(logits, y, alpha=0.25, gamma=2.0):
    p = torch.sigmoid(logits)
    pt = torch.where(y == 1, p, 1 - p)
    alpha_t = torch.where(y == 1, alpha, 1 - alpha)
    return -(alpha_t * (1 - pt).pow(gamma) * pt.clamp_min(1e-8).log()).mean()
```

### Triplet & InfoNCE
```python
triplet = F.triplet_margin_loss(anchor, pos, neg, margin=1.0)

def info_nce(q, k_pos, k_neg, tau=0.07):
    # q: (B,D), k_pos: (B,D), k_neg: (B,N,D)
    pos = (q * k_pos).sum(-1, keepdim=True) / tau
    neg = torch.einsum("bd,bnd->bn", q, k_neg) / tau
    logits = torch.cat([pos, neg], dim=1)
    target = torch.zeros(q.size(0), dtype=torch.long, device=q.device)
    return F.cross_entropy(logits, target)
```

### Dice + BCE (segmentation 표준)
```python
def dice_loss(logits, y, eps=1e-6):
    p = torch.sigmoid(logits)
    inter = (p * y).sum(dim=(2, 3))
    union = p.sum(dim=(2, 3)) + y.sum(dim=(2, 3))
    return 1 - (2 * inter + eps) / (union + eps)

def combo_loss(logits, y):
    return 0.5 * F.binary_cross_entropy_with_logits(logits, y) + dice_loss(logits, y).mean()
```

### Class imbalance 가중
```python
weights = torch.tensor([1.0, 5.0, 2.0])           # 클래스별
ce_w = F.cross_entropy(logits, y_int, weight=weights)
```

## 매 결정 기준
| Task | Default | 변형 |
|---|---|---|
| Regression normal | MSE | outlier→Huber, robust→MAE |
| Binary classification | BCE w/ logits | imbalance→Focal |
| Multi-class | CE w/ label smoothing | imbalance→class weights |
| Multi-label | BCE per-class |  |
| Embedding learning | InfoNCE | small batch→Triplet |
| Segmentation | BCE+Dice | small object→Tversky |
| Object detection | Focal + IoU/GIoU | (RetinaNet, YOLO) |

**기본값**: classification CE+label smoothing 0.1, regression Huber.

## 🔗 Graph
- 부모: [[Optimization]]
- 변형: [[Focal-Loss]]
- 응용: [[Image-Classification-Mastery]], [[Segmentation]], [[Object-Detection]]
- Adjacent: [[Activation-Functions]], [[Class-Imbalance]], [[L1-and-L2-Regularization|Regularization]]

## 🤖 LLM 활용
**언제**: task→loss 매핑, gradient 직관, 코드 템플릿 생성.
**언제 X**: domain-specific custom loss 설계는 검증 필수 (분포·gradient 분석).

## ❌ 안티패턴
- `softmax` 후 `nll_loss` 손수 (numerical) ← `cross_entropy` 사용
- BCE에 `binary_cross_entropy(sigmoid(...))` ← `_with_logits` 사용
- Imbalance 무시한 CE
- Dice loss만 단독 (gradient 불안정) → BCE+Dice 혼합
- Focal γ를 imbalance 없을 때 사용 (성능↓)

## 🧪 검증 / 중복
- Verified (Goodfellow DL ch5, Lin 2017 Focal, SimCLR, Milletari V-Net Dice). 신뢰도 A.
- Canonical for [[Loss-Functions-Foundations|Loss Functions]] (redirect).

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — canonical 강화, segmentation/metric 추가 |