2nd/10_Wiki/Topics/AI_and_ML/Label-Noise-and-Robustness.md

---
id: wiki-2026-0508-label-noise-and-robustness
title: Label Noise and Robustness
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Noisy Labels, Label Cleaning, Confident Learning]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [ml, label-noise, cleanlab, robust-loss, gce, symmetric-ce]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack: { language: python, framework: cleanlab/pytorch }
---

# Label Noise and Robustness

## 매 한 줄
> **"매 데이터셋의 라벨은 거짓말한다"**. Confident learning 으로 의심 라벨 찾고, robust loss 로 학습 자체를 노이즈에 둔감하게.

## 매 핵심
### 매 노이즈 종류
- **Symmetric** (uniform): 모든 클래스로 균등 flip
- **Asymmetric** (class-conditional): 비슷한 class 로 flip (cat→dog 高)
- **Instance-dependent**: 어려운 sample 일수록 noise 높음 (가장 현실적, 가장 어려움)

### 매 두 갈래
- **Data-centric**: 의심 라벨 찾아서 제거/수정 → cleanlab, confident learning
- **Model-centric**: 노이즈가 있어도 학습 강건 → robust loss (GCE, SCE), co-teaching, MixUp

### 매 응용
1. Crowdsourced 라벨 정리
2. Web-scraped dataset cleaning
3. Auto-label 후 검증
4. Class imbalance + noise 동시
5. Active learning 의 라벨 효율 극대화

## 💻 패턴

### Pattern 1: cleanlab basic
```python
from cleanlab.classification import CleanLearning
from sklearn.linear_model import LogisticRegression

cl = CleanLearning(clf=LogisticRegression())
cl.fit(X, y)  # 자동으로 의심 label 찾고 제거 후 재학습

# 또는 의심 label 만 받기
from cleanlab.filter import find_label_issues
pred_probs = model.predict_proba(X)
issues = find_label_issues(labels=y, pred_probs=pred_probs, return_indices_ranked_by="self_confidence")
print(f"의심 label {len(issues)} 개")
```

### Pattern 2: Confident learning manual
```python
# 1. K-fold OOF prediction
from sklearn.model_selection import cross_val_predict
pred_probs = cross_val_predict(model, X, y, cv=5, method="predict_proba")

# 2. cleanlab 으로 issue 탐지
from cleanlab import Datalab
lab = Datalab(data={"X": X, "y": y}, label_name="y")
lab.find_issues(pred_probs=pred_probs)
lab.report()
```

### Pattern 3: Generalized Cross Entropy (GCE)
```python
import torch, torch.nn.functional as F

class GCELoss(torch.nn.Module):
    def __init__(self, q=0.7):
        super().__init__()
        self.q = q
    def forward(self, logits, targets):
        probs = F.softmax(logits, dim=-1)
        p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1).clamp(min=1e-7)
        return ((1 - p_t.pow(self.q)) / self.q).mean()
# q→0 = CE, q→1 = MAE. 0.5~0.7 권장
```

### Pattern 4: Symmetric Cross Entropy
```python
class SCELoss(torch.nn.Module):
    def __init__(self, alpha=0.1, beta=1.0, num_classes=10):
        super().__init__(); self.a, self.b, self.K = alpha, beta, num_classes
    def forward(self, logits, targets):
        probs = F.softmax(logits, dim=-1).clamp(1e-7, 1.0)
        oh = F.one_hot(targets, self.K).float().clamp(1e-4, 1.0)
        ce = F.cross_entropy(logits, targets)
        rce = -(probs * oh.log()).sum(dim=1).mean()  # Reverse CE
        return self.a * ce + self.b * rce
```

### Pattern 5: Co-teaching (small-loss trick)
```python
def co_teaching_step(net1, net2, x, y, opt1, opt2, forget_rate):
    logits1, logits2 = net1(x), net2(x)
    losses1 = F.cross_entropy(logits1, y, reduction="none")
    losses2 = F.cross_entropy(logits2, y, reduction="none")
    # 서로의 small-loss sample 만 학습에 사용
    keep = int(len(y) * (1 - forget_rate))
    idx1 = losses1.argsort()[:keep]
    idx2 = losses2.argsort()[:keep]
    opt1.zero_grad(); F.cross_entropy(net1(x[idx2]), y[idx2]).backward(); opt1.step()
    opt2.zero_grad(); F.cross_entropy(net2(x[idx1]), y[idx1]).backward(); opt2.step()
```

### Pattern 6: Label smoothing as mild regularization
```python
loss = F.cross_entropy(logits, y, label_smoothing=0.1)
# 0.05~0.15. 가벼운 noise 에 효과 있음, heavy noise 엔 부족
```

### Pattern 7: MixUp (간접적 robustness)
```python
def mixup(x, y, alpha=0.2):
    lam = np.random.beta(alpha, alpha)
    idx = torch.randperm(x.size(0))
    return lam * x + (1 - lam) * x[idx], y, y[idx], lam

x_m, ya, yb, lam = mixup(x, y)
loss = lam * F.cross_entropy(model(x_m), ya) + (1 - lam) * F.cross_entropy(model(x_m), yb)
```

### Pattern 8: Cleanlab + confident learning + retraining loop
```python
issues = find_label_issues(y, pred_probs)
clean_idx = [i for i in range(len(y)) if i not in issues]
model.fit(X[clean_idx], y[clean_idx])  # 또는 issues 를 사람에게 review
```

## 매 결정 기준

| 상황 | Approach |
|---|---|
| 의심 label 찾아서 사람이 검토 | cleanlab |
| 라벨 수정 못 함, 학습만 가능 | GCE / SCE |
| 메모리/compute 충분 | Co-teaching (2 model) |
| 가벼운 noise (~5%) | Label smoothing + MixUp |
| Heavy noise (>30%) | GCE + cleanlab filter |

**기본값**: cleanlab 으로 1차 정리 + GCE loss + label smoothing.

## 🔗 Graph
- 부모: [[Machine_Learning]], [[Data_Quality]]
- 변형: [[Active Learning]]
- 응용: [[Image-Classification-Mastery]]
- Adjacent: [[LLM_Ops_and_Tuning]]

## 🤖 LLM 활용
**언제**: 의심 라벨 sample 을 LLM 으로 재라벨 (annotator 보조), 라벨링 가이드 자동 생성.
**언제 X**: high-stakes ground truth (의료, 법률) — 사람 검증 필수.

## ❌ 안티패턴
- 의심 label 자동 삭제 후 사람 검토 X → 진짜 hard sample 손실
- CE 만 쓰고 noise 30% 데이터 학습 → overfitting to noise
- Train/val 둘 다 noisy → eval 자체가 거짓말 (clean test set 분리 필수)
- cleanlab 한 번 돌리고 끝 → 모델 개선되면 다시 돌려야 함
- Co-teaching 에 동일 모델 2개 → diversity 0, 효과 없음

## 🧪 검증 / 중복
- Verified (cleanlab Northcutt 2021 confident learning, GCE Zhang 2018, SCE Wang 2019, Co-teaching Han 2018). 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — data-centric vs model-centric, GCE/SCE/co-teaching code |