2nd/10_Wiki/Topics/AI_and_ML/Label-Noise-and-Robustness.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-label-noise-and-robustness	Label Noise and Robustness	10_Wiki/Topics	verified	self	Noisy Labels Label Cleaning Confident Learning	none	A	0.9	applied	ml label-noise cleanlab robust-loss gce symmetric-ce		2026-05-10	pending	language framework python cleanlab/pytorch

Label Noise and Robustness

매 한 줄

"매 데이터셋의 라벨은 거짓말한다". Confident learning 으로 의심 라벨 찾고, robust loss 로 학습 자체를 노이즈에 둔감하게.

매 핵심

매 노이즈 종류

• Symmetric (uniform): 모든 클래스로 균등 flip
• Asymmetric (class-conditional): 비슷한 class 로 flip (cat→dog 高)
• Instance-dependent: 어려운 sample 일수록 noise 높음 (가장 현실적, 가장 어려움)

매 두 갈래

• Data-centric: 의심 라벨 찾아서 제거/수정 → cleanlab, confident learning
• Model-centric: 노이즈가 있어도 학습 강건 → robust loss (GCE, SCE), co-teaching, MixUp

매 응용

1. Crowdsourced 라벨 정리
2. Web-scraped dataset cleaning
3. Auto-label 후 검증
4. Class imbalance + noise 동시
5. Active learning 의 라벨 효율 극대화

💻 패턴

Pattern 1: cleanlab basic

from cleanlab.classification import CleanLearning
from sklearn.linear_model import LogisticRegression

cl = CleanLearning(clf=LogisticRegression())
cl.fit(X, y)  # 자동으로 의심 label 찾고 제거 후 재학습

# 또는 의심 label 만 받기
from cleanlab.filter import find_label_issues
pred_probs = model.predict_proba(X)
issues = find_label_issues(labels=y, pred_probs=pred_probs, return_indices_ranked_by="self_confidence")
print(f"의심 label {len(issues)} 개")

Pattern 2: Confident learning manual

# 1. K-fold OOF prediction
from sklearn.model_selection import cross_val_predict
pred_probs = cross_val_predict(model, X, y, cv=5, method="predict_proba")

# 2. cleanlab 으로 issue 탐지
from cleanlab import Datalab
lab = Datalab(data={"X": X, "y": y}, label_name="y")
lab.find_issues(pred_probs=pred_probs)
lab.report()

Pattern 3: Generalized Cross Entropy (GCE)

import torch, torch.nn.functional as F

class GCELoss(torch.nn.Module):
    def __init__(self, q=0.7):
        super().__init__()
        self.q = q
    def forward(self, logits, targets):
        probs = F.softmax(logits, dim=-1)
        p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1).clamp(min=1e-7)
        return ((1 - p_t.pow(self.q)) / self.q).mean()
# q→0 = CE, q→1 = MAE. 0.5~0.7 권장

Pattern 4: Symmetric Cross Entropy

class SCELoss(torch.nn.Module):
    def __init__(self, alpha=0.1, beta=1.0, num_classes=10):
        super().__init__(); self.a, self.b, self.K = alpha, beta, num_classes
    def forward(self, logits, targets):
        probs = F.softmax(logits, dim=-1).clamp(1e-7, 1.0)
        oh = F.one_hot(targets, self.K).float().clamp(1e-4, 1.0)
        ce = F.cross_entropy(logits, targets)
        rce = -(probs * oh.log()).sum(dim=1).mean()  # Reverse CE
        return self.a * ce + self.b * rce

Pattern 5: Co-teaching (small-loss trick)

def co_teaching_step(net1, net2, x, y, opt1, opt2, forget_rate):
    logits1, logits2 = net1(x), net2(x)
    losses1 = F.cross_entropy(logits1, y, reduction="none")
    losses2 = F.cross_entropy(logits2, y, reduction="none")
    # 서로의 small-loss sample 만 학습에 사용
    keep = int(len(y) * (1 - forget_rate))
    idx1 = losses1.argsort()[:keep]
    idx2 = losses2.argsort()[:keep]
    opt1.zero_grad(); F.cross_entropy(net1(x[idx2]), y[idx2]).backward(); opt1.step()
    opt2.zero_grad(); F.cross_entropy(net2(x[idx1]), y[idx1]).backward(); opt2.step()

Pattern 6: Label smoothing as mild regularization

loss = F.cross_entropy(logits, y, label_smoothing=0.1)
# 0.05~0.15. 가벼운 noise 에 효과 있음, heavy noise 엔 부족

Pattern 7: MixUp (간접적 robustness)

def mixup(x, y, alpha=0.2):
    lam = np.random.beta(alpha, alpha)
    idx = torch.randperm(x.size(0))
    return lam * x + (1 - lam) * x[idx], y, y[idx], lam

x_m, ya, yb, lam = mixup(x, y)
loss = lam * F.cross_entropy(model(x_m), ya) + (1 - lam) * F.cross_entropy(model(x_m), yb)

Pattern 8: Cleanlab + confident learning + retraining loop

issues = find_label_issues(y, pred_probs)
clean_idx = [i for i in range(len(y)) if i not in issues]
model.fit(X[clean_idx], y[clean_idx])  # 또는 issues 를 사람에게 review

매 결정 기준

상황	Approach
의심 label 찾아서 사람이 검토	cleanlab
라벨 수정 못 함, 학습만 가능	GCE / SCE
메모리/compute 충분	Co-teaching (2 model)
가벼운 noise (~5%)	Label smoothing + MixUp
Heavy noise (>30%)	GCE + cleanlab filter

기본값: cleanlab 으로 1차 정리 + GCE loss + label smoothing.

🔗 Graph

• 부모: Machine_Learning, Data_Quality
• 변형: Active Learning
• 응용: Image-Classification-Mastery
• Adjacent: LLM_Ops_and_Tuning

🤖 LLM 활용

언제: 의심 라벨 sample 을 LLM 으로 재라벨 (annotator 보조), 라벨링 가이드 자동 생성. 언제 X: high-stakes ground truth (의료, 법률) — 사람 검증 필수.

❌ 안티패턴

• 의심 label 자동 삭제 후 사람 검토 X → 진짜 hard sample 손실
• CE 만 쓰고 noise 30% 데이터 학습 → overfitting to noise
• Train/val 둘 다 noisy → eval 자체가 거짓말 (clean test set 분리 필수)
• cleanlab 한 번 돌리고 끝 → 모델 개선되면 다시 돌려야 함
• Co-teaching 에 동일 모델 2개 → diversity 0, 효과 없음

🧪 검증 / 중복

• Verified (cleanlab Northcutt 2021 confident learning, GCE Zhang 2018, SCE Wang 2019, Co-teaching Han 2018). 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — data-centric vs model-centric, GCE/SCE/co-teaching code