2nd/10_Wiki/Topics/AI_and_ML/Catastrophic-Forgetting.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-catastrophic-forgetting	Catastrophic Forgetting & Continual Learning	10_Wiki/Topics	verified	self	파괴적 망각 continual learning lifelong learning EWC replay buffer LoRA mixture of experts	none	A	0.93	applied	continual-learning catastrophic-forgetting ewc replay transfer-learning lifelong-learning lora llm-finetune		2026-05-10	pending	language framework Python PyTorch / Avalanche / continual-learning libs

Catastrophic Forgetting

📌 한 줄 통찰

"매 new task 의 학습 의 매 old 의 destroy". 매 NN 의 weight 의 overwrite. 매 lifelong learning 의 fundamental challenge. 매 modern LLM era 의 highly relevant — 매 fine-tune 의 base capability 의 lose.

📖 핵심

매 mechanism

• 매 SGD 의 모든 weight 의 update.
• 매 same weight 가 매 multiple task 의 store.
• 매 new task 의 gradient 의 old 의 wipe.

매 3 approach (Continual Learning)

1. Regularization-based

• EWC (Elastic Weight Consolidation): 매 past 의 important weight 의 protect.
• SI (Synaptic Intelligence).
• LwF (Learning without Forgetting): 매 distillation.

2. Replay-based

• Experience replay: 매 old data 의 sample.
• Generative replay (DGR): 매 generative model 의 old 의 reconstruct.
• Reservoir sampling.

3. Architecture-based

• Progressive Networks: 매 column 의 add.
• PackNet: 매 weight 의 mask.
• Dynamic expansion.

4. Modern (LLM)

• LoRA: 매 base 의 frozen + 매 adapter 의 train.
• Adapter modules.
• Mixture of Experts (MoE).
• Soft prompt tuning.

매 evaluation metric

• Average accuracy: 매 모든 past task.
• Backward transfer (BWT): 매 old task 의 degradation.
• Forward transfer (FWT): 매 new task 의 boost.
• Forgetting rate.

매 setting

• Class-incremental: 매 new class.
• Task-incremental: 매 distinct task.
• Domain-incremental: 매 same task, 매 new domain.
• Online: 매 stream.

매 modern LLM 의 응용

1. Fine-tune drift: 매 helpful 의 acquire 가, 매 reasoning 의 lose.
2. Domain adapt: 매 medical fine-tune 가, 매 general 의 weak.
3. RLHF: 매 alignment tax.
4. Continual pretraining: 매 new knowledge.

→ 매 LoRA 의 popular reason: 매 base 의 keep.

매 lib

• Avalanche (PyTorch): 매 best.
• Continual-AI: 매 community.
• Mammoth.

매 biological 의 inspiration

• 매 brain 의 hippocampus 의 fast learning + 매 neocortex 의 consolidation.
• 매 sleep 의 replay 의 role (Bayesian brain).
• 매 modular brain.

💻 패턴

EWC (Elastic Weight Consolidation)

import torch
import torch.nn as nn

class EWC:
    def __init__(self, model, dataset, lambda_=1000):
        self.model = model
        self.lambda_ = lambda_
        self.params = {n: p for n, p in model.named_parameters() if p.requires_grad}
        self.fisher = self._compute_fisher(dataset)
        self.opt_params = {n: p.data.clone() for n, p in self.params.items()}
    
    def _compute_fisher(self, dataset):
        fisher = {n: torch.zeros_like(p) for n, p in self.params.items()}
        self.model.eval()
        for x, y in dataset:
            self.model.zero_grad()
            output = self.model(x)
            loss = F.cross_entropy(output, y)
            loss.backward()
            for n, p in self.params.items():
                fisher[n] += p.grad.data.pow(2) / len(dataset)
        return fisher
    
    def penalty(self):
        loss = 0
        for n, p in self.params.items():
            loss += (self.fisher[n] * (p - self.opt_params[n]).pow(2)).sum()
        return self.lambda_ * loss

# 매 train new task with EWC
ewc = EWC(model, old_task_loader)
for x, y in new_task_loader:
    loss = F.cross_entropy(model(x), y) + ewc.penalty()
    loss.backward()
    optimizer.step()

Experience Replay

class ReplayBuffer:
    def __init__(self, capacity=10000):
        self.buffer = []
        self.capacity = capacity
    
    def add(self, data):
        if len(self.buffer) >= self.capacity:
            # 매 reservoir sampling
            idx = random.randint(0, len(self.buffer))
            if idx < self.capacity:
                self.buffer[idx] = data
        else:
            self.buffer.append(data)
    
    def sample(self, n):
        return random.sample(self.buffer, min(n, len(self.buffer)))

replay = ReplayBuffer()

# 매 train new task + 매 mix replay
for x, y in new_task_loader:
    new_loss = F.cross_entropy(model(x), y)
    
    if replay.buffer:
        replay_batch = replay.sample(BATCH_SIZE // 2)
        rx, ry = collate(replay_batch)
        replay_loss = F.cross_entropy(model(rx), ry)
        loss = new_loss + replay_loss
    else:
        loss = new_loss
    
    loss.backward()
    optimizer.step()
    
    # 매 store new for future
    for x_i, y_i in zip(x, y):
        replay.add((x_i, y_i))

LoRA (modern, LLM-friendly)

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3-8B')

# 매 task 1 의 LoRA
lora_t1 = LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj'])
model_t1 = get_peft_model(base, lora_t1)
train(model_t1, task1_data)
model_t1.save_pretrained('./lora-task1')

# 매 task 2 — 매 base 의 fresh + 매 다른 LoRA
model_t2 = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj']))
train(model_t2, task2_data)
model_t2.save_pretrained('./lora-task2')

# 매 inference 시 의 swap
def serve(prompt, task):
    base.load_adapter(f'./lora-{task}')
    return base.generate(prompt)

→ 매 base 의 untouched — 매 forgetting X.

Generative Replay

def generative_replay(generator, classifier, new_task_loader):
    for x, y in new_task_loader:
        # 매 new task loss
        new_loss = F.cross_entropy(classifier(x), y)
        
        # 매 old replay (generated)
        n_replay = x.size(0)
        replay_x = generator.sample(n_replay)
        replay_y = classifier_old(replay_x).argmax(-1)  # 매 old 의 prediction 의 supervise
        replay_loss = F.cross_entropy(classifier(replay_x), replay_y)
        
        loss = new_loss + 0.5 * replay_loss
        loss.backward()
        optimizer.step()

PackNet (architecture-based)

class PackNet:
    """매 weight 의 task 별 mask."""
    def __init__(self, model, prune_ratio=0.5):
        self.model = model
        self.task_masks = {}  # 매 task → 매 mask
    
    def train_task(self, task_id, loader):
        # 매 train normally
        train(self.model, loader)
        
        # 매 prune low-magnitude weights
        for name, p in self.model.named_parameters():
            threshold = p.abs().quantile(self.prune_ratio)
            mask = p.abs() > threshold
            self.task_masks[(task_id, name)] = mask
            p.data *= mask  # 매 freeze unmasked
    
    def forward_task(self, task_id, x):
        # 매 use only task's mask
        with torch.no_grad():
            for name, p in self.model.named_parameters():
                p.data *= self.task_masks[(task_id, name)]
        return self.model(x)

Continual eval (Avalanche)

from avalanche.benchmarks.classic import SplitMNIST
from avalanche.training import EWC
from avalanche.evaluation.metrics import accuracy_metrics, forgetting_metrics

scenario = SplitMNIST(n_experiences=5)
model = MyModel()
strategy = EWC(model, optimizer, criterion=F.cross_entropy, ewc_lambda=400)

for experience in scenario.train_stream:
    strategy.train(experience)
    results = strategy.eval(scenario.test_stream)
    print(f'Avg accuracy: {results["Top1_Acc_Stream/eval_phase/test_stream/Task000"]}')

LLM fine-tune drift detection

def detect_drift(base_model, finetuned_model, eval_set):
    """매 base capability 의 forgetting 의 measure."""
    base_scores = []
    ft_scores = []
    for example in eval_set:
        base_scores.append(score(base_model, example))
        ft_scores.append(score(finetuned_model, example))
    
    drift = np.mean(base_scores) - np.mean(ft_scores)
    if drift > 0.05:
        log(f'Significant capability loss: {drift:.3f}')
    return drift

🤔 결정 기준

상황	Approach
LLM fine-tune	LoRA / Adapter
Class-incremental	EWC + Replay
Streaming	Reservoir + Online EWC
Few-shot	Prompt tuning
Domain shift	Domain-adversarial
Strong constraint	Architecture-based (PackNet)
General	Replay (best)

기본값: LoRA / Adapter for LLM. Replay + EWC for vision.

🔗 Graph

• 부모: Continual-Learning
• 변형: EWC · Replay-Buffer
• 응용: LoRA · Adapter · Mixture-of-Experts
• Adjacent: Bayesian-Brain-Hypothesis · Biological-Intelligence · Bias vs Variance Trade-off · Auto-Encoding

🤖 LLM 활용

언제: 매 sequential task. 매 LLM domain adapt. 매 streaming data. 매 lifelong agent. 언제 X: 매 single static dataset. 매 IID assumption.

❌ 안티패턴

• Naive fine-tune: 매 catastrophic forgetting.
• No EWC / replay: 매 old task 의 lose.
• Replay buffer 의 unbounded: 매 storage 폭발.
• No drift measurement: 매 silent capability loss.
• Same LR for all task: 매 some 의 dominate.

🧪 검증 / 중복

• Verified (Kirkpatrick EWC 2017, Lopez-Paz GEM, Rebuffi iCaRL).
• 신뢰도 A.
• Related: LoRA · Mixture-of-Experts · Continual-Learning · Bayesian-Brain-Hypothesis · Biological-Intelligence.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — 3 approach + 매 EWC / replay / LoRA / PackNet code + LLM drift