2nd/10_Wiki/Topics/AI_and_ML/Data Distillation (데이터 증류).md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-data-distillation	Data Distillation	10_Wiki/Topics	verified	self	data distillation dataset distillation synthetic data knowledge distillation sLLM self-distillation	none	A	0.88	applied	data-efficiency dataset-distillation knowledge-distillation llm sllm self-distillation synthetic		2026-05-10	pending	language framework Python PyTorch / Diffusers

Data Distillation

매 한 줄

"매 1000 의 image 의 매 10 의 essence". 매 huge dataset → 매 tiny synthetic. 매 model 의 same performance. 매 modern: 매 sLLM (small LLM) 의 trend — 매 GPT-4 의 distill 의 매 7B model. 매 Phi-3, 매 distillation 의 frontier.

매 핵심

매 vs Knowledge Distillation

• Knowledge Distillation (Hinton 2015): 매 teacher model → 매 student model.
• Data Distillation (Wang 2018): 매 dataset → 매 small synthetic dataset.
• 둘 다 의 commonly combine.

매 dataset distillation methods

Gradient Matching (Zhao 2021, DC)

• 매 synthetic 의 train 의 gradient ≈ 매 real 의.

Distribution Matching (DM)

• 매 feature distribution 의 match.

Trajectory Matching (MTT, Cazenavette 2022)

• 매 training trajectory 의 match.

Generative

• 매 GAN / Diffusion 의 generate small.

매 modern (LLM era)

Self-Distillation

• 매 large model 의 generate → 매 small model 의 train.
• 매 Phi-3 (Microsoft).
• 매 Llama-3 의 cleaning.

Distillation from Frontier

• 매 GPT-4 의 reasoning trace → 매 7B fine-tune.
• 매 ChatGPT → Vicuna, Alpaca.

Quality > Quantity

• 매 LIMA paper (Zhou 2023): 매 1K example 의 powerful.
• 매 careful curation > 매 raw scrape.

매 응용

1. Edge ML: 매 small training data.
2. Privacy: 매 raw data 의 X.
3. Continual learning: 매 replay 의 compact.
4. Architecture search: 매 fast NAS.
5. sLLM: 매 small but capable.

매 trade-off

• Compression ratio: 매 1% size 매 95% performance.
• Generalization to new architecture: 매 weak (architecture-specific).
• Data privacy + 매 utility 의 balance.

💻 패턴

Knowledge distillation (model)

import torch
import torch.nn.functional as F

def distill_loss(student_logits, teacher_logits, labels, T=4, alpha=0.7):
    soft_loss = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.softmax(teacher_logits / T, dim=-1),
        reduction='batchmean',
    ) * T * T
    hard_loss = F.cross_entropy(student_logits, labels)
    return alpha * soft_loss + (1 - alpha) * hard_loss

# 매 training
for x, y in loader:
    with torch.no_grad():
        teacher_out = teacher(x)
    student_out = student(x)
    loss = distill_loss(student_out, teacher_out, y)
    loss.backward()

Dataset distillation (Gradient Matching, simplified)

def gradient_matching(real_loader, synthetic_data, model, n_iters=1000):
    """매 매 step 의 synthetic 의 update 의 real 의 gradient 와 의 match."""
    syn_x = torch.randn(N_SYN, C, H, W, requires_grad=True)
    syn_y = torch.randint(0, n_classes, (N_SYN,))
    
    for it in range(n_iters):
        # 매 real gradient
        real_x, real_y = next(iter(real_loader))
        real_loss = F.cross_entropy(model(real_x), real_y)
        real_grad = torch.autograd.grad(real_loss, model.parameters())
        
        # 매 synthetic gradient
        syn_loss = F.cross_entropy(model(syn_x), syn_y)
        syn_grad = torch.autograd.grad(syn_loss, model.parameters(), create_graph=True)
        
        # 매 match
        match_loss = sum((rg - sg).pow(2).mean() for rg, sg in zip(real_grad, syn_grad))
        syn_x.grad = torch.autograd.grad(match_loss, syn_x)[0]
        syn_x.data -= LR * syn_x.grad
    
    return syn_x, syn_y

Self-distillation (LLM)

def self_distill_pipeline(teacher_model, base_student, prompts, n_examples=10000):
    """매 teacher 의 generate → 매 student 의 train."""
    
    # 매 1. teacher 의 generate (with reasoning)
    examples = []
    for prompt in prompts[:n_examples]:
        response = teacher_model.generate(
            prompt + "\n\nThink step by step.",
            max_tokens=2048,
        )
        examples.append({'prompt': prompt, 'response': response})
    
    # 매 2. quality filter
    examples = filter_quality(examples)
    
    # 매 3. SFT student
    train_sft(base_student, examples)
    
    return base_student

LIMA-style curated data

def lima_curate(candidate_dataset, n_keep=1000):
    """매 quality > quantity. 매 careful manual + 매 LLM filter."""
    
    # 매 1. format check
    candidates = [c for c in candidate_dataset if format_valid(c)]
    
    # 매 2. quality filter (LLM-as-judge)
    scored = []
    for c in candidates:
        score = llm_quality_judge(c)
        scored.append((c, score))
    
    # 매 3. diversity sample
    scored.sort(key=lambda x: -x[1])
    diverse_top = diverse_sample(scored, n=n_keep)  # 매 cluster + 매 1 per cluster
    
    return [c for c, _ in diverse_top]

Synthetic data generation (LLM)

def generate_synthetic_qa(domain, n=10000):
    seed_examples = load_seed(domain)
    
    synthetic = []
    for _ in range(n):
        prompt = f"""Generate a high-quality Q&A pair about {domain}.

Style examples:
{format_examples(seed_examples[:3])}

Output format:
Q: ...
A: ...

Generate a NEW Q&A:"""
        result = llm.generate(prompt)
        synthetic.append(parse_qa(result))
    
    return synthetic

Compression evaluation

def evaluate_distilled_dataset(distilled, full_test):
    """매 매 architecture 의 evaluate."""
    architectures = ['ResNet18', 'VGG11', 'ConvNet']
    results = {}
    
    for arch in architectures:
        model = create(arch)
        train(model, distilled)
        results[arch] = evaluate(model, full_test)
    
    return results  # 매 cross-arch generalization

Replay buffer compression (continual learning)

class CompressedReplay:
    def __init__(self, capacity=100, compression_method='distill'):
        self.buffer = []
        self.capacity = capacity
        self.method = compression_method
    
    def add_task(self, task_data):
        if self.method == 'distill':
            distilled = dataset_distill(task_data, n_per_class=10)
            self.buffer.append(distilled)
        else:
            # 매 random subsample
            self.buffer.append(random.sample(task_data, self.capacity // len(self.buffer)))

Privacy via DP-distillation

def dp_distill(real_data, epsilon=1.0):
    """매 differentially private 의 distill."""
    # 매 add noise to gradients
    syn_data = init_synthetic()
    for step in range(N_STEPS):
        grad = compute_gradient_match(syn_data, real_data)
        noise = laplace_noise(scale=1/epsilon)
        grad_private = grad + noise
        syn_data -= LR * grad_private
    return syn_data

매 결정 기준

상황	Approach
Tiny budget	Dataset distillation (Gradient Matching)
Small LLM	Self-distillation from frontier
Privacy	DP-distillation
Continual learning	Compressed replay
Quality + small data	LIMA curate
Synthetic gen	LLM-driven
Edge deploy	Knowledge distill (model)

기본값: Self-distillation from larger model (LLM era).

🔗 Graph

• 부모: LLM_Optimization_and_Deployment_Strategies
• 변형: Dataset-Distillation · Self-Distillation · Synthetic-Data
• 응용: sLLM · Continual-Learning
• Adjacent: Cross-Entropy Loss · Catastrophic-Forgetting · Computational_Creativity · CV_Synthesis

🤖 LLM 활용

언제: 매 sLLM training. 매 cost-efficient deploy. 매 privacy-sensitive. 매 continual replay. 언제 X: 매 abundant real data + cheap compute.

❌ 안티패턴

• Synthetic only (no real validation): 매 model collapse risk.
• Architecture-specific distill 의 cross-architecture expect: 매 fail.
• No quality filter: 매 noise amplify.
• Frontier 의 imitation 의 commercial use: 매 ToS violation (some).
• DP epsilon 의 too low: 매 utility lose.

🧪 검증 / 중복

• Verified (Wang 2018 dataset distillation, Hinton 2015 KD, Phi-3 paper, LIMA paper).
• 신뢰도 A.
• Related: Cross-Entropy Loss · Catastrophic-Forgetting · Computational_Creativity · CV_Synthesis · Cost-Benefit Analysis in AI.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — methods + 매 KD / GM / self-distill / LIMA / DP code