---
id: wiki-2026-0508-data-distillation
title: Data Distillation
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [data distillation, dataset distillation, synthetic data, knowledge distillation, sLLM, self-distillation]
duplicate_of: none
source_trust_level: A
confidence_score: 0.88
verification_status: applied
tags: [data-efficiency, dataset-distillation, knowledge-distillation, llm, sllm, self-distillation, synthetic]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: PyTorch / Diffusers
---

# Data Distillation

## 매 한 줄
> **"매 1000 의 image 의 매 10 의 essence"**. 매 huge dataset → 매 tiny synthetic. 매 model 의 same performance. 매 modern: 매 sLLM (small LLM) 의 trend — 매 GPT-4 의 distill 의 매 7B model. 매 Phi-3, 매 distillation 의 frontier.

## 매 핵심

### 매 vs Knowledge Distillation
- **Knowledge Distillation** (Hinton 2015): 매 teacher model → 매 student model.
- **Data Distillation** (Wang 2018): 매 dataset → 매 small synthetic dataset.
- **둘 다** 의 commonly combine.

### 매 dataset distillation methods

#### Gradient Matching (Zhao 2021, DC)
- 매 synthetic 의 train 의 gradient ≈ 매 real 의.

#### Distribution Matching (DM)
- 매 feature distribution 의 match.

#### Trajectory Matching (MTT, Cazenavette 2022)
- 매 training trajectory 의 match.

#### Generative
- 매 GAN / Diffusion 의 generate small.

### 매 modern (LLM era)

#### Self-Distillation
- 매 large model 의 generate → 매 small model 의 train.
- 매 Phi-3 (Microsoft).
- 매 Llama-3 의 cleaning.

#### Distillation from Frontier
- 매 GPT-4 의 reasoning trace → 매 7B fine-tune.
- 매 ChatGPT → Vicuna, Alpaca.

#### Quality > Quantity
- 매 LIMA paper (Zhou 2023): 매 1K example 의 powerful.
- 매 careful curation > 매 raw scrape.

### 매 응용
1. **Edge ML**: 매 small training data.
2. **Privacy**: 매 raw data 의 X.
3. **Continual learning**: 매 replay 의 compact.
4. **Architecture search**: 매 fast NAS.
5. **sLLM**: 매 small but capable.

### 매 trade-off
- **Compression ratio**: 매 1% size 매 95% performance.
- **Generalization to new architecture**: 매 weak (architecture-specific).
- **Data privacy** + 매 utility 의 balance.

## 💻 패턴

### Knowledge distillation (model)
```python
import torch
import torch.nn.functional as F

def distill_loss(student_logits, teacher_logits, labels, T=4, alpha=0.7):
    soft_loss = F.kl_div(
        F.log_softmax(student_logits / T, dim=-1),
        F.softmax(teacher_logits / T, dim=-1),
        reduction='batchmean',
    ) * T * T
    hard_loss = F.cross_entropy(student_logits, labels)
    return alpha * soft_loss + (1 - alpha) * hard_loss

# 매 training
for x, y in loader:
    with torch.no_grad():
        teacher_out = teacher(x)
    student_out = student(x)
    loss = distill_loss(student_out, teacher_out, y)
    loss.backward()
```

### Dataset distillation (Gradient Matching, simplified)
```python
def gradient_matching(real_loader, synthetic_data, model, n_iters=1000):
    """매 매 step 의 synthetic 의 update 의 real 의 gradient 와 의 match."""
    syn_x = torch.randn(N_SYN, C, H, W, requires_grad=True)
    syn_y = torch.randint(0, n_classes, (N_SYN,))
    
    for it in range(n_iters):
        # 매 real gradient
        real_x, real_y = next(iter(real_loader))
        real_loss = F.cross_entropy(model(real_x), real_y)
        real_grad = torch.autograd.grad(real_loss, model.parameters())
        
        # 매 synthetic gradient
        syn_loss = F.cross_entropy(model(syn_x), syn_y)
        syn_grad = torch.autograd.grad(syn_loss, model.parameters(), create_graph=True)
        
        # 매 match
        match_loss = sum((rg - sg).pow(2).mean() for rg, sg in zip(real_grad, syn_grad))
        syn_x.grad = torch.autograd.grad(match_loss, syn_x)[0]
        syn_x.data -= LR * syn_x.grad
    
    return syn_x, syn_y
```

### Self-distillation (LLM)
```python
def self_distill_pipeline(teacher_model, base_student, prompts, n_examples=10000):
    """매 teacher 의 generate → 매 student 의 train."""
    
    # 매 1. teacher 의 generate (with reasoning)
    examples = []
    for prompt in prompts[:n_examples]:
        response = teacher_model.generate(
            prompt + "\n\nThink step by step.",
            max_tokens=2048,
        )
        examples.append({'prompt': prompt, 'response': response})
    
    # 매 2. quality filter
    examples = filter_quality(examples)
    
    # 매 3. SFT student
    train_sft(base_student, examples)
    
    return base_student
```

### LIMA-style curated data
```python
def lima_curate(candidate_dataset, n_keep=1000):
    """매 quality > quantity. 매 careful manual + 매 LLM filter."""
    
    # 매 1. format check
    candidates = [c for c in candidate_dataset if format_valid(c)]
    
    # 매 2. quality filter (LLM-as-judge)
    scored = []
    for c in candidates:
        score = llm_quality_judge(c)
        scored.append((c, score))
    
    # 매 3. diversity sample
    scored.sort(key=lambda x: -x[1])
    diverse_top = diverse_sample(scored, n=n_keep)  # 매 cluster + 매 1 per cluster
    
    return [c for c, _ in diverse_top]
```

### Synthetic data generation (LLM)
```python
def generate_synthetic_qa(domain, n=10000):
    seed_examples = load_seed(domain)
    
    synthetic = []
    for _ in range(n):
        prompt = f"""Generate a high-quality Q&A pair about {domain}.

Style examples:
{format_examples(seed_examples[:3])}

Output format:
Q: ...
A: ...

Generate a NEW Q&A:"""
        result = llm.generate(prompt)
        synthetic.append(parse_qa(result))
    
    return synthetic
```

### Compression evaluation
```python
def evaluate_distilled_dataset(distilled, full_test):
    """매 매 architecture 의 evaluate."""
    architectures = ['ResNet18', 'VGG11', 'ConvNet']
    results = {}
    
    for arch in architectures:
        model = create(arch)
        train(model, distilled)
        results[arch] = evaluate(model, full_test)
    
    return results  # 매 cross-arch generalization
```

### Replay buffer compression (continual learning)
```python
class CompressedReplay:
    def __init__(self, capacity=100, compression_method='distill'):
        self.buffer = []
        self.capacity = capacity
        self.method = compression_method
    
    def add_task(self, task_data):
        if self.method == 'distill':
            distilled = dataset_distill(task_data, n_per_class=10)
            self.buffer.append(distilled)
        else:
            # 매 random subsample
            self.buffer.append(random.sample(task_data, self.capacity // len(self.buffer)))
```

### Privacy via DP-distillation
```python
def dp_distill(real_data, epsilon=1.0):
    """매 differentially private 의 distill."""
    # 매 add noise to gradients
    syn_data = init_synthetic()
    for step in range(N_STEPS):
        grad = compute_gradient_match(syn_data, real_data)
        noise = laplace_noise(scale=1/epsilon)
        grad_private = grad + noise
        syn_data -= LR * grad_private
    return syn_data
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Tiny budget | Dataset distillation (Gradient Matching) |
| Small LLM | Self-distillation from frontier |
| Privacy | DP-distillation |
| Continual learning | Compressed replay |
| Quality + small data | LIMA curate |
| Synthetic gen | LLM-driven |
| Edge deploy | Knowledge distill (model) |

**기본값**: Self-distillation from larger model (LLM era).

## 🔗 Graph
- 부모: [[ML-Training]] · [[Knowledge-Distillation]] · [[Data-Efficiency]]
- 변형: [[Dataset-Distillation]] · [[Self-Distillation]] · [[LIMA]] · [[Synthetic-Data]]
- 응용: [[Phi-3]] · [[Vicuna]] · [[sLLM]] · [[Continual-Learning]]
- Adjacent: [[Cross-Entropy Loss]] · [[Catastrophic-Forgetting]] · [[Computational-Creativity]] · [[CV_Synthesis]]

## 🤖 LLM 활용
**언제**: 매 sLLM training. 매 cost-efficient deploy. 매 privacy-sensitive. 매 continual replay.
**언제 X**: 매 abundant real data + cheap compute.

## ❌ 안티패턴
- **Synthetic only (no real validation)**: 매 model collapse risk.
- **Architecture-specific distill 의 cross-architecture expect**: 매 fail.
- **No quality filter**: 매 noise amplify.
- **Frontier 의 imitation 의 commercial use**: 매 ToS violation (some).
- **DP epsilon 의 too low**: 매 utility lose.

## 🧪 검증 / 중복
- Verified (Wang 2018 dataset distillation, Hinton 2015 KD, Phi-3 paper, LIMA paper).
- 신뢰도 A.
- Related: [[Cross-Entropy Loss]] · [[Catastrophic-Forgetting]] · [[Computational-Creativity]] · [[CV_Synthesis]] · [[Cost-Benefit Analysis in AI]].

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — methods + 매 KD / GM / self-distill / LIMA / DP code |