---
id: wiki-2026-0508-information-theory
title: Information Theory
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Shannon, entropy, KL divergence, mutual information, MDL, cross-entropy]
duplicate_of: none
source_trust_level: A
confidence_score: 0.97
verification_status: applied
tags: [information-theory, shannon, entropy, kl-divergence, mutual-information, ml-theory]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Math / Python
  applicable_to: [ML, Compression, Comms]
---

# Information Theory

## 매 한 줄
> **"매 information 의 의 quantify 의 의 의 framework"**. Shannon 1948. 매 entropy, KL, mutual info, channel capacity. 매 ML 의 의: 매 cross-entropy loss, 매 information bottleneck, 매 MDL. 매 modern: 매 LLM scaling laws + compression analogy (Kaplan).

## 매 핵심

### 매 quantities
- **Entropy** H(X) = -Σ p log p. 매 uncertainty.
- **Cross-entropy** H(p, q) = -Σ p log q.
- **KL divergence** KL(p||q) = Σ p log(p/q). 매 not symmetric.
- **Mutual info** I(X; Y) = H(X) - H(X|Y).
- **Channel capacity** = max I(X; Y).

### 매 ML 응용
1. **Cross-entropy loss** (classification).
2. **KL** (VAE, RL, distillation).
3. **Mutual info** (feature selection, IB).
4. **Information bottleneck** (Tishby).
5. **MDL** (model selection).
6. **Compression ratio** (LLM Kaplan).

### 매 modern context
- **LLM 의 의 의 의 compressor** (Delétang 2023).
- **Scaling laws** = compression efficiency.
- **MDL principle** in deep learning.

## 💻 패턴

### Entropy
```python
import numpy as np
def entropy(p):
    p = np.asarray(p)
    p = p[p > 0]
    return -(p * np.log2(p)).sum()
```

### Cross-entropy
```python
def cross_entropy(p, q):
    p, q = np.asarray(p), np.asarray(q)
    return -(p * np.log2(q + 1e-9)).sum()
```

### KL divergence
```python
def kl_divergence(p, q):
    p, q = np.asarray(p), np.asarray(q)
    mask = p > 0
    return (p[mask] * np.log2(p[mask] / q[mask])).sum()
```

### Mutual information (sklearn)
```python
from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X, y)
```

### MI (custom, discrete)
```python
def mutual_info_discrete(x, y):
    """매 X, Y discrete."""
    from collections import Counter
    p_xy = Counter(zip(x, y))
    p_x = Counter(x); p_y = Counter(y)
    n = len(x)
    mi = 0
    for (xi, yi), nxy in p_xy.items():
        p_x_y = nxy / n
        p_x_marg = p_x[xi] / n
        p_y_marg = p_y[yi] / n
        mi += p_x_y * np.log2(p_x_y / (p_x_marg * p_y_marg))
    return mi
```

### Cross-entropy loss (PyTorch)
```python
import torch.nn.functional as F
loss = F.cross_entropy(logits, labels)  # 매 internally: log_softmax + nll_loss
```

### KL (VAE)
```python
def kl_normal(mu, logvar):
    """매 KL(N(mu, var) || N(0, 1))."""
    return -0.5 * (1 + logvar - mu.pow(2) - logvar.exp()).mean()
```

### Knowledge distillation (KL)
```python
def distillation_loss(student_logits, teacher_logits, T=3, alpha=0.5, labels=None):
    soft = F.kl_div(F.log_softmax(student_logits / T, -1),
                    F.softmax(teacher_logits / T, -1),
                    reduction='batchmean') * T * T
    hard = F.cross_entropy(student_logits, labels) if labels is not None else 0
    return alpha * soft + (1 - alpha) * hard
```

### Information Bottleneck (Tishby)
```python
def ib_loss(x, t, y, beta=0.1):
    """매 min I(T; X) - β I(T; Y)."""
    return mi(t, x) - beta * mi(t, y)
```

### MDL (model selection)
```python
def mdl(model_complexity_bits, data_likelihood_bits):
    """매 best model = 매 lowest total."""
    return model_complexity_bits + data_likelihood_bits
```

### Channel capacity (binary symmetric)
```python
def bsc_capacity(p):
    """매 BSC with crossover prob p."""
    if p == 0 or p == 1: return 1
    return 1 - (-(p * np.log2(p)) - ((1-p) * np.log2(1-p)))
```

### Conditional entropy
```python
def conditional_entropy(y, x):
    """매 H(Y|X)."""
    h_y = entropy(np.bincount(y) / len(y))
    h_xy = entropy(np.bincount(x * 10 + y) / len(x))
    h_x = entropy(np.bincount(x) / len(x))
    return h_xy - h_x
```

### LLM compression (Delétang 2023)
```python
def llm_compression_ratio(text, llm):
    """매 매 LLM 의 의 의 compressor."""
    log_probs = llm.compute_log_prob(text)
    bits_per_byte = -log_probs.sum() / (len(text.encode('utf-8')) * 8)
    return bits_per_byte  # 매 lower = better compression
```

### Feature selection (MI)
```python
from sklearn.feature_selection import mutual_info_regression
mi = mutual_info_regression(X, y)
top_features = X.columns[np.argsort(mi)[-10:]]
```

### JS divergence (symmetric)
```python
def js_divergence(p, q):
    m = (p + q) / 2
    return 0.5 * kl_divergence(p, m) + 0.5 * kl_divergence(q, m)
```

### Population Stability Index (drift)
```python
def psi(expected, actual, bins=10):
    e_hist, edges = np.histogram(expected, bins=bins)
    a_hist, _ = np.histogram(actual, bins=edges)
    e_pct = e_hist / len(expected) + 1e-9
    a_pct = a_hist / len(actual) + 1e-9
    return ((a_pct - e_pct) * np.log(a_pct / e_pct)).sum()
```

### Free energy (variational)
```python
def free_energy(q_log_prob, p_log_prob, log_likelihood):
    """매 F = E_q[log q] - E_q[log p] - E_q[log p(x|z)]."""
    return q_log_prob.mean() - p_log_prob.mean() - log_likelihood.mean()
```

## 매 결정 기준
| 상황 | Quantity |
|---|---|
| Classification loss | Cross-entropy |
| Distillation | KL |
| Feature selection | Mutual info |
| Drift detection | PSI / KL |
| Model selection | MDL |
| VAE | ELBO (KL + likelihood) |

**기본값**: 매 cross-entropy classification + 매 KL for distill / VAE / RL + 매 MI for features + 매 PSI for drift.

## 🔗 Graph
- 부모: [[Math]] · [[Statistics]]
- 변형: [[Shannon-Entropy]] · [[Mutual-Information]]
- 응용: [[Cross-Entropy-Loss]] · [[KL-Divergence]]
- Adjacent: [[MDL]] · [[Free-Energy-Principle]] · [[Foundation-Models]]

## 🤖 LLM 활용
**언제**: 매 ML loss design. 매 compression. 매 feature selection.
**언제 X**: 매 trivial baseline.

## ❌ 안티패턴
- **Symmetric KL assumed**: 매 KL 의 not symmetric.
- **MI for high-D**: 매 estimation hard.
- **Log natural vs log2 confusion**: 매 unit ↔.
- **Cross-entropy without softmax**: 매 PyTorch CE 의 internal.

## 🧪 검증 / 중복
- Verified (Shannon 1948, Cover & Thomas).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — quantities + 매 entropy / KL / MI / IB / PSI / VAE code |
