---
id: wiki-2026-0508-entropy-in-information-theory
title: Entropy in Information Theory
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Shannon Entropy, Information Entropy, H(X)]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [information-theory, probability, entropy, shannon, machine-learning]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: NumPy / SciPy
---

# Entropy in Information Theory

## 매 한 줄
> **"매 random variable 의 uncertainty 의 quantification — H(X) = -Σp(x)log p(x)"**. Shannon 1948 "A Mathematical Theory of Communication" 의 birth — 매 modern compression, channel coding, ML loss function (cross-entropy), variational inference 의 모두 의 foundation.

## 매 핵심

### 매 정의
- **Shannon entropy**: H(X) = -Σ p(x) log p(x), unit log_2 의 bit / log_e 의 nat.
- **Joint entropy**: H(X,Y) = -ΣΣ p(x,y) log p(x,y).
- **Conditional entropy**: H(Y|X) = H(X,Y) - H(X) = E_X[H(Y|X=x)].
- **Mutual information**: I(X;Y) = H(X) + H(Y) - H(X,Y) = H(X) - H(X|Y).
- **KL divergence**: D_KL(P||Q) = Σ p(x) log(p(x)/q(x)) — 매 not symmetric, ≥ 0.
- **Cross-entropy**: H(P,Q) = H(P) + D_KL(P||Q) = -Σ p(x) log q(x).

### 매 properties
- H(X) ≥ 0, H(X) ≤ log |𝒳| (uniform 시 max).
- H(X,Y) ≤ H(X) + H(Y) (equality ⟺ independent).
- I(X;Y) ≥ 0 (independence iff =0).
- Data processing inequality: X→Y→Z ⇒ I(X;Z) ≤ I(X;Y).

### 매 응용
1. **Compression**: 매 lower bound (Shannon source coding) — Huffman, arithmetic, ANS 의 modern (Zstd, Brotli).
2. **Channel capacity**: C = max_p I(X;Y) — 매 AWGN channel: C = ½log(1+SNR).
3. **Cross-entropy loss**: 매 classification 의 standard loss (PyTorch/TensorFlow default).
4. **Variational inference**: ELBO = E[log p(x|z)] - D_KL(q(z|x) || p(z)) — 매 VAE/diffusion 의 basis.
5. **Decision trees**: 매 information gain split criterion.
6. **Maximum entropy principle**: 매 prior choice 의 (Jaynes 1957).

## 💻 패턴

### 매 Discrete Entropy (NumPy)
```python
import numpy as np

def shannon_entropy(p, base=2):
    """매 probability vector → entropy.
    매 zero-handling: 0 log 0 = 0."""
    p = np.asarray(p, dtype=float)
    p = p[p > 0]
    return -np.sum(p * np.log(p) / np.log(base))

# Example: fair coin = 1 bit
print(shannon_entropy([0.5, 0.5]))  # 1.0
# biased coin (p=0.9): low entropy
print(shannon_entropy([0.9, 0.1]))  # ≈ 0.469
```

### KL Divergence / Cross-Entropy
```python
def kl_divergence(p, q, base=2):
    """매 D_KL(P||Q). P, Q 의 same support 가정."""
    p = np.asarray(p, dtype=float)
    q = np.asarray(q, dtype=float)
    mask = (p > 0) & (q > 0)
    return np.sum(p[mask] * np.log(p[mask] / q[mask]) / np.log(base))

def cross_entropy(p, q, base=2):
    p = np.asarray(p, dtype=float)
    q = np.asarray(q, dtype=float)
    mask = (p > 0)
    return -np.sum(p[mask] * np.log(q[mask]) / np.log(base))

p = [0.5, 0.5]
q = [0.9, 0.1]
print(kl_divergence(p, q))      # 매 not zero
print(cross_entropy(p, q))      # 매 H(p) + KL(p||q)
```

### PyTorch Cross-Entropy Loss (매 ML standard)
```python
import torch
import torch.nn as nn

# Classification logits → softmax → -Σ y log p
logits = torch.tensor([[2.0, 1.0, 0.1]])  # batch=1, classes=3
target = torch.tensor([0])  # true class
loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(logits, target)
print(f"Cross-entropy loss: {loss.item():.4f}")

# 매 internal: log_softmax + nll_loss for numerical stability
# 매 직접 softmax → log → nll 분리 시 매 underflow 위험
```

### Mutual Information Estimation (매 continuous)
```python
from sklearn.feature_selection import mutual_info_regression
from sklearn.metrics import mutual_info_score

# 매 discrete: histogram-based
def mutual_info_discrete(x, y, bins=20):
    c_xy, _, _ = np.histogram2d(x, y, bins=bins)
    return mutual_info_score(None, None, contingency=c_xy)

# 매 continuous: KSG estimator (Kraskov-Stögbauer-Grassberger 2004)
# sklearn 의 mutual_info_regression 의 internal 의 KSG 사용
x = np.random.randn(1000)
y = x + 0.5 * np.random.randn(1000)
mi = mutual_info_regression(x.reshape(-1, 1), y)
print(f"MI(x; y) ≈ {mi[0]:.4f} nats")
```

### Decision Tree Information Gain
```python
def information_gain(parent_labels, splits):
    """매 split 의 information gain.
    IG = H(parent) - Σ (|child|/|parent|) H(child)."""
    def H(labels):
        _, counts = np.unique(labels, return_counts=True)
        p = counts / counts.sum()
        return -np.sum(p * np.log2(p + 1e-12))
    
    parent_H = H(parent_labels)
    n = len(parent_labels)
    weighted_child_H = sum((len(s) / n) * H(s) for s in splits)
    return parent_H - weighted_child_H
```

### Variational Inference (매 ELBO)
```python
import torch.nn.functional as F

def vae_elbo(x, x_recon, mu, logvar):
    """ELBO = E[log p(x|z)] - D_KL(q(z|x) || p(z)).
    매 p(z) = N(0,I), q(z|x) = N(mu, σ²I)."""
    recon_loss = F.binary_cross_entropy(x_recon, x, reduction='sum')
    # 매 KL closed-form for Gaussians:
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon_loss + kl  # negative ELBO 매 minimize
```

### Differential Entropy (매 continuous variable)
```python
from scipy.stats import differential_entropy

# 매 sample-based estimator
samples = np.random.randn(10000)
h = differential_entropy(samples)
print(f"h(N(0,1)) ≈ {h:.4f}")  # closed-form: 0.5 log(2πe) ≈ 1.4189

# 매 differential entropy 매 negative 가능 (e.g., narrow distribution)
samples_narrow = np.random.randn(10000) * 0.1
print(differential_entropy(samples_narrow))  # negative
```

## 매 결정 기준

| 상황 | Quantity |
|---|---|
| Compression lower bound | H(X) |
| Classification loss | Cross-entropy H(P,Q) |
| Distribution comparison | KL divergence D_KL(P||Q) |
| Symmetric divergence | Jensen-Shannon (½KL(P||M) + ½KL(Q||M)) |
| Feature selection | Mutual information I(X;Y) |
| VAE training | ELBO (recon + KL) |
| Decision tree split | Information gain |
| Channel design | Mutual information capacity |

**기본값**: 매 ML loss 의 cross-entropy. 매 distribution distance 의 KL (asymmetric). 매 symmetric 가 필요하면 JS divergence.

## 🔗 Graph
- 부모: [[Information_Theory|Information-Theory]] · [[Probability-Theory]]
- 응용: [[Cross-Entropy-Loss]] · [[KL-Divergence]] · [[Mutual-Information]] · [[Variational-Inference]]

## 🤖 LLM 활용
**언제**: 매 concept explanation, 매 derivation 의 walk-through, 매 ML loss function selection, 매 KL/cross-entropy 의 confused 시 disambiguation.
**언제 X**: 매 numerical computation 의 large-scale data — 매 dedicated library (NumPy, scikit-learn) 사용. 매 differential entropy 의 sample size 부족 시 hallucinate 위험.

## ❌ 안티패턴
- **KL 의 symmetric 가정**: 매 D_KL(P||Q) ≠ D_KL(Q||P). 매 forward/reverse KL 의 다른 behavior (mode-seeking vs mean-seeking).
- **log(0) 처리 누락**: 매 0 log 0 의 limit 의 0 — 매 mask 또는 ε 추가.
- **Cross-entropy 의 softmax 두 번 적용**: PyTorch CrossEntropyLoss 매 logits 받음, softmax 직접 적용 X.
- **Differential entropy 의 Shannon entropy 와 혼동**: 매 differential entropy 매 negative 가능, 매 not invariant under change of variable.
- **MI 의 직접 estimation 의 high-dim**: 매 sample complexity exponential — copula / NN-based estimator 사용.

## 🧪 검증 / 중복
- Verified (Shannon 1948, Cover-Thomas "Elements of Information Theory", MacKay "Information Theory, Inference and Learning Algorithms").
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — entropy/KL/MI/cross-entropy formulas, ML applications, VAE ELBO |
