---
id: wiki-2026-0508-information-entropy
title: Information Entropy
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Shannon Entropy, H(X), Surprise]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [information-theory, entropy, shannon, ml-foundations]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: scipy
---

# Information Entropy

## 매 한 줄
> **"매 surprise 의 average"**. Shannon entropy $H(X) = -\sum p(x) \log p(x)$ — 매 distribution 의 uncertainty 매 bit 단위 measure. 매 1948 Shannon, 매 2026 LLM token sampling, compression, decision-tree split, MI-based feature selection 매 foundation.

## 매 핵심

### 매 정의
- $H(X) = -\sum_{x} p(x) \log_2 p(x)$ — 매 bit (log2), nat (ln), dit (log10).
- $0 \le H(X) \le \log_2 |\mathcal{X}|$ — 매 uniform 시 max.
- **Joint**: $H(X,Y)$ · **Conditional**: $H(Y|X) = H(X,Y) - H(X)$.
- **Mutual Information**: $I(X;Y) = H(Y) - H(Y|X)$.

### 매 핵심 properties
- **Non-negative**: $H(X) \ge 0$.
- **Concavity**: 매 distribution mixing 시 entropy 증가.
- **Chain rule**: $H(X,Y) = H(X) + H(Y|X)$.
- **Data Processing**: $I(X;Y) \ge I(X;f(Y))$.

### 매 응용
1. 매 lossless compression (Huffman, arithmetic) — limit $H(X)$.
2. 매 ML loss (cross-entropy, KL).
3. 매 decision tree split (ID3, C4.5 information gain).
4. 매 LLM sampling (entropy-aware temperature, min-p).

## 💻 패턴

### 1. Shannon Entropy
```python
import numpy as np
def entropy(probs, base=2):
    p = np.asarray(probs)
    p = p[p > 0]
    return -np.sum(p * np.log(p)) / np.log(base)

# H([0.5,0.5]) = 1.0 bit
# H([1.0,0.0]) = 0.0 bit
# H([0.25]*4) = 2.0 bit
```

### 2. Cross-Entropy Loss (PyTorch)
```python
import torch.nn.functional as F
# 매 true distribution q vs predicted p
# H(q, p) = -Σ q(x) log p(x)
loss = F.cross_entropy(logits, targets)  # softmax + NLL
```

### 3. KL Divergence
```python
def kl_div(p, q):
    p, q = np.asarray(p), np.asarray(q)
    return np.sum(p * np.log(p / q + 1e-12))
# 매 D_KL(p||q) = H(p,q) - H(p)
```

### 4. Mutual Information (sklearn)
```python
from sklearn.feature_selection import mutual_info_classif
mi = mutual_info_classif(X, y, random_state=0)
# 매 feature ranking — 매 non-linear dependency capture
```

### 5. Decision Tree Split (information gain)
```python
def info_gain(parent_y, splits):
    H_parent = entropy(np.bincount(parent_y) / len(parent_y))
    H_children = sum((len(s)/len(parent_y)) * entropy(np.bincount(s)/len(s))
                     for s in splits if len(s) > 0)
    return H_parent - H_children
```

### 6. LLM Min-p Sampling (entropy-aware)
```python
# 매 2024+ standard — 매 temperature alternative
def min_p_sample(logits, p_base=0.05):
    probs = softmax(logits)
    threshold = p_base * probs.max()
    mask = probs >= threshold
    probs = probs * mask
    probs /= probs.sum()
    return np.random.choice(len(probs), p=probs)
```

### 7. Differential Entropy (Continuous)
```python
from scipy.stats import differential_entropy
samples = np.random.normal(0, 1, 10_000)
h = differential_entropy(samples)  # ≈ 0.5*log(2πe) ≈ 1.42 nat
```

## 매 결정 기준
| 상황 | Form |
|---|---|
| Discrete classification | Cross-entropy loss |
| Distribution compare | KL divergence |
| Feature selection | Mutual information |
| Tree split | Information gain |
| LLM sample diversity | Min-p / temperature |

**기본값**: Cross-entropy loss + softmax, 매 ML classification 매 standard.

## 🔗 Graph
- 부모: [[Entropy in Information Theory]] · [[Probability Theory]]
- 변형: [[Mutual-Information]] · [[Kullback-Leibler-Divergence]] · [[Kolmogorov-Complexity]]
- 응용: [[Entropy in Information Theory|Information Theory]] · [[Information Retrieval (IR)]]
- Adjacent: [[Statistical-Power]] · [[Bayesian-Inference]]

## 🤖 LLM 활용
**언제**: 매 loss design, 매 feature ranking, 매 sampling strategy 설명, 매 compression bound 추정.
**언제 X**: 매 small-sample MI estimation 매 unreliable. 매 high-dim differential entropy 매 hard.

## ❌ 안티패턴
- **MI without binning correction**: 매 small-n bias.
- **KL asymmetry 무시**: $D_{KL}(p\|q) \ne D_{KL}(q\|p)$.
- **Cross-entropy on wrong base**: 매 nat vs bit confusion.
- **Negative differential entropy 의 panic**: 매 continuous 매 자연스럽다.

## 🧪 검증 / 중복
- Verified (Cover & Thomas, *Elements of IT*; MacKay, *ITILA*).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — full Shannon/KL/MI + min-p pattern |
