---
id: wiki-2026-0508-mutual-information
title: Mutual Information
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Mutual-Information, MI, 상호정보량, I(X;Y)]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [information-theory, statistics, dependence, feature-selection, representation-learning]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: scikit-learn-pytorch
---

# Mutual Information

## 매 한 줄
> **"매 X를 알면 Y에 대해 매 얼마나 덜 놀라는가"**. I(X;Y) = H(X) − H(X|Y) = KL(P_XY ‖ P_X⊗P_Y); 매 어떤 monotonic transform에도 invariant한 dependence measure다. 2026 self-supervised learning(InfoNCE, CLIP)과 disentanglement(β-VAE) 핵심 도구.

## 매 핵심

### 매 정의
- **Discrete**: I(X;Y) = ΣΣ p(x,y) log(p(x,y)/(p(x)p(y))).
- **Continuous**: integral 형태.
- **KL form**: I(X;Y) = D_KL(P_XY ‖ P_X⊗P_Y) ≥ 0.
- 매 0 iff X⊥Y.

### 매 추정의 어려움
- High-dim continuous에서 매 매 어렵다.
- KSG(k-NN) — Kraskov-Stögbauer-Grassberger.
- MINE — neural net 기반 (Donsker-Varadhan).
- InfoNCE bound — contrastive.
- KDE 기반 (low dim only).

### 매 응용
1. Feature selection (mRMR).
2. Self-supervised learning (SimCLR, CLIP, InfoNCE).
3. Representation disentanglement (β-VAE, InfoGAN).
4. Causal discovery (CMI tests).
5. Information bottleneck.

## 💻 패턴

### Discrete MI (sklearn)
```python
from sklearn.metrics import mutual_info_score
mi = mutual_info_score(x_labels, y_labels)
```

### Continuous MI (KSG via sklearn)
```python
from sklearn.feature_selection import mutual_info_regression
mi = mutual_info_regression(X, y, n_neighbors=3)
```

### MINE (PyTorch)
```python
import torch, torch.nn as nn
class MINE(nn.Module):
    def __init__(self, dim_x, dim_y, hid=256):
        super().__init__()
        self.f = nn.Sequential(nn.Linear(dim_x+dim_y, hid),
                               nn.ELU(), nn.Linear(hid, 1))
    def forward(self, x, y):
        joint = self.f(torch.cat([x, y], -1))
        y_perm = y[torch.randperm(y.size(0))]
        marg = self.f(torch.cat([x, y_perm], -1))
        return joint.mean() - torch.log(marg.exp().mean())
```

### InfoNCE lower bound
```python
def infonce(z_x, z_y, tau=0.1):
    logits = z_x @ z_y.T / tau
    labels = torch.arange(z_x.size(0), device=z_x.device)
    return -torch.nn.functional.cross_entropy(logits, labels) + np.log(z_x.size(0))
```

### Conditional MI (KSG-style via partition)
```python
def cmi_estimate(X, Y, Z, n_neighbors=5):
    # I(X;Y|Z) = I(X;Y,Z) - I(X;Z)
    return (mutual_info_regression(X, np.column_stack([Y, Z]), n_neighbors=n_neighbors)
            - mutual_info_regression(X, Z, n_neighbors=n_neighbors))
```

### Maximal Information Coefficient (MIC)
```python
from minepy import MINE as MIC
mine = MIC(); mine.compute_score(x, y)
print(mine.mic())
```

## 매 결정 기준
| 상황 | Estimator |
|---|---|
| Discrete labels | sklearn `mutual_info_score` |
| Low-dim continuous | KSG (k-NN) |
| High-dim, ML training | InfoNCE / MINE |
| Equitability needed | MIC |

**기본값**: KSG for analysis, InfoNCE for SSL training.

## 🔗 Graph
- 부모: [[Entropy in Information Theory|Information Theory]] · [[Information-Entropy]]
- Adjacent: [[Kullback-Leibler-Divergence]] · [[Cross-Entropy]] · [[Causal-Inference]]

## 🤖 LLM 활용
**언제**: Non-linear dependence detection, contrastive representation learning, causal screening.
**언제 X**: Sample size 작거나 high-dim continuous (estimator 신뢰도 ↓).

## ❌ 안티패턴
- **Histogram MI in high-dim**: bias 폭발.
- **MI = correlation 가정**: MI는 매 모든 dependence 잡지만 correlation은 linear만.
- **Plug-in estimator 그대로**: 매 bias correction 필수.

## 🧪 검증 / 중복
- Verified (Cover & Thomas "Elements of Information Theory" ch.2).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — MI estimators + MINE/InfoNCE patterns |
