2nd/10_Wiki/Topics/Computer_Science_and_Theory/Expectation-Maximization.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-expectation-maximization	Expectation Maximization	10_Wiki/Topics	verified	self	EM Algorithm Expectation-Maximization GMM-EM Baum-Welch	none	A	0.95	applied	statistics machine-learning latent-variables optimization probabilistic-models		2026-05-10	pending	language framework Python scikit-learn / NumPy / PyTorch

Expectation Maximization

매 한 줄

"매 latent variable 가진 model 의 maximum likelihood 의 iterative 추정 — E-step (posterior) ↔ M-step (parameter update) 교차". Dempster-Laird-Rubin 1977 의 unification — 매 GMM, HMM (Baum-Welch), LDA, factor analysis, missing data imputation 의 모두 instances. 매 modern variational autoencoder 의 amortized EM.

매 핵심

매 Algorithm

Goal: maximize log p(X|θ) where X observed, Z latent.

• E-step: 매 posterior q(Z) = p(Z|X, θ_old).
• M-step: θ_new = argmax_θ E_q[log p(X, Z | θ)].
• Repeat: until convergence (likelihood plateau).

매 ELBO interpretation

log p(X|θ) ≥ E_q[log p(X,Z|θ)] - E_q[log q(Z)] = ELBO(q, θ).

• E-step: 매 maximize ELBO over q (equiv. KL(q||p(Z|X,θ))=0 — 매 exact).
• M-step: 매 maximize ELBO over θ.
• 매 monotonic increase of log-likelihood guaranteed.

매 Convergence

• 매 local optimum 으로만 converge (matter 의 likelihood 의 multimodal).
• 매 multiple random init 권장.
• 매 K-means 의 EM 의 hard-assignment limit (Gaussian variance → 0).

매 응용

1. Gaussian Mixture Models: 매 clustering with soft assignments.
2. Hidden Markov Models (Baum-Welch): 매 speech recognition, NLP, bioinformatics.
3. Latent Dirichlet Allocation (variational EM): topic modeling.
4. Factor analysis / PPCA: 매 dimensionality reduction.
5. Missing data imputation: 매 MICE.
6. VAE training (amortized EM): 매 modern deep generative.

💻 패턴

GMM-EM (매 from scratch, NumPy)

import numpy as np

class GaussianMixtureEM:
    def __init__(self, K, max_iter=100, tol=1e-6):
        self.K = K
        self.max_iter = max_iter
        self.tol = tol
    
    def fit(self, X):
        n, d = X.shape
        # 매 init: random + uniform priors
        self.pi = np.ones(self.K) / self.K
        idx = np.random.choice(n, self.K, replace=False)
        self.mu = X[idx]
        self.sigma = np.array([np.cov(X.T) for _ in range(self.K)])
        
        log_lik_old = -np.inf
        for it in range(self.max_iter):
            # E-step: posterior responsibilities γ_ik
            log_resp = self._log_responsibilities(X)  # (n, K)
            resp = np.exp(log_resp - log_resp.max(axis=1, keepdims=True))
            resp /= resp.sum(axis=1, keepdims=True)
            
            # M-step
            Nk = resp.sum(axis=0)              # (K,)
            self.pi = Nk / n
            self.mu = (resp.T @ X) / Nk[:, None]
            for k in range(self.K):
                diff = X - self.mu[k]
                self.sigma[k] = (resp[:, k:k+1] * diff).T @ diff / Nk[k]
                self.sigma[k] += 1e-6 * np.eye(d)  # 매 regularization
            
            # convergence
            log_lik = self._log_likelihood(X)
            if abs(log_lik - log_lik_old) < self.tol:
                break
            log_lik_old = log_lik
        return self
    
    def _log_gaussian(self, X, mu, sigma):
        d = X.shape[1]
        diff = X - mu
        inv = np.linalg.inv(sigma)
        det = np.linalg.det(sigma)
        return -0.5 * (d * np.log(2 * np.pi) + np.log(det) +
                       np.einsum('ni,ij,nj->n', diff, inv, diff))
    
    def _log_responsibilities(self, X):
        log_resp = np.zeros((X.shape[0], self.K))
        for k in range(self.K):
            log_resp[:, k] = np.log(self.pi[k] + 1e-12) + \
                             self._log_gaussian(X, self.mu[k], self.sigma[k])
        return log_resp
    
    def _log_likelihood(self, X):
        log_resp = self._log_responsibilities(X)
        from scipy.special import logsumexp
        return logsumexp(log_resp, axis=1).sum()

# Demo
np.random.seed(42)
X1 = np.random.randn(100, 2) + np.array([5, 0])
X2 = np.random.randn(100, 2) + np.array([-5, 0])
X = np.vstack([X1, X2])
model = GaussianMixtureEM(K=2).fit(X)
print(f"Means:\n{model.mu}")
print(f"Mixing:\n{model.pi}")

scikit-learn (매 production)

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(n_components=3, covariance_type='full', 
                      max_iter=100, n_init=10, random_state=42)
gmm.fit(X)
print(f"Converged: {gmm.converged_}")
print(f"BIC: {gmm.bic(X):.2f}")  # 매 model selection
labels = gmm.predict(X)
proba = gmm.predict_proba(X)  # 매 soft assignment

Baum-Welch (HMM, 매 EM 의 instance)

def baum_welch(observations, n_states, n_iter=100):
    """매 HMM 의 forward-backward + EM updates."""
    T = len(observations)
    pi = np.ones(n_states) / n_states
    A = np.random.rand(n_states, n_states); A /= A.sum(axis=1, keepdims=True)
    B = np.random.rand(n_states, max(observations)+1); B /= B.sum(axis=1, keepdims=True)
    
    for it in range(n_iter):
        # E-step: forward α, backward β
        alpha = np.zeros((T, n_states))
        alpha[0] = pi * B[:, observations[0]]
        for t in range(1, T):
            alpha[t] = (alpha[t-1] @ A) * B[:, observations[t]]
        
        beta = np.zeros((T, n_states))
        beta[T-1] = 1
        for t in range(T-2, -1, -1):
            beta[t] = A @ (B[:, observations[t+1]] * beta[t+1])
        
        # γ_t(i), ξ_t(i,j)
        gamma = alpha * beta
        gamma /= gamma.sum(axis=1, keepdims=True)
        xi = np.zeros((T-1, n_states, n_states))
        for t in range(T-1):
            num = alpha[t][:, None] * A * B[:, observations[t+1]] * beta[t+1]
            xi[t] = num / num.sum()
        
        # M-step
        pi = gamma[0]
        A = xi.sum(axis=0) / gamma[:-1].sum(axis=0)[:, None]
        for k in range(B.shape[1]):
            mask = (observations == k)
            B[:, k] = gamma[mask].sum(axis=0) / gamma.sum(axis=0)
    return pi, A, B

VAE — 매 amortized variational EM

import torch
import torch.nn as nn

class VAE(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.enc_mu = nn.Linear(input_dim, latent_dim)
        self.enc_logvar = nn.Linear(input_dim, latent_dim)
        self.dec = nn.Linear(latent_dim, input_dim)
    
    def forward(self, x):
        # 매 E-step approximation: q(z|x) = N(μ_φ(x), σ²_φ(x))
        mu = self.enc_mu(x)
        logvar = self.enc_logvar(x)
        eps = torch.randn_like(mu)
        z = mu + torch.exp(0.5 * logvar) * eps
        x_recon = torch.sigmoid(self.dec(z))
        return x_recon, mu, logvar

def vae_loss(x, x_recon, mu, logvar):
    recon = nn.functional.binary_cross_entropy(x_recon, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon + kl
# 매 SGD 의 joint optimization 의 amortized E+M

MAP-EM (매 with prior, regularized)

# 매 prior 의 add 시 monotonic posterior 증가.
# Example: GMM 에 Dirichlet prior on π, NIW on (μ, Σ).
# 매 sklearn BayesianGaussianMixture 의 internal.
from sklearn.mixture import BayesianGaussianMixture
bgmm = BayesianGaussianMixture(n_components=10, weight_concentration_prior=1e-2)
bgmm.fit(X)
# 매 effective K 의 자동 sparsification.

매 결정 기준

상황	Variant
Standard mixture clustering	Vanilla EM (sklearn)
Sequential / temporal	Baum-Welch (HMM)
Topic modeling	Variational EM (LDA)
Scalable / online	Online EM, stochastic
Deep latent model	VAE (amortized)
Need MAP / regularization	MAP-EM, Bayesian-EM
Hard assignment baseline	K-means (EM degenerate)
Discrete latent	Categorical EM

기본값: 매 GMM clustering 매 sklearn GaussianMixture(n_init=10). 매 deep 매 VAE.

🔗 Graph

• 부모: Maximum-Likelihood-Estimation · Latent-Variable-Models
• 변형: Variational-EM · Stochastic-EM · MAP-EM · Hard-EM
• 응용: Gaussian-Mixture-Models · Hidden-Markov-Models · LDA-Topic-Modeling · VAE
• Adjacent: Variational-Inference · Maximum-A-Posteriori · K-Means-Clustering-Foundations · Baum-Welch

🤖 LLM 활용

언제: 매 derivation 의 walk-through, 매 ELBO 의 explain, 매 model selection (BIC) 의 advice, 매 troubleshooting (e.g., 매 singular covariance). 언제 X: 매 large-scale fitting — 매 sklearn / dedicated library 사용. 매 numerical issue 의 diagnosis 시 actual data 의 inspection 필요.

❌ 안티패턴

• Single random init: 매 local optimum trap — n_init=10 권장.
• Singular covariance ignore: 매 sigma += εI 의 regularization 필수.
• Convergence 의 likelihood 가 아닌 parameter 의 monitor: 매 wrong — likelihood / ELBO 의 monitor.
• K 의 randomly choose: 매 BIC / AIC / cross-validation 사용.
• K-means 의 GMM 결과 비교: 매 different — GMM 의 soft assignment + covariance.
• EM 의 global optimum 가정: 매 local optimum 만 — multi-start 필수.

🧪 검증 / 중복

• Verified (Dempster-Laird-Rubin 1977, Bishop "PRML" Ch9, Murphy "Probabilistic ML" Ch11).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — algorithm, ELBO, GMM/HMM/VAE applications, NumPy from-scratch