---
id: wiki-2026-0508-pca-and-dimension-reduction
title: PCA and Dimension Reduction
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Principal Component Analysis, Dimensionality Reduction]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [statistics, ml, unsupervised, embeddings, visualization]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: scikit-learn/umap
---

# PCA and Dimension Reduction

## 매 한 줄
> **"매 high-dim 데이터를 variance 보존하는 lower-dim 부분공간으로 사영"**. Pearson 1901 / Hotelling 1933 의 PCA 가 시초. 매 2026 의 modern landscape: linear PCA 는 여전히 baseline + interpretation, t-SNE/UMAP 가 visualization 의 default, autoencoder + contrastive 가 representation learning 의 핵심. 매 LLM embedding 의 PCA whitening 도 흔함.

## 매 핵심

### 매 PCA 수학
- **목표**: orthogonal directions 중 variance 최대화.
- **계산**: covariance Σ = X^T X / (n-1) → eigendecomposition Σ = V Λ V^T, top-k columns = principal components.
- **SVD form**: X = UΣV^T → top-k V_k 가 components, score = X V_k.
- **Explained variance ratio**: λ_i / Σ λ_j.
- **Whitening**: X V_k Λ_k^{-1/2} → unit variance per dim.

### 매 family
- **Linear**: PCA, Truncated SVD, Factor Analysis, ICA.
- **Manifold (preserve local)**: t-SNE, UMAP, LLE, Isomap.
- **Neural**: Autoencoder, VAE, contrastive (SimCLR), DINO.
- **Random**: Random projection (Johnson-Lindenstrauss).
- **Sparse**: Sparse PCA, NMF.

### 매 응용
1. Visualization (t-SNE, UMAP for embeddings/scRNA-seq).
2. Compression (PCA whitening before downstream task).
3. Denoising (project, then reconstruct).
4. Feature engineering pre-classifier.
5. LLM embedding analysis (cluster interpretation, anisotropy fix).

## 💻 패턴

### sklearn PCA
```python
from sklearn.decomposition import PCA
import numpy as np

X = np.random.randn(1000, 100)
pca = PCA(n_components=10, whiten=True, random_state=42)
X_proj = pca.fit_transform(X)
print(pca.explained_variance_ratio_.cumsum())  # 누적 explained
print(pca.components_.shape)  # (10, 100)
```

### Choose k via cumulative variance
```python
def choose_k(X, threshold=0.95):
    pca = PCA().fit(X)
    cum = pca.explained_variance_ratio_.cumsum()
    return int(np.searchsorted(cum, threshold) + 1)
```

### Truncated SVD (sparse / very large)
```python
from sklearn.decomposition import TruncatedSVD
from scipy.sparse import csr_matrix

X_sparse = csr_matrix(X)  # 매 PCA centers → dense; TruncatedSVD 매 sparse-friendly
svd = TruncatedSVD(n_components=50, n_iter=7, random_state=42)
X_proj = svd.fit_transform(X_sparse)  # 매 LSA 의 핵심
```

### UMAP (manifold, fast)
```python
import umap

reducer = umap.UMAP(
    n_neighbors=15,    # local vs global
    min_dist=0.1,      # cluster separation
    n_components=2,
    metric="cosine",   # 매 LLM embedding
    random_state=42,
)
X_2d = reducer.fit_transform(X)
```

### t-SNE (visualization)
```python
from sklearn.manifold import TSNE

# 매 항상 PCA → t-SNE (속도/안정성)
X_pca = PCA(n_components=50).fit_transform(X)
X_2d = TSNE(n_components=2, perplexity=30, init="pca",
            learning_rate="auto").fit_transform(X_pca)
```

### Autoencoder (PyTorch)
```python
import torch.nn as nn

class AE(nn.Module):
    def __init__(self, d_in, d_latent):
        super().__init__()
        self.enc = nn.Sequential(
            nn.Linear(d_in, 256), nn.GELU(),
            nn.Linear(256, d_latent),
        )
        self.dec = nn.Sequential(
            nn.Linear(d_latent, 256), nn.GELU(),
            nn.Linear(256, d_in),
        )
    def forward(self, x):
        z = self.enc(x); return self.dec(z), z
# loss = MSE(x, x_hat). Nonlinear PCA 의 generalization.
```

### LLM embedding whitening (anisotropy 완화)
```python
def whiten_embeddings(E, k=None):
    """Anisotropy fix: subtract mean, decorrelate."""
    mu = E.mean(axis=0, keepdims=True)
    Ec = E - mu
    U, S, Vt = np.linalg.svd(Ec, full_matrices=False)
    if k is None:
        k = E.shape[1]
    W = (Vt[:k].T) / S[:k]   # whitening matrix
    return Ec @ W, mu, W
```

### Random projection (very high-dim, fast)
```python
from sklearn.random_projection import GaussianRandomProjection

rp = GaussianRandomProjection(n_components="auto", eps=0.1)
X_proj = rp.fit_transform(X)  # 매 Johnson-Lindenstrauss 보존
```

## 매 결정 기준
| 상황 | Method |
|---|---|
| baseline, interpretable | PCA |
| sparse text-term-matrix | TruncatedSVD (LSA) |
| visualization 2D/3D | UMAP > t-SNE |
| nonlinear, learnable, downstream supervised | Autoencoder / SimCLR |
| n >> d, very high-dim | Random projection |
| non-negative parts (topics) | NMF |
| count data | LDA / NMF |

**기본값**: 매 baseline PCA → 매 visualization UMAP → 매 representation learning contrastive.

## 🔗 Graph
- 부모: [[Linear-Algebra-Foundations]] · [[Statistics]]
- 변형: [[ICA]]
- 응용: [[Feature Engineering]]
- Adjacent: [[t-SNE]] · [[UMAP]] · [[Autoencoder]]

## 🤖 LLM 활용
**언제**: 매 embedding analysis, 매 anisotropy whitening, 매 visualizing high-dim attention/activations.
**언제 X**: 매 매우 nonlinear task — autoencoder/contrastive 사용. 매 preserving exact distances 필요 — RP 만 보장.

## ❌ 안티패턴
- **PCA without scaling**: 매 큰-단위 feature 가 dominate. 매 StandardScaler 필수.
- **t-SNE 결과 해석 으로 cluster 크기/거리 신뢰**: 매 t-SNE 매 local-only — 매 global geometry 왜곡.
- **PCA 사용 후 inverse_transform 으로 outlier 제거** — 매 그러면 다시 fit X 매 outlier 의 영향 그대로.
- **n_components 선택을 임의 값** (e.g., 항상 2): 매 cumulative variance + downstream metric 기반 선택.
- **Test set 도 fit_transform**: 매 leakage. 매 fit 은 train 만, test 는 transform.

## 🧪 검증 / 중복
- Verified (Bishop *PRML* Ch 12; *ESL* Hastie et al.; UMAP McInnes 2018; sklearn docs).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — PCA & DR FULL with PCA/UMAP/AE/whitening patterns |