2nd/10_Wiki/Topics/AI_and_ML/Linear-Algebra.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-linear-algebra	Linear Algebra	10_Wiki/Topics	verified	self	Linear Algebra for ML Vectors and Matrices SVD Eigendecomposition	none	A	0.95	applied	math linear-algebra numpy svd eigen pca ml-foundations		2026-05-10	pending	language framework Python numpy/torch

Linear Algebra

매 한 줄

"매 ML은 행렬 곱셈이다". Vector·Matrix·Tensor 위에서 projection, rotation, decomposition으로 모든 게 표현된다.

매 핵심

매 객체

• Scalar, Vector ∈ ℝⁿ, Matrix ∈ ℝᵐˣⁿ, Tensor (high-rank).
• Norms: ‖x‖₁ (sparsity), ‖x‖₂ (energy), ‖x‖∞.
• Inner product, outer product, cosine similarity.

매 연산 핵심

• Matmul C=AB. shape (m,k)·(k,n)=(m,n).
• Transpose, inverse, pseudoinverse (Moore-Penrose).
• Determinant (scaling), trace (diagonal sum, eigen sum).
• Rank: 독립 column 수. low-rank → 압축 가능.

매 분해

• Eigendecomposition A = QΛQ⁻¹ (square, diagonalizable). PCA covariance.
• SVD A = UΣVᵀ (any matrix). 가장 일반적.
• QR Gram-Schmidt. least squares 안정.
• Cholesky A = LLᵀ (symm. PD). 빠른 solve, GP, Kalman.
• LU general solve.

매 ML 응용

1. PCA: covariance eigen / data SVD → top-k.
2. Linear regression: x̂ = (XᵀX)⁻¹Xᵀy 또는 SVD pseudoinverse.
3. Recommendation MF: A ≈ UVᵀ.
4. Word embeddings: LSA SVD, word2vec implicit MF.
5. Attention: softmax(QKᵀ/√d)V — 전부 matmul.

💻 패턴

NumPy 핵심

import numpy as np
A = np.random.randn(4, 3); x = np.random.randn(3)
y  = A @ x                            # matmul
G  = A.T @ A                          # 3x3 Gram
inv = np.linalg.inv(G)
sol = np.linalg.solve(G, A.T @ y)     # 안정적인 normal eq

SVD & truncated rank-k

U, S, Vt = np.linalg.svd(A, full_matrices=False)
k = 2
A_k = U[:, :k] @ np.diag(S[:k]) @ Vt[:k]

Eigen / PCA

X = np.random.randn(1000, 10)
Xc = X - X.mean(0)
cov = Xc.T @ Xc / (len(X) - 1)
vals, vecs = np.linalg.eigh(cov)              # symmetric → eigh
order = np.argsort(-vals)
PCs = vecs[:, order[:3]]
Z = Xc @ PCs                                  # (N, 3)

Least squares 4가지

# 1) normal eq (불안정)
b1 = np.linalg.inv(A.T @ A) @ A.T @ y
# 2) solve (better)
b2 = np.linalg.solve(A.T @ A, A.T @ y)
# 3) lstsq (SVD-based, 가장 안정)
b3, *_ = np.linalg.lstsq(A, y, rcond=None)
# 4) pseudoinverse
b4 = np.linalg.pinv(A) @ y

einsum (general tensor)

# batch matmul (B,M,K)·(B,K,N) → (B,M,N)
C = np.einsum("bmk,bkn->bmn", X, Y)
# attention scores
scores = np.einsum("bqd,bkd->bqk", Q, K) / np.sqrt(Q.shape[-1])

Norms / cosine

def cosine(a, b):
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12)

PyTorch (autograd)

import torch
A = torch.randn(4, 3, requires_grad=True)
loss = (A @ x - y).pow(2).sum()
loss.backward()                       # dL/dA
U, S, Vt = torch.linalg.svd(A, full_matrices=False)

Cholesky (GP / Kalman)

L = np.linalg.cholesky(K + 1e-6 * np.eye(n))   # K SPD + jitter
alpha = np.linalg.solve(L.T, np.linalg.solve(L, y))

매 결정 기준

작업	함수
일반 solve	np.linalg.solve
Least squares	np.linalg.lstsq (SVD)
Symm. eigen	eigh
일반 eigen	eig
일반 분해	svd
SPD solve 빠르게	Cholesky
Sparse 큰 행렬	scipy.sparse.linalg (eigsh, svds)
GPU	torch.linalg / cupy

기본값: 정확도/안정성은 SVD/Cholesky, 속도는 solve, 빠른 prototype은 lstsq.

🔗 Graph

• 부모: Mathematics, Numerical-Methods
• 변형: SVD, Eigendecomposition, QR-Decomposition, Cholesky
• 응용: PCA, Linear-Regression, Latent-Semantic-Analysis-LSA, Attention
• Adjacent: Tensor, Numpy, Optimization, Calculus

🤖 LLM 활용

언제: 식 유도, einsum 변환, 함수 선택, shape debug. 언제 X: numerical conditioning / iterative solver tuning은 전문가.

❌ 안티패턴

• inv(A) @ b 대신 solve(A, b) 안 씀
• Symm 행렬에 일반 eig (느림+정확도)
• Large dense에 raw SVD (메모리) → randomized/truncated
• Loop matmul (vectorize)
• 차원/축 mismatch — einsum으로 명시
• Float32 누적 오차 (PCA covariance) → float64 또는 standardize

🧪 검증 / 중복

• Verified (Strang, Trefethen NLA, NumPy/PyTorch docs). 신뢰도 A.
• 중복: 없음.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — einsum, lstsq, Cholesky 패턴 추가