2nd/10_Wiki/Topics/Computer_Science_and_Theory/Linear-Algebra-Foundations.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-linear-algebra-foundations	Linear Algebra Foundations	10_Wiki/Topics	verified	self	Matrix Algebra Vector Spaces Linear Algebra	none	A	0.95	applied	linear-algebra math ml-foundations matrix		2026-05-10	pending	language framework python numpy, jax, torch

Linear Algebra Foundations

매 한 줄

"매 ML 의 universal language". Linear Algebra 는 vector space, linear map, eigendecomposition, SVD 의 study — 매 modern ML/DL 의 numerical backbone. 2026 LLM era 에서 attention 은 매 Q@K.T softmax @V — 매 pure linear algebra 의 chain. GPU/TPU 의 design 자체가 매 LA primitive (GEMM) 위에 built.

매 핵심

매 Vector spaces

• field \mathbb{F} (보통 \mathbb{R}, \mathbb{C})
• closed under linear combinations
• basis: minimal spanning set, dim
• linear map T: V \to W — represent as matrix in basis

매 Decompositions

• LU: A = LU — Gaussian elimination, O(n^3)
• QR: A = QR, Q orthogonal — least squares
• Eigendecomposition: A = V\Lambda V^{-1}, square only
• SVD: A = U\Sigma V^\top — universal, rectangular
• Cholesky: A = LL^\top, SPD only, fastest

매 Norms / inner products

• \|x\|_2 = \sqrt{x^\top x}, \|x\|_p, \|x\|_\infty
• Frobenius: \|A\|_F = \sqrt{\sum a_{ij}^2}
• Spectral: \|A\|_2 = \sigma_{\max}(A)
• Cauchy-Schwarz: |x^\top y| \le \|x\| \|y\|

매 응용

1. Attention: \text{softmax}(QK^\top / \sqrt{d}) V.
2. PCA: SVD of centered X.
3. Linear regression: normal eq w = (X^\top X)^{-1} X^\top y.
4. Graph Laplacian: spectral clustering.
5. Quantum states: complex Hilbert space.

💻 패턴

NumPy basics

import numpy as np
A = np.random.randn(5, 3)
# Solve Ax = b in least-squares
b = np.random.randn(5)
x, *_ = np.linalg.lstsq(A, b, rcond=None)
# rank, condition number, determinant
print(np.linalg.matrix_rank(A), np.linalg.cond(A))

SVD + low-rank approx

U, S, Vt = np.linalg.svd(A, full_matrices=False)
k = 2
A_lr = U[:, :k] @ np.diag(S[:k]) @ Vt[:k]   # rank-k approx
err = np.linalg.norm(A - A_lr, "fro")

Eigendecomposition (symmetric)

S = np.random.randn(4, 4); S = S + S.T
w, V = np.linalg.eigh(S)   # use eigh for symmetric (stable, real)
# reconstruction
assert np.allclose(V @ np.diag(w) @ V.T, S, atol=1e-10)

Cholesky for SPD systems

import scipy.linalg as la
A = np.random.randn(50, 50)
A = A.T @ A + np.eye(50)   # SPD
L = la.cholesky(A, lower=True)
b = np.random.randn(50)
y = la.solve_triangular(L, b, lower=True)
x = la.solve_triangular(L.T, y, lower=False)

JAX matmul + autodiff

import jax, jax.numpy as jnp
@jax.jit
def loss(W, x, y):
    return jnp.mean((x @ W - y) ** 2)
grad_fn = jax.grad(loss)

Power iteration (top eigenvector)

def power_iter(A, n_iter=200, tol=1e-10):
    x = np.random.randn(A.shape[0])
    x /= np.linalg.norm(x)
    for _ in range(n_iter):
        x_new = A @ x
        x_new /= np.linalg.norm(x_new)
        if np.linalg.norm(x_new - x) < tol: break
        x = x_new
    eig = x @ A @ x
    return eig, x

PyTorch attention (linear algebra core)

import torch, math
def attention(Q, K, V, mask=None):
    d = Q.size(-1)
    s = Q @ K.transpose(-2, -1) / math.sqrt(d)
    if mask is not None: s = s.masked_fill(mask == 0, -1e9)
    return torch.softmax(s, dim=-1) @ V

매 결정 기준

Problem	Method
solve Ax = b, square nonsingular	LU (np.linalg.solve)
SPD	Cholesky
least squares	QR or SVD
dimensionality reduction	SVD / PCA
symmetric eigen	eigh
sparse large	scipy.sparse.linalg / iterative

기본값: SVD when in doubt — most stable, universal.

🔗 Graph

• 변형: SVD · Eigendecomposition
• 응용: PCA · Attention Mechanism · Linear-Regression

🤖 LLM 활용

언제: 매 ML 의 derivation, debugging matrix shapes, performance reasoning. 언제 X: 매 task 가 combinatorial — graph algorithms 등.

❌ 안티패턴

• inv(A) @ b 사용: numerically unstable + slow → use solve.
• eig vs eigh 혼동: symmetric 인데 eig 사용 → complex eigenvalues from numerical noise.
• Memory layout 무시: row vs column major → 10× slowdown.
• Condition number 무시: ill-conditioned matrix → inversion blows up.
• Dense for sparse: huge sparse → use scipy.sparse.

🧪 검증 / 중복

• Verified (Strang 2016 textbook, Trefethen & Bau 1997, Golub & Van Loan 2013).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — vector spaces, decompositions, norms, NumPy/JAX patterns