---
id: wiki-2026-0508-linear-algebra-foundations
title: Linear Algebra Foundations
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Matrix Algebra, Vector Spaces, Linear Algebra]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [linear-algebra, math, ml-foundations, matrix]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: numpy, jax, torch
---

# Linear Algebra Foundations

## 매 한 줄
> **"매 ML 의 universal language"**. Linear Algebra 는 vector space, linear map, eigendecomposition, SVD 의 study — 매 modern ML/DL 의 numerical backbone. 2026 LLM era 에서 attention 은 매 Q@K.T softmax @V — 매 pure linear algebra 의 chain. GPU/TPU 의 design 자체가 매 LA primitive (GEMM) 위에 built.

## 매 핵심

### 매 Vector spaces
- field $\mathbb{F}$ (보통 $\mathbb{R}, \mathbb{C}$)
- closed under linear combinations
- basis: minimal spanning set, dim
- linear map $T: V \to W$ — represent as matrix in basis

### 매 Decompositions
- **LU**: $A = LU$ — Gaussian elimination, $O(n^3)$
- **QR**: $A = QR$, $Q$ orthogonal — least squares
- **Eigendecomposition**: $A = V\Lambda V^{-1}$, square only
- **SVD**: $A = U\Sigma V^\top$ — universal, rectangular
- **Cholesky**: $A = LL^\top$, SPD only, fastest

### 매 Norms / inner products
- $\|x\|_2 = \sqrt{x^\top x}$, $\|x\|_p$, $\|x\|_\infty$
- Frobenius: $\|A\|_F = \sqrt{\sum a_{ij}^2}$
- Spectral: $\|A\|_2 = \sigma_{\max}(A)$
- Cauchy-Schwarz: $|x^\top y| \le \|x\| \|y\|$

### 매 응용
1. **Attention**: $\text{softmax}(QK^\top / \sqrt{d}) V$.
2. **PCA**: SVD of centered $X$.
3. **Linear regression**: normal eq $w = (X^\top X)^{-1} X^\top y$.
4. **Graph Laplacian**: spectral clustering.
5. **Quantum states**: complex Hilbert space.

## 💻 패턴

### NumPy basics
```python
import numpy as np
A = np.random.randn(5, 3)
# Solve Ax = b in least-squares
b = np.random.randn(5)
x, *_ = np.linalg.lstsq(A, b, rcond=None)
# rank, condition number, determinant
print(np.linalg.matrix_rank(A), np.linalg.cond(A))
```

### SVD + low-rank approx
```python
U, S, Vt = np.linalg.svd(A, full_matrices=False)
k = 2
A_lr = U[:, :k] @ np.diag(S[:k]) @ Vt[:k]   # rank-k approx
err = np.linalg.norm(A - A_lr, "fro")
```

### Eigendecomposition (symmetric)
```python
S = np.random.randn(4, 4); S = S + S.T
w, V = np.linalg.eigh(S)   # use eigh for symmetric (stable, real)
# reconstruction
assert np.allclose(V @ np.diag(w) @ V.T, S, atol=1e-10)
```

### Cholesky for SPD systems
```python
import scipy.linalg as la
A = np.random.randn(50, 50)
A = A.T @ A + np.eye(50)   # SPD
L = la.cholesky(A, lower=True)
b = np.random.randn(50)
y = la.solve_triangular(L, b, lower=True)
x = la.solve_triangular(L.T, y, lower=False)
```

### JAX matmul + autodiff
```python
import jax, jax.numpy as jnp
@jax.jit
def loss(W, x, y):
    return jnp.mean((x @ W - y) ** 2)
grad_fn = jax.grad(loss)
```

### Power iteration (top eigenvector)
```python
def power_iter(A, n_iter=200, tol=1e-10):
    x = np.random.randn(A.shape[0])
    x /= np.linalg.norm(x)
    for _ in range(n_iter):
        x_new = A @ x
        x_new /= np.linalg.norm(x_new)
        if np.linalg.norm(x_new - x) < tol: break
        x = x_new
    eig = x @ A @ x
    return eig, x
```

### PyTorch attention (linear algebra core)
```python
import torch, math
def attention(Q, K, V, mask=None):
    d = Q.size(-1)
    s = Q @ K.transpose(-2, -1) / math.sqrt(d)
    if mask is not None: s = s.masked_fill(mask == 0, -1e9)
    return torch.softmax(s, dim=-1) @ V
```

## 매 결정 기준
| Problem | Method |
|---|---|
| solve Ax = b, square nonsingular | LU (np.linalg.solve) |
| SPD | Cholesky |
| least squares | QR or SVD |
| dimensionality reduction | SVD / PCA |
| symmetric eigen | eigh |
| sparse large | scipy.sparse.linalg / iterative |

**기본값**: SVD when in doubt — most stable, universal.

## 🔗 Graph
- 변형: [[SVD]] · [[Eigendecomposition]]
- 응용: [[PCA]] · [[Attention Mechanism]] · [[Linear-Regression]]

## 🤖 LLM 활용
**언제**: 매 ML 의 derivation, debugging matrix shapes, performance reasoning.
**언제 X**: 매 task 가 combinatorial — graph algorithms 등.

## ❌ 안티패턴
- **`inv(A) @ b` 사용**: numerically unstable + slow → use `solve`.
- **eig vs eigh 혼동**: symmetric 인데 `eig` 사용 → complex eigenvalues from numerical noise.
- **Memory layout 무시**: row vs column major → 10× slowdown.
- **Condition number 무시**: ill-conditioned matrix → inversion blows up.
- **Dense for sparse**: huge sparse → use scipy.sparse.

## 🧪 검증 / 중복
- Verified (Strang 2016 textbook, Trefethen & Bau 1997, Golub & Van Loan 2013).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — vector spaces, decompositions, norms, NumPy/JAX patterns |