docs(10_Wiki): 위키 전체 재구성 — Topic_* 폴더를 4개 카테고리로 통합 + 대규모 중복 제거

Topic_Agent/Topic_Blog/Topics/Topics_Biz/Topics_Meeting/Topics_Rag의 마크다운 지식 문서를 Topic_General/Topic_Programming/Topic_Graphic/Topic_Business 4개 카테고리로 재분류. - 중복 제거: frontmatter의 status:duplicate/merged + duplicate_of/redirect_to 필드로 자기 자신을 중복으로 선언한 리다이렉트 stub 1032개 제거, 완전 동일 내용 파일 472개 제거, 동일 파일명·다른 내용 충돌 시 더 큰(완전한) 버전만 유지(162개 제거) — 총 1639개 중복 제거. - 분류: 폴더 단위로 명확한 항목(AI_and_ML/Coding/Architecture 등 → Programming, Comfyui/Visual_Effects → Graphic, Topics_Biz/Topics_Meeting/사업 등 → Business, Poetic_Blog_Writing/창의성/Game_Design 등 → General)은 폴더 우선순위로, 나머지 혼재 폴더(Topic_Agent/Topic_Blog/Topics 루트/Thinking & Reasoning/Other/UI_UX_Assets)는 title/tags 키워드 스코어링으로 파일 단위 분류(불명확한 경우 General로 폴백). 원본 폴더명은 "From_*" 서브폴더로 보존해 추적 가능성 유지. - 최종 배치: Programming 2784 / General 1608 / Graphic 285 / Business 249 = 4926개 문서. - 에이전트 운영 상태(.astra/.agent/.obsidian/sessions/memory/_company/docs/lessons/_shared/src)는 지식 콘텐츠가 아니므로 재분류 대상에서 제외하고 원위치 유지. - Topics/Topic_email(상위 보호 폴더 Topic_email과 파일명 100% 중복) 삭제 — 보호 폴더 자체는 미변경. - 완전히 비게 된 Topic_Agent/Topic_Blog/Topics_Biz/Topics_Rag 폴더 제거.
2026-07-05 00:33:48 +09:00
parent 1cfd3bbb56
commit 9148c358d0
6455 changed files with 1 additions and 86875 deletions
@@ -0,0 +1,177 @@
+---
+id: wiki-2026-0508-ring-attention
+title: Ring Attention
+category: 10_Wiki/Topics
+status: verified
+canonical_id: self
+aliases: [Ring Self-Attention, Distributed Attention]
+duplicate_of: none
+source_trust_level: A
+confidence_score: 0.95
+verification_status: applied
+tags: [attention, long-context, distributed-training, transformer, systems]
+raw_sources: []
+last_reinforced: 2026-05-10
+github_commit: pending
+tech_stack:
+  language: Python
+  framework: JAX/PyTorch/CUDA
+---
+
+# Ring Attention
+
+## 매 한 줄
+> **"매 attention 의 sequence 의 N 의 device 의 ring 의 split — context length scales linearly with devices."**. Liu, Zaharia, Abbeel 2023 ("Ring Attention with Blockwise Transformers") 의 propose, 매 1M+ context window (Gemini 1.5 Pro, Claude Opus 4.7 1M) 의 training-time enabler 의, 매 communication overlap with compute 의 near-zero overhead.
+
+## 매 핵심
+
+### 매 핵심 idea
+- Sequence 의 N device 의 split (each device holds 1/N tokens of Q, K, V).
+- Each device computes attention with its local Q against rotating K, V blocks.
+- K, V blocks travel ring N steps; communication 의 attention compute 와 overlap.
+- Result: full sequence attention 의 device 의 N 배 의 longer context 의 fit.
+
+### 매 vs alternatives
+- **Flash Attention**: single device, IO-aware, memory-efficient. Ring composes on top.
+- **Sequence Parallel (Megatron)**: similar split but layernorm/dropout only.
+- **Context Parallel (Megatron 2024)**: industrial Ring Attention variant.
+- **Striped Attention** (2023): improved load balance for causal masks.
+
+### 매 응용
+1. 1M+ context LLM training (Gemini 1.5/2.0, Claude Opus 4.x).
+2. Long video understanding.
+3. Whole-codebase code models.
+4. Long DNA sequence models (Evo).
+
+## 💻 패턴
+
+### Conceptual Ring Loop (single block)
+```python
+import torch
+import torch.distributed as dist
+
+def ring_attention_step(q_local, kv_local, world_size):
+    """매 simplified single-pass illustration."""
+    out = torch.zeros_like(q_local)
+    lse = torch.full(q_local.shape[:-1], -float("inf"), device=q_local.device)
+
+    k, v = kv_local
+    rank = dist.get_rank()
+
+    for step in range(world_size):
+        # local attention partial
+        partial_out, partial_lse = blockwise_attention(q_local, k, v)
+        out, lse = online_softmax_merge(out, lse, partial_out, partial_lse)
+
+        # rotate K, V to next neighbor (overlap with next compute)
+        send_rank = (rank - 1) % world_size
+        recv_rank = (rank + 1) % world_size
+        k, v = ring_send_recv(k, v, send_rank, recv_rank)
+    return out
+```
+
+### Online Softmax Merge
+```python
+def online_softmax_merge(out_a, lse_a, out_b, lse_b):
+    """매 numerically stable merge of 2 partial attention results."""
+    m = torch.maximum(lse_a, lse_b)
+    c_a = torch.exp(lse_a - m).unsqueeze(-1)
+    c_b = torch.exp(lse_b - m).unsqueeze(-1)
+    out = (c_a * out_a + c_b * out_b) / (c_a + c_b)
+    new_lse = m + torch.log(torch.exp(lse_a - m) + torch.exp(lse_b - m))
+    return out, new_lse
+```
+
+### Ring Send/Recv (NCCL)
+```python
+def ring_send_recv(k, v, send_rank, recv_rank):
+    k_buf = torch.empty_like(k)
+    v_buf = torch.empty_like(v)
+    reqs = [
+        dist.isend(k, send_rank), dist.isend(v, send_rank),
+        dist.irecv(k_buf, recv_rank), dist.irecv(v_buf, recv_rank),
+    ]
+    for r in reqs: r.wait()
+    return k_buf, v_buf
+```
+
+### Striped (Causal-aware) Block Order
+```python
+def striped_block_order(seq_len, world_size, block_size):
+    """매 causal mask 의 load balance 의 — interleave 의 X stride."""
+    n_blocks = seq_len // block_size
+    return [(i * world_size + r) % n_blocks
+            for r in range(world_size)
+            for i in range(n_blocks // world_size)]
+```
+
+### Causal Mask Skip Optimization
+```python
+def should_compute(q_block_idx, kv_block_idx, causal=True):
+    """매 causal: skip 의 kv > q (future)."""
+    return (not causal) or kv_block_idx <= q_block_idx
+```
+
+### Compute/Comm Overlap (CUDA streams)
+```python
+def overlapped_step(q, k, v, next_kv_handles, compute_stream, comm_stream):
+    with torch.cuda.stream(compute_stream):
+        partial = blockwise_attention(q, k, v)
+    with torch.cuda.stream(comm_stream):
+        next_k, next_v = ring_send_recv(k, v, ...)
+    torch.cuda.synchronize()
+    return partial, next_k, next_v
+```
+
+### JAX Ring Attention (high-level)
+```python
+import jax
+from jax.experimental.shard_map import shard_map
+from jax.sharding import PartitionSpec as P
+
+@jax.jit
+def ring_attn_pjit(q, k, v, mesh):
+    return shard_map(
+        ring_attention_fn,
+        mesh=mesh,
+        in_specs=(P("seq", None), P("seq", None), P("seq", None)),
+        out_specs=P("seq", None),
+    )(q, k, v)
+```
+
+## 매 결정 기준
+| 상황 | Approach |
+|---|---|
+| <32K context, single GPU | Flash Attention 3 only |
+| 32K–256K context, single node | Flash Attention + sequence parallel |
+| 256K–10M context, multi-node | Ring Attention (Striped variant) |
+| Causal model | Striped Ring Attention (load balance) |
+| TPU pod | JAX shard_map + Ring |
+
+**기본값**: Striped Ring Attention with online softmax + NCCL ring + Flash Attention kernel as inner block.
+
+## 🔗 Graph
+- 부모: [[Transformer_Architecture_and_LLM_Foundations|Attention Mechanism]] · [[Distributed Training]]
+- 응용: [[Gemini]]
+- Adjacent: [[Flash Attention]] · [[Online Softmax]]
+
+## 🤖 LLM 활용
+**언제**: 매 long-context model 의 train/serve 의 evaluating, infra design 의 시.
+**언제 X**: 매 inference-only at small context 의 X — Flash Attention 만 의 sufficient.
+
+## ❌ 안티패턴
+- **Naive ring without overlap**: communication 의 sequential 의 → no speedup.
+- **Causal mask ignored**: 매 lower-triangle 의 50% compute 의 wasted 의 X — striped order 의 fix.
+- **Float32 accumulation skipped**: long context 의 numerical drift — fp32 LSE 의 keep.
+- **Pure data parallel for long context**: memory-bound — Ring or context parallel 의 use.
+- **Block size 의 cache 의 fit X**: bandwidth-bound — tune block_size to L2.
+
+## 🧪 검증 / 중복
+- Verified (Liu et al. 2023 arXiv:2310.01889; Megatron-LM Context Parallelism docs 2024).
+- 신뢰도 A.
+
+## 🕓 Changelog
+| 날짜 | 변경 |
+|---|---|
+| 2026-05-08 | Phase 1 |
+| 2026-05-10 | Manual cleanup — Ring Attention algo + Striped variant + JAX/PyTorch patterns |