docs(10_Wiki): 위키 전체 재구성 — Topic_* 폴더를 4개 카테고리로 통합 + 대규모 중복 제거
Topic_Agent/Topic_Blog/Topics/Topics_Biz/Topics_Meeting/Topics_Rag의 마크다운 지식 문서를 Topic_General/Topic_Programming/Topic_Graphic/Topic_Business 4개 카테고리로 재분류. - 중복 제거: frontmatter의 status:duplicate/merged + duplicate_of/redirect_to 필드로 자기 자신을 중복으로 선언한 리다이렉트 stub 1032개 제거, 완전 동일 내용 파일 472개 제거, 동일 파일명·다른 내용 충돌 시 더 큰(완전한) 버전만 유지(162개 제거) — 총 1639개 중복 제거. - 분류: 폴더 단위로 명확한 항목(AI_and_ML/Coding/Architecture 등 → Programming, Comfyui/Visual_Effects → Graphic, Topics_Biz/Topics_Meeting/사업 등 → Business, Poetic_Blog_Writing/창의성/Game_Design 등 → General)은 폴더 우선순위로, 나머지 혼재 폴더(Topic_Agent/Topic_Blog/Topics 루트/Thinking & Reasoning/Other/UI_UX_Assets)는 title/tags 키워드 스코어링으로 파일 단위 분류(불명확한 경우 General로 폴백). 원본 폴더명은 "From_*" 서브폴더로 보존해 추적 가능성 유지. - 최종 배치: Programming 2784 / General 1608 / Graphic 285 / Business 249 = 4926개 문서. - 에이전트 운영 상태(.astra/.agent/.obsidian/sessions/memory/_company/docs/lessons/_shared/src)는 지식 콘텐츠가 아니므로 재분류 대상에서 제외하고 원위치 유지. - Topics/Topic_email(상위 보호 폴더 Topic_email과 파일명 100% 중복) 삭제 — 보호 폴더 자체는 미변경. - 완전히 비게 된 Topic_Agent/Topic_Blog/Topics_Biz/Topics_Rag 폴더 제거.
This commit is contained in:
Antigravity Agent
@@ -0,0 +1,208 @@
|
||||
---
|
||||
id: wiki-2026-0508-scalability-in-ai-systems
|
||||
title: Scalability in AI Systems
|
||||
category: 10_Wiki/Topics
|
||||
status: verified
|
||||
canonical_id: self
|
||||
aliases: [AI Scalability, Distributed Training, LLM Scaling]
|
||||
duplicate_of: none
|
||||
source_trust_level: A
|
||||
confidence_score: 0.9
|
||||
verification_status: applied
|
||||
tags: [scalability, distributed-training, inference, vllm, fsdp, llm]
|
||||
raw_sources: []
|
||||
last_reinforced: 2026-05-10
|
||||
github_commit: pending
|
||||
tech_stack:
|
||||
language: Python
|
||||
framework: PyTorch / vLLM / DeepSpeed
|
||||
---
|
||||
|
||||
# Scalability in AI Systems
|
||||
|
||||
## 매 한 줄
|
||||
> **"매 model size × data × users × latency 의 4축 동시 만족 — 매 single GPU → multi-node training (FSDP/ZeRO/TP/PP) 과 매 10K+ QPS inference (vLLM/SGLang/PagedAttention) 의 양 갈래"**. 매 2020 GPT-3 (175B) 부터 매 2026 trillion-param MoE (Llama 4 Behemoth, Claude Opus 4.7, GPT-5) 까지 매 scaling laws (Chinchilla, Hoffmann 2022) 가 산업 의 compass.
|
||||
|
||||
## 매 핵심
|
||||
|
||||
### 매 Training scalability axes
|
||||
- **Data parallel (DP)**: replicate model, shard batch — gradient all-reduce.
|
||||
- **Tensor parallel (TP)**: split single layer across GPUs (Megatron-LM).
|
||||
- **Pipeline parallel (PP)**: split layers across stages (GPipe, 1F1B).
|
||||
- **FSDP / ZeRO**: shard params + grads + optimizer state (ZeRO-1/2/3).
|
||||
- **Sequence/Context parallel**: shard along sequence (Ring Attention, DeepSpeed-Ulysses).
|
||||
- **MoE expert parallel**: route tokens to expert subsets across GPUs.
|
||||
|
||||
### 매 Inference scalability
|
||||
- **Continuous batching**: vLLM / TGI — token-level scheduling, no head-of-line block.
|
||||
- **PagedAttention**: KV cache paged like virtual memory → high concurrency.
|
||||
- **Prefix caching**: shared system prompt cache (vLLM, SGLang).
|
||||
- **Speculative decoding**: small draft model proposes, large verifies → 2-3x.
|
||||
- **Quantization**: FP8, INT4 (AWQ, GPTQ), MX formats (MXFP4) — 2026 H200/B200 native.
|
||||
- **Disaggregated serving**: prefill nodes vs decode nodes (Splitwise, DistServe).
|
||||
|
||||
### 매 응용
|
||||
1. Pretraining 70B+ on 256+ GPU cluster.
|
||||
2. LoRA/QLoRA fine-tune on single H100 (24GB shard).
|
||||
3. 100K+ concurrent chatbot QPS via vLLM cluster.
|
||||
4. RAG at 1M+ docs with sharded vector DB.
|
||||
5. Multi-tenant inference with tenant-aware caching.
|
||||
|
||||
## 💻 패턴
|
||||
|
||||
### FSDP2 with PyTorch (2026 idiom)
|
||||
```python
|
||||
import torch
|
||||
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy
|
||||
|
||||
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B")
|
||||
mp_policy = MixedPrecisionPolicy(
|
||||
param_dtype=torch.bfloat16,
|
||||
reduce_dtype=torch.float32,
|
||||
)
|
||||
for layer in model.model.layers:
|
||||
fully_shard(layer, mp_policy=mp_policy)
|
||||
fully_shard(model, mp_policy=mp_policy)
|
||||
|
||||
# Gradient checkpointing for memory
|
||||
model.gradient_checkpointing_enable()
|
||||
```
|
||||
|
||||
### DeepSpeed ZeRO-3 config
|
||||
```json
|
||||
{
|
||||
"train_batch_size": 1024,
|
||||
"gradient_accumulation_steps": 4,
|
||||
"bf16": {"enabled": true},
|
||||
"zero_optimization": {
|
||||
"stage": 3,
|
||||
"offload_param": {"device": "cpu"},
|
||||
"offload_optimizer": {"device": "nvme", "nvme_path": "/mnt/nvme"},
|
||||
"overlap_comm": true,
|
||||
"contiguous_gradients": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### vLLM serving (continuous batching + prefix cache)
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
llm = LLM(
|
||||
model="meta-llama/Llama-3.1-70B-Instruct",
|
||||
tensor_parallel_size=4,
|
||||
enable_prefix_caching=True,
|
||||
max_model_len=128000,
|
||||
quantization="fp8",
|
||||
gpu_memory_utilization=0.92,
|
||||
)
|
||||
out = llm.generate(prompts, SamplingParams(max_tokens=512, temperature=0.7))
|
||||
```
|
||||
|
||||
### SGLang RadixAttention (shared prefix tree)
|
||||
```python
|
||||
import sglang as sgl
|
||||
|
||||
sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))
|
||||
|
||||
@sgl.function
|
||||
def multi_turn(s, system, turns):
|
||||
s += sgl.system(system) # cached across all calls
|
||||
for t in turns:
|
||||
s += sgl.user(t["q"]) + sgl.assistant(sgl.gen(max_tokens=256))
|
||||
|
||||
# Massive throughput when many users share system prompt
|
||||
```
|
||||
|
||||
### Speculative decoding (vLLM)
|
||||
```python
|
||||
llm = LLM(
|
||||
model="meta-llama/Llama-3.1-70B-Instruct",
|
||||
speculative_model="meta-llama/Llama-3.2-1B-Instruct",
|
||||
num_speculative_tokens=5,
|
||||
use_v2_block_manager=True,
|
||||
)
|
||||
```
|
||||
|
||||
### Megatron-LM tensor + pipeline parallel
|
||||
```bash
|
||||
torchrun --nproc_per_node=8 --nnodes=8 \
|
||||
pretrain_gpt.py \
|
||||
--tensor-model-parallel-size 8 \
|
||||
--pipeline-model-parallel-size 8 \
|
||||
--num-layers 80 --hidden-size 8192 \
|
||||
--num-attention-heads 64 --seq-length 8192 \
|
||||
--micro-batch-size 1 --global-batch-size 1024 \
|
||||
--bf16 --use-flash-attn
|
||||
```
|
||||
|
||||
### Distributed inference K8s (vLLM + Ray Serve)
|
||||
```python
|
||||
from ray import serve
|
||||
from vllm.entrypoints.openai.api_server import app as vllm_app
|
||||
|
||||
@serve.deployment(num_replicas=4, ray_actor_options={"num_gpus": 2})
|
||||
@serve.ingress(vllm_app)
|
||||
class LLMService:
|
||||
def __init__(self, model):
|
||||
self.engine = AsyncLLMEngine.from_engine_args(
|
||||
AsyncEngineArgs(model=model, tensor_parallel_size=2)
|
||||
)
|
||||
```
|
||||
|
||||
### KV cache sizing rule of thumb
|
||||
```python
|
||||
def kv_cache_bytes(layers, hidden_dim, seq_len, batch, dtype_bytes=2, kv_heads=None):
|
||||
h = kv_heads or hidden_dim # GQA: kv_heads < hidden
|
||||
return 2 * batch * layers * seq_len * h * dtype_bytes
|
||||
# Llama-3-70B at 128K context, batch=1, GQA(8): ~10 GB KV
|
||||
```
|
||||
|
||||
### Disaggregated prefill/decode (Splitwise pattern)
|
||||
```python
|
||||
# Prefill cluster: H100, optimized for compute
|
||||
# Decode cluster: H100/L40S, optimized for memory bandwidth
|
||||
# KV transfer over NVLink/RDMA between clusters
|
||||
prefill_resp = await prefill_client.prefill(prompt) # returns KV blocks ref
|
||||
decode_stream = decode_client.decode(kv_ref=prefill_resp.kv_ref, max_tokens=512)
|
||||
```
|
||||
|
||||
## 매 결정 기준
|
||||
| 상황 | Approach |
|
||||
|---|---|
|
||||
| <13B model train | DDP / FSDP1 single node 8xGPU |
|
||||
| 70B train | FSDP2 / ZeRO-3 multi-node + activation checkpoint |
|
||||
| 400B+ MoE | TP + PP + EP + ZeRO-1 (Megatron-Core / NeMo) |
|
||||
| Chatbot serving | vLLM + prefix cache + speculative decoding |
|
||||
| Long context (1M+) | Ring attention / context parallel |
|
||||
| High concurrency | SGLang RadixAttention + disaggregated |
|
||||
|
||||
**기본값**: FSDP2 for training under 100B; vLLM with FP8 + prefix cache + spec decoding for serving.
|
||||
|
||||
## 🔗 Graph
|
||||
- 부모: [[Distributed-Systems]]
|
||||
- 변형: [[Pipeline-Parallelism]]
|
||||
- 응용: [[Fine-tuning]]
|
||||
- Adjacent: [[LLM_Optimization_and_Deployment_Strategies|vLLM]] · [[Flash Attention]] · [[LLM_Optimization_and_Deployment_Strategies|Quantization]]
|
||||
|
||||
## 🤖 LLM 활용
|
||||
**언제**: capacity planning estimates, config-file generation (deepspeed, accelerate), debugging OOM via log triage, scaling-law back-of-envelope.
|
||||
**언제 X**: actually scheduling kernels — the orchestrator (PyTorch/vLLM) handles deterministic scheduling.
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **All-reduce on tiny gradients**: communication dominates; bucket gradients.
|
||||
- **Naive batching for serving**: head-of-line blocking — use continuous batching.
|
||||
- **Unbounded KV cache**: OOM at peak; use PagedAttention + admission control.
|
||||
- **No prefix cache for shared system prompt**: 5-10x throughput left on table.
|
||||
- **PP without enough microbatches**: bubble dominates pipeline.
|
||||
- **MoE without expert balance loss**: dead experts, capacity waste.
|
||||
|
||||
## 🧪 검증 / 중복
|
||||
- Verified (Megatron-LM/Megatron-Core, vLLM paper SOSP 2023, Chinchilla 2022, NVIDIA H100/B200 perf guides 2026).
|
||||
- 신뢰도 A.
|
||||
|
||||
## 🕓 Changelog
|
||||
| 날짜 | 변경 |
|
||||
|---|---|
|
||||
| 2026-05-08 | Phase 1 |
|
||||
| 2026-05-10 | Manual cleanup — FSDP/ZeRO/TP/PP, vLLM/SGLang, disaggregated serving |
|
||||
Reference in New Issue
Block a user