2nd/10_Wiki/Topic_Programming/Architecture/Pipeline-Parallelism.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-pipeline-parallelism	Pipeline Parallelism	10_Wiki/Topics	verified	self	PP GPipe 1F1B	none	A	0.9	applied	parallelism distributed-training deep-learning llm		2026-05-10	pending	language framework python pytorch

Pipeline Parallelism

매 한 줄

"매 모델을 layer-wise로 잘라 GPU pipeline 위로 micro-batch가 흐르게 한다". 매 GPipe(2018)에서 시작, PipeDream / 1F1B / Interleaved 1F1B로 진화. 매 2026 LLM 학습(>100B params)에서 TP+PP+DP 조합의 한 축.

매 핵심

매 왜 PP인가

• 매 단일 GPU 의 memory(HBM3 80–192GB) 의 초과 → layer 분할 필수.
• 매 Tensor Parallelism 의 NVLink 안 high-bandwidth requirement → 매 node 간 한계.
• 매 Pipeline Parallelism 의 stage 간 activation 만 전달 → 매 inter-node OK.

매 stage / micro-batch

• Stage = 매 연속 layer 묶음, GPU 1개 차지.
• Mini-batch 의 micro-batch K개로 split → 매 동시에 다른 stage에서 처리.
• Bubble = 매 idle time. Bubble ratio ≈ (stages - 1) / K.

매 schedule 계열

1. GPipe: 매 forward all → backward all. 매 simple, 큰 bubble.
2. 1F1B (PipeDream): 매 1 forward, 1 backward 교대. 매 activation memory 절감.
3. Interleaved 1F1B (Megatron): 매 stage 마다 여러 chunk → bubble 감소.
4. Zero Bubble PP (2024): 매 backward를 W/B로 split → 매 거의 0 bubble.

💻 패턴

PyTorch native PipelineStage (torch.distributed.pipelining)

import torch
import torch.nn as nn
from torch.distributed.pipelining import pipeline, ScheduleGPipe, SplitPoint

class Block(nn.Module):
    def __init__(self, d): super().__init__(); self.l = nn.Linear(d, d)
    def forward(self, x): return torch.relu(self.l(x))

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.b1 = Block(1024); self.b2 = Block(1024)
        self.b3 = Block(1024); self.b4 = Block(1024)
    def forward(self, x):
        return self.b4(self.b3(self.b2(self.b1(x))))

model = Net()
example = torch.randn(8, 1024)

pipe = pipeline(
    model, mb_args=(example,),
    split_spec={"b3": SplitPoint.BEGINNING},  # stage0: b1-b2, stage1: b3-b4
)
stage = pipe.build_stage(stage_index=rank, device=f"cuda:{rank}")
sched = ScheduleGPipe(stage, n_microbatches=4, loss_fn=nn.MSELoss())

1F1B schedule 계산

def schedule_1f1b(num_stages: int, num_microbatches: int):
    """매 stage 별 forward/backward 순서 emit"""
    ops = [[] for _ in range(num_stages)]
    warmup = num_stages
    for s in range(num_stages):
        n_warm = min(warmup - s, num_microbatches)
        for mb in range(n_warm):
            ops[s].append(("F", mb))
        for mb in range(num_microbatches - n_warm):
            ops[s].append(("F", n_warm + mb))
            ops[s].append(("B", mb))
        for mb in range(num_microbatches - n_warm, num_microbatches):
            ops[s].append(("B", mb))
    return ops

Megatron-LM virtual pipeline

# v_chunks=2 → stage0 holds {layer 0-7, layer 16-23}, stage1 holds {8-15, 24-31}
config = TransformerConfig(
    num_layers=32, hidden_size=8192,
    pipeline_model_parallel_size=4,
    virtual_pipeline_model_parallel_size=2,  # interleaved chunks
    num_microbatches=64,
)

Activation recompute (memory bubble 완화)

from torch.utils.checkpoint import checkpoint

class CheckpointedBlock(nn.Module):
    def forward(self, x):
        return checkpoint(self._fwd, x, use_reentrant=False)
    def _fwd(self, x): return self.attn(self.norm(x)) + x

DeepSpeed PipelineModule

import deepspeed
from deepspeed.pipe import PipelineModule, LayerSpec

specs = [LayerSpec(Block, 1024) for _ in range(8)]
model = PipelineModule(layers=specs, num_stages=4, partition_method="uniform")
engine, _, _, _ = deepspeed.initialize(model=model, config=ds_config)
loss = engine.train_batch(data_iter)

3D parallelism (TP × PP × DP)

# 매 Megatron / NeMo 의 conventional layout
# world_size = TP × PP × DP
# Llama 3 405B 학습: TP=8, PP=16, DP=128 → 16384 GPUs
mesh = init_device_mesh("cuda", (DP, PP, TP), mesh_dim_names=("dp","pp","tp"))

매 결정 기준

상황	Approach
매 single node, ≤8 GPU	TP only (NVLink)
매 multi-node, model > node mem	TP intra-node + PP inter-node
매 100B+ params	TP × PP × DP (3D)
매 inference latency 중요	TP > PP (PP의 bubble 손해)
매 throughput 중심 training	PP + DP 큰 micro-batch

기본값: 매 LLM 학습은 1F1B + activation recompute + 3D parallel.

🔗 Graph

• 부모: Distributed Training

🤖 LLM 활용

언제: 매 모델 weight 가 단일 GPU mem 초과 + 매 multi-node training. 매 cross-node bandwidth 가 TP에 부족할 때. 언제 X: 매 단일 node 안 fits. 매 매우 작은 batch (bubble 비율 폭증). 매 inference latency-critical.

❌ 안티패턴

• Bubble ignore: 매 micro-batch K=1 → 매 GPU의 (stages-1)/stages 가 idle.
• Uneven partition: 매 stage 별 FLOPs 불균형 → 매 가장 느린 stage 가 throughput 결정.
• PP only no DP: 매 K 늘려도 batch size 한계 → 매 DP 병행 필수.
• Embedding 분리 무시: 매 input/output embedding 의 같은 stage 배치 → tied weight sync 단순.

🧪 검증 / 중복

• Verified (Megatron-LM paper, GPipe, PipeDream, PyTorch pipelining docs 2026).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — PP schedules + 3D parallel patterns