2nd/10_Wiki/Topics/AI_and_ML/NLP-Attention-Mechanisms.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-nlp-attention-mechanisms	NLP Attention Mechanisms	10_Wiki/Topics	verified	self	Attention Self-Attention Multi-Head Attention Bahdanau Luong Flash Attention	none	A	0.95	applied	nlp attention transformer deep-learning flash-attention		2026-05-10	pending	language framework python pytorch

NLP Attention Mechanisms

매 한 줄

Attention은 시퀀스 내 토큰 간 가중 의존성을 동적으로 학습하는 메커니즘으로, Bahdanau(2014) additive → Luong multiplicative → scaled dot-product → multi-head → Flash Attention 진화를 거쳐 모든 현대 LLM의 코어가 되었다.

매 핵심

1. Attention 일반 공식

attention(Q, K, V) = softmax(score(Q, K)) · V

• score 함수가 변형의 핵심: additive vs multiplicative vs dot-product.
• 출력은 V의 가중합, 가중치는 Q-K 유사도.

2. Bahdanau (Additive, 2014)

score(q, k) = vᵀ tanh(W_q q + W_k k)

• MLP 기반 — 학습 파라미터 많음.
• 작은 차원에서 더 풍부한 매칭.
• seq2seq 번역 (RNN encoder-decoder) 출시.

3. Luong (Multiplicative, 2015)

score(q, k) = qᵀ W k         (general)
score(q, k) = qᵀ k           (dot)

• 행렬 곱 1번 — 빠름.
• GPU 친화적.

4. Scaled Dot-Product (Vaswani 2017, Transformer)

Attention(Q,K,V) = softmax(QKᵀ / √d_k) V

• √d_k로 나눠 큰 차원 softmax saturation 방지.
• 행렬 연산 — 병렬 GPU 최적.
• 이게 Transformer의 핵심.

5. Multi-Head Attention

MHA(Q,K,V) = Concat(head_1, ..., head_h) W^O
head_i = Attention(Q W^Q_i, K W^K_i, V W^V_i)

• h개의 head가 서로 다른 sub-space에서 attention.
• 표현력 ↑ — 한 head는 syntactic, 다른 head는 semantic.
• 표준 h = 8, 16, 32, 64 (모델 크기 의존).

6. Self vs Cross Attention

• Self: Q=K=V (같은 시퀀스) — encoder, decoder masked.
• Cross: Q from decoder, K=V from encoder — encoder-decoder bridge.

7. Causal / Masked Attention

• Decoder에서 미래 토큰 참조 차단 (-inf 마스크).
• LLM autoregressive 생성 표준.

8. Positional Encoding

• Attention은 순서 무인지 → 위치 정보 추가 필요.
• Sinusoidal (원조), Learned, RoPE (Rotary, LLaMA/현대 LLM 표준), ALiBi.

9. Modern: Flash Attention (Dao 2022, FA2 2023, FA3 2024)

• IO-aware 알고리즘: GPU SRAM 활용해 HBM 왕복 최소화.
• 정확한 attention (근사 X) — 2-4× 빠름, 메모리 5-20× 절감.
• 긴 컨텍스트(100K-1M token)의 게임 체인저.
• FA2: warp-level 병렬화. FA3 (Hopper): WGMMA + async.

10. 효율 변형

• MQA (Multi-Query Attention): KV 헤드 1개 — 추론 빠름.
• GQA (Grouped-Query): KV 헤드 그룹화 — LLaMA-2/3 표준.
• Sliding Window: local attention (Mistral).
• Sparse / Linear / Linformer / Performer: O(n²) → O(n log n) 또는 O(n).
• Ring Attention: 분산 long-context (Gemini 2M).

💻 패턴

# 1. Bahdanau additive (PyTorch)
import torch, torch.nn as nn
class BahdanauAttention(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.W_q = nn.Linear(d, d); self.W_k = nn.Linear(d, d); self.v = nn.Linear(d, 1)
    def forward(self, q, k, v):
        # q: (B, 1, d), k/v: (B, T, d)
        score = self.v(torch.tanh(self.W_q(q) + self.W_k(k))).squeeze(-1)  # (B, T)
        a = torch.softmax(score, dim=-1)
        return (a.unsqueeze(-1) * v).sum(dim=1)

# 2. Scaled dot-product
def scaled_dot_product(Q, K, V, mask=None):
    d_k = Q.size(-1)
    scores = (Q @ K.transpose(-2, -1)) / d_k**0.5
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    return torch.softmax(scores, dim=-1) @ V

# 3. Multi-head (from scratch)
class MHA(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h, self.dh = h, d // h
        self.qkv = nn.Linear(d, 3*d); self.o = nn.Linear(d, d)
    def forward(self, x, mask=None):
        B, T, D = x.shape
        q,k,v = self.qkv(x).chunk(3, dim=-1)
        q,k,v = [t.view(B,T,self.h,self.dh).transpose(1,2) for t in (q,k,v)]
        out = scaled_dot_product(q,k,v,mask).transpose(1,2).reshape(B,T,D)
        return self.o(out)

# 4. PyTorch built-in
mha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
out, attn_weights = mha(x, x, x)  # self-attention

# 5. Causal mask (decoder)
T = 1024
mask = torch.tril(torch.ones(T, T)).bool()  # lower-tri: 1 keep, 0 mask

# 6. Flash Attention (xformers / pytorch SDPA backend)
import torch.nn.functional as F
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
# PyTorch 2.0+ 자동 Flash Attention backend

# 7. RoPE (Rotary Position Embedding)
def rope(x, freqs):
    # x: (..., d), freqs: (T, d/2)
    x1, x2 = x.chunk(2, dim=-1)
    cos, sin = freqs.cos(), freqs.sin()
    return torch.cat([x1*cos - x2*sin, x1*sin + x2*cos], dim=-1)

# 8. GQA (Grouped-Query)
class GQA(nn.Module):
    def __init__(self, d, n_q_heads, n_kv_heads):
        super().__init__()
        self.n_q, self.n_kv = n_q_heads, n_kv_heads
        self.dh = d // n_q_heads
        self.q = nn.Linear(d, n_q_heads * self.dh)
        self.k = nn.Linear(d, n_kv_heads * self.dh)
        self.v = nn.Linear(d, n_kv_heads * self.dh)
    # KV broadcast n_q / n_kv 배 반복하여 attention

# 9. Sliding window (Mistral-style)
def sliding_window_mask(T, window=4096):
    m = torch.tril(torch.ones(T, T))
    m = m - torch.tril(torch.ones(T, T), diagonal=-window-1)
    return m.bool()

# 10. Visualizing attention
import matplotlib.pyplot as plt
attn = mha(x, x, x, need_weights=True, average_attn_weights=False)[1]  # (B, h, T, T)
plt.imshow(attn[0, 0].cpu().numpy())  # head 0

매 결정 기준

상황	추천
Transformer 표준	Scaled dot-product MHA
긴 컨텍스트 (>32K)	Flash Attention 2/3
추론 속도 (LLM)	GQA / MQA + KV cache
Local 패턴 충분	Sliding window (Mistral)
Encoder-decoder 번역	Cross attention
작은 모델 + 작은 d	Bahdanau additive (legacy)
위치 표현	RoPE (modern) / ALiBi (long ctx)
1M+ 컨텍스트 분산	Ring Attention

🔗 Graph

• 부모: Transformer_Architecture_and_LLM_Foundations
• 변형: Multi-Head Attention, Flash Attention, Grouped-Query Attention
• 응용: Transformer_Architecture_and_LLM_Foundations, Transformer_Architecture_and_LLM_Foundations, GPT
• Adjacent: KV Cache, Long Context

🤖 LLM 활용

• "이 attention map을 보고 모델이 어떤 토큰에 의존하는지 분석" — interpretability.
• 코드에 SDPA / FlashAttention 적용 자동 리팩토링.
• Attention 변형 비교표 생성, ablation 가이드.

❌ 안티패턴

• √d_k 정규화 누락: 큰 d에서 softmax saturation → gradient 소실.
• Causal mask 없는 decoder: 미래 leak → 학습/추론 불일치.
• 벡터화 안 한 attention 루프: 100배 느림.
• MHA 추론 + KV cache 없음: 긴 생성에서 O(n²) 재계산.
• Vanilla attention + 100K context: OOM — Flash Attention 필수.
• Position encoding 누락: bag-of-words처럼 동작.

🧪 검증 / 중복

• 검증: Vaswani(2017) "Attention is All You Need", Bahdanau(2014), Dao(2022 FlashAttention), HuggingFace docs.
• 중복: Multi-Head Attention, Flash Attention (specific) — 본 문서는 family overview.

🕓 Changelog

• 2026-05-10: 신규 작성. Bahdanau→Luong→SDPA→MHA→Flash→GQA/MQA/Sliding 진화 + 코드 패턴.