docs(10_Wiki): Topic_Business/General/Graphic/Programming을 Topics/ 하위로 이동
최상위 10_Wiki/Topic_*였던 4개 카테고리 폴더를 10_Wiki/Topics/Topic_* 로 재배치. 콘텐츠 변경 없음(순수 폴더 이동) — Topics/ 하위 나머지 폴더는 이미 지난 커밋에서 전부 정리된 상태(잔존 항목은 에이전트 운영 상태 및 사용자가 보존을 요청한 업데이트0615/무제 3.canvas 뿐).
This commit is contained in:
Antigravity Agent
@@ -0,0 +1,236 @@
|
||||
---
|
||||
id: wiki-2026-0508-grouped-query-attention-gqa
|
||||
title: Grouped-Query Attention (GQA)
|
||||
category: 10_Wiki/Topics
|
||||
status: verified
|
||||
canonical_id: self
|
||||
aliases: [GQA, grouped-query attention, MQA, multi-query attention, KV cache reduction, Llama]
|
||||
duplicate_of: none
|
||||
source_trust_level: A
|
||||
confidence_score: 0.96
|
||||
verification_status: applied
|
||||
tags: [transformer, attention, gqa, mqa, kv-cache, llama, inference-optimization]
|
||||
raw_sources: []
|
||||
last_reinforced: 2026-05-10
|
||||
github_commit: pending
|
||||
tech_stack:
|
||||
language: Python
|
||||
framework: PyTorch / vLLM
|
||||
---
|
||||
|
||||
# Grouped-Query Attention (GQA)
|
||||
|
||||
## 매 한 줄
|
||||
> **"매 multi-head attention 와 multi-query attention 의 가운데"**. Ainslie 2023 (Google). 매 Q heads = N, K/V heads = G (G < N). 매 KV cache size ↓ + 매 quality 의 MHA 와 가까움. 매 Llama 2 70B+, Mistral, 모든 modern LLM 의 standard.
|
||||
|
||||
## 매 핵심
|
||||
|
||||
### 매 spectrum
|
||||
- **MHA**: Q=N, K=N, V=N (예: 32/32/32).
|
||||
- **MQA**: Q=N, K=1, V=1 (예: 32/1/1).
|
||||
- **GQA**: Q=N, K=G, V=G (예: 32/8/8).
|
||||
|
||||
### 매 trade-off
|
||||
- **MHA**: 매 best quality, 매 largest KV cache.
|
||||
- **MQA**: 매 smallest cache, 매 quality 매 ↓.
|
||||
- **GQA**: 매 sweet spot.
|
||||
|
||||
### 매 inference impact
|
||||
- **KV cache** = batch × seq_len × n_layers × n_kv_heads × head_dim × 2 (K, V).
|
||||
- 매 GQA: 매 N → G 의 의 의 cache 의 N/G 배 reduce.
|
||||
|
||||
### 매 응용
|
||||
- **Llama 2 70B**: 32 Q heads, 8 KV heads.
|
||||
- **Llama 3**: GQA 표준.
|
||||
- **Mistral**, **Mixtral**: GQA.
|
||||
- **Gemma**, **Qwen**: GQA.
|
||||
|
||||
## 💻 패턴
|
||||
|
||||
### MHA (baseline)
|
||||
```python
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
class MultiHeadAttention(torch.nn.Module):
|
||||
def __init__(self, dim, n_heads):
|
||||
super().__init__()
|
||||
self.n_heads = n_heads
|
||||
self.head_dim = dim // n_heads
|
||||
self.q_proj = torch.nn.Linear(dim, dim)
|
||||
self.k_proj = torch.nn.Linear(dim, dim)
|
||||
self.v_proj = torch.nn.Linear(dim, dim)
|
||||
self.o_proj = torch.nn.Linear(dim, dim)
|
||||
|
||||
def forward(self, x):
|
||||
B, T, C = x.shape
|
||||
q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
|
||||
k = self.k_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
|
||||
v = self.v_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
|
||||
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
||||
return self.o_proj(out.transpose(1, 2).reshape(B, T, C))
|
||||
```
|
||||
|
||||
### GQA
|
||||
```python
|
||||
class GQA(torch.nn.Module):
|
||||
def __init__(self, dim, n_q_heads, n_kv_heads):
|
||||
super().__init__()
|
||||
self.n_q_heads = n_q_heads
|
||||
self.n_kv_heads = n_kv_heads
|
||||
self.n_rep = n_q_heads // n_kv_heads
|
||||
self.head_dim = dim // n_q_heads
|
||||
|
||||
self.q_proj = torch.nn.Linear(dim, n_q_heads * self.head_dim)
|
||||
self.k_proj = torch.nn.Linear(dim, n_kv_heads * self.head_dim)
|
||||
self.v_proj = torch.nn.Linear(dim, n_kv_heads * self.head_dim)
|
||||
self.o_proj = torch.nn.Linear(dim, dim)
|
||||
|
||||
def forward(self, x):
|
||||
B, T, C = x.shape
|
||||
q = self.q_proj(x).view(B, T, self.n_q_heads, self.head_dim).transpose(1, 2)
|
||||
k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
||||
v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
||||
|
||||
# 매 repeat KV heads to match Q
|
||||
k = k.repeat_interleave(self.n_rep, dim=1)
|
||||
v = v.repeat_interleave(self.n_rep, dim=1)
|
||||
|
||||
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
||||
return self.o_proj(out.transpose(1, 2).reshape(B, T, C))
|
||||
```
|
||||
|
||||
### MQA (extreme)
|
||||
```python
|
||||
class MQA(torch.nn.Module):
|
||||
"""매 1 KV head."""
|
||||
def __init__(self, dim, n_q_heads):
|
||||
super().__init__()
|
||||
self.n_q_heads = n_q_heads
|
||||
self.head_dim = dim // n_q_heads
|
||||
self.q_proj = torch.nn.Linear(dim, dim)
|
||||
self.k_proj = torch.nn.Linear(dim, self.head_dim) # 매 single head
|
||||
self.v_proj = torch.nn.Linear(dim, self.head_dim)
|
||||
self.o_proj = torch.nn.Linear(dim, dim)
|
||||
|
||||
def forward(self, x):
|
||||
B, T, C = x.shape
|
||||
q = self.q_proj(x).view(B, T, self.n_q_heads, self.head_dim).transpose(1, 2)
|
||||
k = self.k_proj(x).view(B, T, 1, self.head_dim).transpose(1, 2).expand(-1, self.n_q_heads, -1, -1)
|
||||
v = self.v_proj(x).view(B, T, 1, self.head_dim).transpose(1, 2).expand(-1, self.n_q_heads, -1, -1)
|
||||
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
||||
return self.o_proj(out.transpose(1, 2).reshape(B, T, C))
|
||||
```
|
||||
|
||||
### KV cache size (calculate)
|
||||
```python
|
||||
def kv_cache_bytes(batch, seq_len, n_layers, n_kv_heads, head_dim, dtype_bytes=2):
|
||||
return batch * seq_len * n_layers * n_kv_heads * head_dim * 2 * dtype_bytes
|
||||
|
||||
# 매 Llama 2 70B (MHA): 80 layers, 64 heads, 128 dim
|
||||
# 매 batch=1, seq=2048
|
||||
# 매 cache = 1 * 2048 * 80 * 64 * 128 * 2 * 2 = 5.4 GB
|
||||
|
||||
# 매 Llama 2 70B (GQA): 8 KV heads
|
||||
# 매 cache = 1 * 2048 * 80 * 8 * 128 * 2 * 2 = 670 MB (8x ↓)
|
||||
```
|
||||
|
||||
### Convert MHA → GQA (mean)
|
||||
```python
|
||||
def mha_to_gqa(k_proj, v_proj, n_q_heads, n_kv_heads):
|
||||
"""매 mean of N/G heads → single GQA head."""
|
||||
n_rep = n_q_heads // n_kv_heads
|
||||
head_dim = k_proj.weight.size(0) // n_q_heads
|
||||
|
||||
new_k_weight = k_proj.weight.view(n_q_heads, head_dim, -1).reshape(n_kv_heads, n_rep, head_dim, -1).mean(dim=1).reshape(n_kv_heads * head_dim, -1)
|
||||
new_v_weight = v_proj.weight.view(n_q_heads, head_dim, -1).reshape(n_kv_heads, n_rep, head_dim, -1).mean(dim=1).reshape(n_kv_heads * head_dim, -1)
|
||||
|
||||
return new_k_weight, new_v_weight
|
||||
```
|
||||
|
||||
### Llama-style GQA + RoPE
|
||||
```python
|
||||
class LlamaAttention(torch.nn.Module):
|
||||
def __init__(self, config):
|
||||
super().__init__()
|
||||
self.n_q = config.num_attention_heads
|
||||
self.n_kv = config.num_key_value_heads # 매 GQA
|
||||
self.head_dim = config.hidden_size // self.n_q
|
||||
self.q_proj = torch.nn.Linear(config.hidden_size, self.n_q * self.head_dim, bias=False)
|
||||
self.k_proj = torch.nn.Linear(config.hidden_size, self.n_kv * self.head_dim, bias=False)
|
||||
self.v_proj = torch.nn.Linear(config.hidden_size, self.n_kv * self.head_dim, bias=False)
|
||||
self.o_proj = torch.nn.Linear(self.n_q * self.head_dim, config.hidden_size, bias=False)
|
||||
self.rotary = RoPE(self.head_dim, config.max_position_embeddings)
|
||||
|
||||
def forward(self, x, kv_cache=None):
|
||||
# 매 q, k, v + RoPE + cache
|
||||
...
|
||||
```
|
||||
|
||||
### vLLM (production GQA serving)
|
||||
```python
|
||||
from vllm import LLM
|
||||
llm = LLM(model='meta-llama/Llama-3.1-70B-Instruct')
|
||||
# 매 internally uses GQA + paged attention + flash attn
|
||||
```
|
||||
|
||||
### Flash Attention with GQA
|
||||
```python
|
||||
from flash_attn import flash_attn_func
|
||||
# 매 supports GQA natively
|
||||
out = flash_attn_func(q, k, v, causal=True) # 매 q heads != kv heads
|
||||
```
|
||||
|
||||
### Benchmark KV cache savings
|
||||
```python
|
||||
import time
|
||||
def benchmark(model, batch_sizes, seq_lens):
|
||||
for b in batch_sizes:
|
||||
for s in seq_lens:
|
||||
try:
|
||||
inputs = torch.randint(0, 32000, (b, s)).cuda()
|
||||
t0 = time.perf_counter()
|
||||
model.generate(inputs, max_new_tokens=128)
|
||||
print(f'b={b}, s={s}: {time.perf_counter()-t0:.2f}s, peak={torch.cuda.max_memory_allocated()/1e9:.2f}GB')
|
||||
except torch.cuda.OutOfMemoryError:
|
||||
print(f'b={b}, s={s}: OOM')
|
||||
torch.cuda.empty_cache()
|
||||
```
|
||||
|
||||
## 매 결정 기준
|
||||
| 상황 | Approach |
|
||||
|---|---|
|
||||
| Long context | GQA (essential) |
|
||||
| High batch | GQA / MQA |
|
||||
| Quality-critical small | MHA |
|
||||
| Mobile / edge | MQA |
|
||||
| Llama-style | GQA (8 KV) |
|
||||
| Modern default | GQA |
|
||||
|
||||
**기본값**: 매 modern LLM = GQA (4-8 KV heads). 매 quality > 1% loss = MHA. 매 extreme constraint = MQA.
|
||||
|
||||
## 🔗 Graph
|
||||
- 부모: [[Attention Mechanism]] · [[Transformer]]
|
||||
- 변형: [[Multi-Head-Attention]] · [[Multi-Query-Attention]]
|
||||
- 응용: [[Llama]] · [[Flash Attention]] · [[LLM_Optimization_and_Deployment_Strategies|vLLM]]
|
||||
- Adjacent: [[KV-Cache]] · [[PagedAttention]] · [[Foundation-Models]]
|
||||
|
||||
## 🤖 LLM 활용
|
||||
**언제**: 매 모든 modern LLM. 매 long context. 매 high-batch serving.
|
||||
**언제 X**: 매 super-small model (< 1B).
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **MHA in production large LLM**: 매 KV cache OOM.
|
||||
- **Wrong N/G ratio**: 매 quality drop.
|
||||
- **No GQA-aware kernel**: 매 inefficient.
|
||||
- **Convert without retraining check**: 매 quality cliff.
|
||||
|
||||
## 🧪 검증 / 중복
|
||||
- Verified (Ainslie GQA 2023, Llama 2/3 papers, Shazeer MQA 2019).
|
||||
- 신뢰도 A.
|
||||
|
||||
## 🕓 Changelog
|
||||
| 날짜 | 변경 |
|
||||
|---|---|
|
||||
| 2026-05-08 | Phase 1 |
|
||||
| 2026-05-10 | Manual cleanup — MHA/MQA/GQA + 매 KV cache calc / Llama / vLLM code |
|
||||
Reference in New Issue
Block a user