---
id: wiki-2026-0508-tokenization-subword-processing
title: "Tokenization & Subword Processing"
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Tokenization, BPE, WordPiece, SentencePiece, Tiktoken]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [nlp, tokenization, llm, bpe]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python / Rust
  framework: tokenizers / tiktoken / sentencepiece
---

# Tokenization & Subword Processing

## 매 한 줄
> **"매 text → discrete unit ID sequence"**. Tokenization 매 LLM input/output pipeline 의 entry/exit point. 매 word-level (OOV 폭발) → character-level (long sequence) → subword (BPE, WordPiece, SentencePiece, Unigram) 의 evolution. 매 modern LLM (GPT, Claude, Llama) 모두 BPE-variant + 100k-256k vocab.

## 매 핵심

### 매 algorithms
- **BPE (Byte-Pair Encoding)** — 매 most common pair 의 iteratively merge. 매 GPT-2/3/4, Claude, Llama 모두 byte-level BPE.
- **WordPiece** — 매 BERT 가 사용. 매 BPE-like but uses log-likelihood for merge.
- **SentencePiece (Unigram)** — 매 language-agnostic, no pre-tokenization. 매 T5, mBART, multilingual.
- **Tiktoken** — 매 OpenAI 의 fast Rust BPE impl.

### 매 byte-level BPE
- 매 raw UTF-8 byte 의 starting alphabet — 매 256 base tokens.
- 매 OOV 매 impossible (모든 byte 의 representable).
- 매 Korean / CJK 매 multi-byte → multi-token (매 3-4 bytes per char).

### 매 modern vocab sizes (2026)
- GPT-4 / Claude 3.x: ~100k.
- GPT-4o: ~200k (cl100k → o200k).
- Llama 3: 128k.
- Gemini: ~256k.
- 매 larger vocab → fewer tokens per text → cheaper, but larger embedding table.

### 매 응용
1. LLM input encoding / output decoding.
2. Cost / context-budget estimation (매 token = $).
3. Multilingual fairness (매 vocab choice 매 non-English speakers 의 hit hard).

## 💻 패턴

### tiktoken (OpenAI)
```python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
ids = enc.encode("Hello, 안녕하세요!")
print(ids, len(ids))             # token IDs + count
print(enc.decode(ids))           # roundtrip
```

### Hugging Face tokenizers (BPE)
```python
from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["<pad>", "<eos>"])
tok.train(["corpus.txt"], trainer)
tok.save("tokenizer.json")
```

### SentencePiece (Unigram)
```python
import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="sp",
    vocab_size=32000,
    model_type="unigram",
    character_coverage=0.9995,  # 매 multilingual 의 essential
)
sp = spm.SentencePieceProcessor(model_file="sp.model")
print(sp.encode("hello", out_type=str))
```

### Anthropic (Claude) token counting
```python
from anthropic import Anthropic
client = Anthropic()
count = client.messages.count_tokens(
    model="claude-opus-4-7",
    messages=[{"role": "user", "content": text}],
)
print(count.input_tokens)
```

### Special-token handling
```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
ids = tok.apply_chat_template(
    [{"role": "user", "content": "hi"}],
    tokenize=True, add_generation_prompt=True,
)
# 매 chat template 의 special token 의 자동 insert
```

### Custom merge inspection
```python
# 매 어떤 token 으로 split 되는지 의 inspect
text = "안녕하세요"
ids = enc.encode(text)
for i in ids:
    print(i, repr(enc.decode([i])))
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Use existing LLM | Match its tokenizer (tiktoken/HF) |
| Train new LLM (English) | Byte-level BPE, 32k-128k vocab |
| Multilingual model | SentencePiece Unigram, high coverage |
| Code model | Larger vocab (200k+), code-heavy corpus |
| Korean / CJK heavy | Larger vocab + ensure char coverage |
| Domain-specific (medical) | Extend vocab with domain merges |

**기본값**: 매 byte-level BPE 32k-128k for English/code, SentencePiece Unigram for multilingual.

## 🔗 Graph
- 부모: [[LLM-Foundations]]
- 변형: [[BPE]] · [[WordPiece]] · [[SentencePiece]]
- Adjacent: [[Embeddings]] · [[Context-Window]]

## 🤖 LLM 활용
**언제**: 매 token cost estimation, custom tokenizer training, multilingual fairness audit, prompt length debug.
**언제 X**: 매 trivial LLM API 사용 — 매 implicit, no explicit work needed.

## ❌ 안티패턴
- **Wrong tokenizer mismatch**: 매 model 과 tokenizer 매 must match — 매 mix 시 garbage output.
- **Korean = 1 token assumption**: 매 byte-level BPE 매 한국어 1 char ≈ 2-3 tokens.
- **No EOS handling**: 매 generation stop token 의 forget — 매 endless output.
- **Whitespace prefix issue**: 매 ` hello` vs `hello` 의 different token — 매 leading-space sensitivity.

## 🧪 검증 / 중복
- Verified (Sennrich et al. 2016 BPE, Kudo SentencePiece 2018, OpenAI tiktoken, HF tokenizers).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — BPE/SP/tiktoken modern coverage |