2nd/10_Wiki/Topics/AI_and_ML/Tokenization & Subword Processing.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-tokenization-subword-processing	Tokenization & Subword Processing	10_Wiki/Topics	verified	self	Tokenization BPE WordPiece SentencePiece Tiktoken	none	A	0.9	applied	nlp tokenization llm bpe		2026-05-10	pending	language framework Python / Rust tokenizers / tiktoken / sentencepiece

Tokenization & Subword Processing

매 한 줄

"매 text → discrete unit ID sequence". Tokenization 매 LLM input/output pipeline 의 entry/exit point. 매 word-level (OOV 폭발) → character-level (long sequence) → subword (BPE, WordPiece, SentencePiece, Unigram) 의 evolution. 매 modern LLM (GPT, Claude, Llama) 모두 BPE-variant + 100k-256k vocab.

매 핵심

매 algorithms

• BPE (Byte-Pair Encoding) — 매 most common pair 의 iteratively merge. 매 GPT-2/3/4, Claude, Llama 모두 byte-level BPE.
• WordPiece — 매 BERT 가 사용. 매 BPE-like but uses log-likelihood for merge.
• SentencePiece (Unigram) — 매 language-agnostic, no pre-tokenization. 매 T5, mBART, multilingual.
• Tiktoken — 매 OpenAI 의 fast Rust BPE impl.

매 byte-level BPE

• 매 raw UTF-8 byte 의 starting alphabet — 매 256 base tokens.
• 매 OOV 매 impossible (모든 byte 의 representable).
• 매 Korean / CJK 매 multi-byte → multi-token (매 3-4 bytes per char).

매 modern vocab sizes (2026)

• GPT-4 / Claude 3.x: ~100k.
• GPT-4o: ~200k (cl100k → o200k).
• Llama 3: 128k.
• Gemini: ~256k.
• 매 larger vocab → fewer tokens per text → cheaper, but larger embedding table.

매 응용

1. LLM input encoding / output decoding.
2. Cost / context-budget estimation (매 token = $).
3. Multilingual fairness (매 vocab choice 매 non-English speakers 의 hit hard).

💻 패턴

tiktoken (OpenAI)

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
ids = enc.encode("Hello, 안녕하세요!")
print(ids, len(ids))             # token IDs + count
print(enc.decode(ids))           # roundtrip

Hugging Face tokenizers (BPE)

from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
trainer = trainers.BpeTrainer(vocab_size=32000, special_tokens=["<pad>", "<eos>"])
tok.train(["corpus.txt"], trainer)
tok.save("tokenizer.json")

SentencePiece (Unigram)

import sentencepiece as spm
spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="sp",
    vocab_size=32000,
    model_type="unigram",
    character_coverage=0.9995,  # 매 multilingual 의 essential
)
sp = spm.SentencePieceProcessor(model_file="sp.model")
print(sp.encode("hello", out_type=str))

Anthropic (Claude) token counting

from anthropic import Anthropic
client = Anthropic()
count = client.messages.count_tokens(
    model="claude-opus-4-7",
    messages=[{"role": "user", "content": text}],
)
print(count.input_tokens)

Special-token handling

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
ids = tok.apply_chat_template(
    [{"role": "user", "content": "hi"}],
    tokenize=True, add_generation_prompt=True,
)
# 매 chat template 의 special token 의 자동 insert

Custom merge inspection

# 매 어떤 token 으로 split 되는지 의 inspect
text = "안녕하세요"
ids = enc.encode(text)
for i in ids:
    print(i, repr(enc.decode([i])))

매 결정 기준

상황	Approach
Use existing LLM	Match its tokenizer (tiktoken/HF)
Train new LLM (English)	Byte-level BPE, 32k-128k vocab
Multilingual model	SentencePiece Unigram, high coverage
Code model	Larger vocab (200k+), code-heavy corpus
Korean / CJK heavy	Larger vocab + ensure char coverage
Domain-specific (medical)	Extend vocab with domain merges

기본값: 매 byte-level BPE 32k-128k for English/code, SentencePiece Unigram for multilingual.

🔗 Graph

• 부모: LLM-Foundations
• 변형: BPE · WordPiece · SentencePiece
• Adjacent: Embeddings · Context-Window

🤖 LLM 활용

언제: 매 token cost estimation, custom tokenizer training, multilingual fairness audit, prompt length debug. 언제 X: 매 trivial LLM API 사용 — 매 implicit, no explicit work needed.

❌ 안티패턴

• Wrong tokenizer mismatch: 매 model 과 tokenizer 매 must match — 매 mix 시 garbage output.
• Korean = 1 token assumption: 매 byte-level BPE 매 한국어 1 char ≈ 2-3 tokens.
• No EOS handling: 매 generation stop token 의 forget — 매 endless output.
• Whitespace prefix issue: 매 hello vs hello 의 different token — 매 leading-space sensitivity.

🧪 검증 / 중복

• Verified (Sennrich et al. 2016 BPE, Kudo SentencePiece 2018, OpenAI tiktoken, HF tokenizers).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — BPE/SP/tiktoken modern coverage