2nd/10_Wiki/Topics/AI_and_ML/Vocabulary-Expansion.md

---
id: wiki-2026-0508-vocabulary-expansion
title: Vocabulary Expansion
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Vocab Expansion, Tokenizer Extension, Domain Vocabulary]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [nlp, tokenizer, vocabulary, llm, fine-tuning]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: transformers, sentencepiece, tokenizers
---

# Vocabulary Expansion

## 매 한 줄
> **"매 base tokenizer 에 domain token 을 grafting 하는 surgery"**. 매 BPE / SentencePiece tokenizer 의 vocab 을 확장 — 매 새 token embedding 의 initialize, 매 LM head 의 resize, 매 continued pretraining 의 alignment. 매 2026 의 Llama 3.x / Qwen 3 / Gemma 3 의 multilingual extension 의 standard recipe.

## 매 핵심

### 매 왜 expand
- **Tokenization efficiency**: 매 Korean / Japanese / code 의 base tokenizer 의 over-fragmentation — "안녕하세요" 의 8 token 의 1 token 의 reduction.
- **Domain coverage**: 매 medical / legal / chemistry term 의 single-token representation.
- **Inference cost**: 매 sequence length 의 reduction 의 latency / cost 의 직접적 saving.
- **Quality**: 매 long-tail token 의 gradient signal 의 improvement.

### 매 expansion 방식
1. **Pure addition**: 매 base vocab 의 그대로 + 매 new token 의 append. Embedding matrix 의 row append.
2. **Merge new tokenizer**: 매 domain corpus 의 새 BPE 의 train → 매 base 와 union → 매 conflict resolution.
3. **Token replacement**: 매 unused token (e.g., `<unused42>`) 의 reuse — 매 vocab size 의 unchanged.

### 매 embedding init 전략
- **Mean init**: 매 새 token 의 sub-word embedding 의 mean.
- **Random + small std**: 매 N(0, 0.02) — 매 risky.
- **FOCUS / WECHSEL**: 매 source language embedding 의 nearest-neighbor mapping.
- **OFA (One For All)**: 매 multilingual transfer 의 SOTA (2024).

### 매 응용
1. 매 English-only LLM 의 Korean / Japanese / Arabic extension.
2. 매 code LLM 의 새 language (Mojo, Zig) 의 token addition.
3. 매 biomedical LLM (PubMedBERT) 의 specialized term integration.
4. 매 retrieval-augmented model 의 special control token (`<doc>`, `<query>`) 추가.

## 💻 패턴

### Tokenizer 확장 (HuggingFace)
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-8B")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.3-8B")

# Add domain tokens
new_tokens = ["[[CHEMICAL]]", "[[GENE]]", "ACE2", "SARS-CoV-2"]
num_added = tokenizer.add_tokens(new_tokens)
print(f"Added {num_added} tokens, new vocab size: {len(tokenizer)}")

# Resize embedding + LM head
model.resize_token_embeddings(len(tokenizer))
```

### Mean-init for new token embeddings
```python
def init_new_embeddings_by_subword_mean(model, tokenizer, new_tokens, base_tokenizer):
    embed = model.get_input_embeddings().weight.data
    with torch.no_grad():
        for tok in new_tokens:
            tok_id = tokenizer.convert_tokens_to_ids(tok)
            # Tokenize the surface form with the BASE tokenizer
            sub_ids = base_tokenizer(tok, add_special_tokens=False).input_ids
            if len(sub_ids) == 0:
                continue
            embed[tok_id] = embed[sub_ids].mean(dim=0)
    return model
```

### SentencePiece merge (Llama-style)
```python
import sentencepiece as spm
from sentencepiece import sentencepiece_model_pb2 as sp_pb2

base = sp_pb2.ModelProto()
base.ParseFromString(open("base.model", "rb").read())
domain = sp_pb2.ModelProto()
domain.ParseFromString(open("domain_korean.model", "rb").read())

base_tokens = {p.piece for p in base.pieces}
added = 0
for piece in domain.pieces:
    if piece.piece not in base_tokens:
        new = sp_pb2.ModelProto().SentencePiece()
        new.piece = piece.piece
        new.score = 0.0
        base.pieces.append(new)
        added += 1

with open("merged.model", "wb") as f:
    f.write(base.SerializeToString())
print(f"Merged: +{added} tokens")
```

### FOCUS-style cross-lingual init
```python
# For each new token: find k-NN among OLD tokens via auxiliary embedding (e.g., fastText)
# Initialize new embedding as weighted sum of those neighbors' LLM embeddings.
def focus_init(new_tokens, aux_embs, llm_embed, old_vocab, k=10):
    init = {}
    for tok in new_tokens:
        if tok not in aux_embs:
            continue
        sims = {o: cos(aux_embs[tok], aux_embs[o]) for o in old_vocab if o in aux_embs}
        top = sorted(sims.items(), key=lambda x: -x[1])[:k]
        weights = torch.softmax(torch.tensor([s for _, s in top]) / 0.1, dim=0)
        ids = [old_vocab[o] for o, _ in top]
        init[tok] = (weights.unsqueeze(1) * llm_embed[ids]).sum(0)
    return init
```

### Tied weights handling (LM head ↔ input embedding)
```python
if model.config.tie_word_embeddings:
    # resize_token_embeddings handles both — verify
    assert model.get_input_embeddings().weight.data_ptr() == \
           model.get_output_embeddings().weight.data_ptr()
else:
    # Independently init the LM head rows for new tokens
    lm_head = model.get_output_embeddings().weight.data
    input_emb = model.get_input_embeddings().weight.data
    with torch.no_grad():
        for tok_id in new_token_ids:
            lm_head[tok_id] = input_emb[tok_id].clone()
```

### Continued pretraining 의 lr schedule
```python
from transformers import get_cosine_schedule_with_warmup

# Freeze old embeddings 의 gradient mask 의 trick
embed = model.get_input_embeddings()
new_token_mask = torch.zeros(len(tokenizer), dtype=torch.bool)
new_token_mask[old_vocab_size:] = True

def mask_grad_hook(grad):
    grad[~new_token_mask] = 0  # only update new tokens initially
    return grad

embed.weight.register_hook(mask_grad_hook)
# ... train for N steps, then remove hook for full fine-tune ...
```

### Vocab unused-slot reuse
```python
# Llama / Mistral 의 reserved <unusedN> token 의 in-place rename
# Vocab size 의 unchanged → 매 inference cost 의 zero-impact upgrade
spm_model = sp_pb2.ModelProto()
spm_model.ParseFromString(open("tokenizer.model", "rb").read())
for i, piece in enumerate(spm_model.pieces):
    if piece.piece.startswith("<reserved_") and i < 256:
        piece.piece = NEW_TOKENS.pop()
        if not NEW_TOKENS:
            break
```

### Validation: tokenization rate
```python
def tokens_per_char(tokenizer, corpus):
    total_tokens = total_chars = 0
    for doc in corpus:
        total_tokens += len(tokenizer(doc).input_ids)
        total_chars += len(doc)
    return total_tokens / total_chars

before = tokens_per_char(base_tok, korean_corpus)   # e.g., 0.8
after  = tokens_per_char(merged_tok, korean_corpus) # e.g., 0.4 — 2x compression
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| 매 small domain (<200 token) | Pure addition + mean init |
| 매 new language (10K+ token) | Tokenizer merge + FOCUS / OFA init |
| 매 inference cost 의 critical | Reserved-slot reuse |
| 매 multilingual extension | OFA / WECHSEL + continued pretraining |
| 매 control token 의 추가 | Pure addition + random small init + SFT |

**기본값**: 매 small additions 의 mean-init + 매 brief continued pretraining (1-5B token).

## 🔗 Graph
- 부모: [[Tokenization]]
- 응용: [[Domain Adaptation]]
- Adjacent: [[BPE]] · [[SentencePiece]] · [[LoRA Fine-tuning]]

## 🤖 LLM 활용
**언제**: 매 base tokenizer 의 target language / domain 의 over-fragmentation 의 measurable. 매 corpus 의 1B+ token 의 continued pretraining budget 의 available.
**언제 X**: 매 small fine-tuning task 의 LoRA 의 sufficient. 매 domain coverage 의 already adequate (tokens_per_char < 0.5). 매 vocab change 의 deployment / serving infra 의 redeploy 의 forced 일 때.

## ❌ 안티패턴
- **Random init without continued PT**: 매 새 token embedding 의 noise 의 catastrophic forgetting 의 trigger.
- **LM head 의 forget**: 매 tied=False 의 model 의 input embedding 만 update — 매 generation broken.
- **Tokenizer merge 의 BOS / EOS 충돌**: 매 special token ID 의 silently shifted — 매 inference 의 corrupt.
- **Vocab size 의 padding 의 무시**: 매 GPU 의 vocab size % 64 == 0 의 efficiency 의 lost.
- **Continued PT skipping**: 매 freshly initialized embedding 의 deployed → 매 hallucination spike.

## 🧪 검증 / 중복
- Verified (HuggingFace transformers docs, FOCUS paper Dobler & de Melo 2023, OFA Liu et al. 2024, Llama 3 tokenizer release notes).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — full content (NLP vocabulary expansion patterns / init strategies) |