---
id: wiki-2026-0508-keyword-search
title: Keyword Search
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Lexical Search, Sparse Retrieval, BM25 Search]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [search, retrieval, ir, bm25, lexical]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: elasticsearch, opensearch, tantivy, rank-bm25
---

# Keyword Search

## 매 한 줄
> **"매 lexical token-match 의 retrieval"**. Keyword Search 는 query 의 token 과 document 의 token overlap 을 ranking signal 로 사용하는 sparse retrieval. TF-IDF (Salton 1975), BM25 (Robertson 1994) 가 backbone 이고, 2026 RAG era 에서도 매 hybrid (BM25 + dense) 의 indispensable component — out-of-domain robustness 와 exact-match precision 으로.

## 매 핵심

### 매 Scoring
- **TF-IDF**: $w_{t,d} = tf(t,d) \cdot \log(N/df_t)$
- **BM25**: $\sum_t \text{IDF}(t) \cdot \frac{tf(t,d)(k_1+1)}{tf(t,d) + k_1(1-b+b\cdot|d|/\bar{|d|})}$
  - $k_1 \approx 1.2$, $b \approx 0.75$
- **BM25F**: per-field weighting
- **Query Likelihood**: $\prod_t P(t | \theta_d)$ with smoothing

### 매 Pipeline
1. **Tokenization** (Unicode segmentation, locale-aware)
2. **Normalization** (lowercase, NFKC, accent fold)
3. **Stopword removal** (optional)
4. **Stemming / lemmatization** (Porter, Snowball, lemmas)
5. **Inverted index** (term → posting list)
6. **Query parse + scoring**

### 매 응용
1. Document search (Google Search lexical layer).
2. Code search (ripgrep, Sourcegraph zoekt).
3. RAG hybrid (BM25 + embeddings → RRF).
4. E-commerce (exact SKU match dominant signal).

## 💻 패턴

### rank-bm25 quickstart
```python
from rank_bm25 import BM25Okapi
import re

corpus = [doc.lower().split() for doc in docs]
bm25 = BM25Okapi(corpus, k1=1.5, b=0.75)
query = "claude opus context window".split()
scores = bm25.get_scores(query)
top = sorted(range(len(docs)), key=lambda i: -scores[i])[:5]
```

### Elasticsearch / OpenSearch query
```python
from opensearchpy import OpenSearch
client = OpenSearch("http://localhost:9200")

client.index(index="docs", id="1",
             body={"title":"Claude 4.7", "body":"1M context window..."})

resp = client.search(index="docs", body={
    "query": {
        "multi_match": {
            "query": "claude context window",
            "fields": ["title^3", "body"],
            "type": "best_fields"
        }
    }
})
```

### Tantivy (Rust, embeddable)
```python
import tantivy
schema_builder = tantivy.SchemaBuilder()
schema_builder.add_text_field("title", stored=True)
schema_builder.add_text_field("body")
schema = schema_builder.build()
index = tantivy.Index(schema)
writer = index.writer()
writer.add_document(tantivy.Document(title="t", body="some body"))
writer.commit()
searcher = index.searcher()
parser = tantivy.QueryParser.for_index(index, ["title","body"])
hits = searcher.search(parser.parse_query("body"), 10).hits
```

### Hybrid search with RRF
```python
def rrf(rank_lists, k=60):
    scores = {}
    for ranks in rank_lists:
        for r, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + r + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

bm25_top = bm25_search(q)
dense_top = vector_search(q)
fused = rrf([bm25_top, dense_top])
```

### Inverted index from scratch
```python
from collections import defaultdict
import math

class InvertedIndex:
    def __init__(self):
        self.idx = defaultdict(dict)  # term → {doc_id: tf}
        self.lens = {}
        self.N = 0
    def add(self, doc_id, tokens):
        self.N += 1
        self.lens[doc_id] = len(tokens)
        for t in tokens:
            self.idx[t][doc_id] = self.idx[t].get(doc_id, 0) + 1
    def bm25(self, query, k1=1.5, b=0.75):
        avgdl = sum(self.lens.values()) / self.N
        scores = defaultdict(float)
        for t in query:
            if t not in self.idx: continue
            df = len(self.idx[t])
            idf = math.log((self.N - df + 0.5) / (df + 0.5) + 1)
            for doc, tf in self.idx[t].items():
                norm = 1 - b + b * self.lens[doc] / avgdl
                scores[doc] += idf * tf * (k1+1) / (tf + k1 * norm)
        return sorted(scores.items(), key=lambda x: -x[1])
```

## 매 결정 기준
| Need | Engine |
|---|---|
| general-purpose | Elasticsearch / OpenSearch |
| embeddable, fast | Tantivy / Lucene |
| code search | Zoekt / ripgrep |
| in-process small | rank-bm25 / Whoosh |
| hybrid RAG | BM25 + dense + RRF |

**기본값**: BM25 + RRF fusion with dense retriever.

## 🔗 Graph
- 부모: [[Information-Retrieval]]
- 변형: [[BM25]] · [[TF-IDF]]
- 응용: [[Hybrid-Search]] · [[Semantic Search|Semantic-Search]]
- Adjacent: [[Tokenization]]

## 🤖 LLM 활용
**언제**: exact-name retrieval, low-resource lang, long-tail terms, hybrid system.
**언제 X**: pure semantic paraphrase matching — dense embeddings 가 강함.

## ❌ 안티패턴
- **Tokenizer mismatch**: 매 query/index 의 다른 tokenizer 사용 → silent recall loss.
- **Default BM25 params**: domain-specific tuning 없이 default → suboptimal.
- **Stopword over-removal**: "to be or not to be" → empty.
- **Synonym 무시**: query expansion 없이 lexical-only 면 recall low.
- **Dense-only 의 obsession**: BM25 baseline 무시 → out-of-domain 에서 실패.

## 🧪 검증 / 중복
- Verified (Robertson 2009 BM25 paper, Manning IR textbook 2008, Thakur 2021 BEIR benchmark).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — BM25, inverted index, hybrid RRF, ES/Tantivy patterns |
