---
id: wiki-2026-0508-bloom-filters-in-search
title: Bloom Filters in Search
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Bloom Filter, BF, Probabilistic Set Membership]
duplicate_of: none
source_trust_level: A
confidence_score: 0.92
verification_status: applied
tags: [data-structures, search, probabilistic, indexing]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: pybloom-live, RedisBloom
---

# Bloom Filters in Search

## 매 한 줄
> **"매 set membership 의 ultra-compact probabilistic test"**. Burton Bloom (1970) 이 spell-checker 위해 designed — modern search 에서 매 inverted-index pruning, cache short-circuit, distributed dedup 의 핵심 primitive 로 사용.

## 매 핵심

### 매 작동 원리
- m-bit array + k independent hash functions.
- Insert: 매 element x → 매 k hashes → 매 set bits at h1(x), …, hk(x).
- Query: 매 모든 k bits set → "possibly in set". 매 하나라도 0 → "definitely not".
- **False positive O, false negative X** — 매 search 의 negative-cache 에 ideal.

### 매 수학
- Optimal k = (m/n) · ln 2.
- FP rate p ≈ (1 − e^(−kn/m))^k.
- 매 1% FP 위해 매 element 당 ~9.6 bits 필요 (load factor independent of element size).

### 매 응용 in Search
1. **Inverted-index shard pruning** — Elasticsearch, Lucene 매 segment-level BF.
2. **CDN cache check** — 매 origin 의 round-trip avoid (Akamai).
3. **Crawler URL dedup** — Googlebot-style frontier.
4. **LSM-tree SSTable pruning** — RocksDB, Cassandra, ScyllaDB.
5. **Vector-DB ANN candidate filter** — Milvus, Weaviate.

## 💻 패턴

### Vanilla Python
```python
import mmh3, math
from bitarray import bitarray

class BloomFilter:
    def __init__(self, n: int, fp: float = 0.01):
        self.m = math.ceil(-(n * math.log(fp)) / (math.log(2) ** 2))
        self.k = max(1, round((self.m / n) * math.log(2)))
        self.bits = bitarray(self.m); self.bits.setall(0)

    def _hashes(self, x: bytes):
        h1, h2 = mmh3.hash64(x, signed=False)
        return [(h1 + i * h2) % self.m for i in range(self.k)]

    def add(self, x: bytes):
        for i in self._hashes(x): self.bits[i] = 1

    def __contains__(self, x: bytes) -> bool:
        return all(self.bits[i] for i in self._hashes(x))
```

### RedisBloom (production)
```python
import redis
r = redis.Redis()
r.execute_command("BF.RESERVE", "urls", 0.001, 10_000_000)
r.execute_command("BF.ADD", "urls", "https://example.com/a")
exists = r.execute_command("BF.EXISTS", "urls", "https://example.com/a")  # 1 or 0
```

### Counting Bloom Filter (supports delete)
```python
class CountingBF:
    def __init__(self, m, k): self.c = [0]*m; self.m, self.k = m, k
    def add(self, x):
        for i in self._h(x): self.c[i] += 1
    def remove(self, x):
        for i in self._h(x):
            if self.c[i] > 0: self.c[i] -= 1
```

### Cuckoo Filter (modern alternative)
```python
# pip install cuckoofilter
from cuckoofilter import CuckooFilter
cf = CuckooFilter(capacity=1_000_000, fingerprint_size=12)
cf.insert(b"item"); cf.contains(b"item"); cf.delete(b"item")
```

### LSM-tree integration (RocksDB-style, conceptual)
```python
def get(key):
    for sstable in reversed(levels):  # newest first
        if key not in sstable.bloom:   # BF rejects → skip disk read
            continue
        v = sstable.disk_lookup(key)
        if v is not None: return v
    return None
```

### Distributed scale-out (partitioned BF)
```python
def shard(key, n_shards): return mmh3.hash(key) % n_shards
# 매 shard 매 own BF — node-local query, gossip-merged for global view.
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Static set, exact 필요 | Perfect hash (CHD, BBHash) |
| Dynamic, FP OK, no delete | **Bloom Filter** |
| Need delete | Counting BF or Cuckoo Filter |
| Need count/freq | Count-Min Sketch |
| Streaming dedup, bounded mem | HyperLogLog (cardinality only) |
| Need sorted range query | B-tree / SSTable index |

**기본값**: 매 dynamic insert + lookup 의 negative-cache 에서는 **Bloom Filter (k = m/n · ln 2, p = 1%)**.

## 🔗 Graph
- 부모: [[Probabilistic-Data-Structures]] · [[Hash-Functions]]
- 변형: [[Counting-Bloom-Filter]] · [[Cuckoo-Filter]] · [[Quotient-Filter]] · [[XOR-Filter]]
- 응용: [[LSM-Tree]] · [[Inverted-Index]] · [[Web-Crawler]] · [[CDN-Cache]]
- Adjacent: [[HyperLogLog]] · [[Count-Min-Sketch]] · [[MinHash]]

## 🤖 LLM 활용
**언제**: 매 candidate-key set 의 fast negative-test, 매 duplicate detection in streaming pipeline, 매 ANN search 의 pre-filter.
**언제 X**: 매 exact membership 필수 (auth, billing), 매 small n (< 1000) 일 때 — 매 hash-set 의 simpler.

## ❌ 안티패턴
- **Underestimate n**: 매 capacity overflow → 매 FP rate 의 explode (saturated bits).
- **Weak hash (e.g., DJB2)**: 매 correlated bits → 매 actual FP > theoretical. **MurmurHash3 / xxHash 사용.**
- **Delete via clearing bits**: 매 corrupts other entries — 매 Counting BF 사용.
- **Sharing BF across security boundary**: 매 timing side-channel 의 set membership leak.

## 🧪 검증 / 중복
- Verified (Bloom 1970, "Space/Time Trade-offs in Hash Coding"; Broder & Mitzenmacher 2002 survey; RocksDB & Cassandra source).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 placeholder |
| 2026-05-10 | Manual cleanup — full content (theory, 6 patterns, decision matrix) |
