Wiki cleanup: error-doc removal, dedup merge, link normalization
10_Wiki/Topics 대규모 정리: - 오류 캡처/미완성 stub 문서 227개 제거 - 교차폴더 중복 43클러스터 병합 (63파일 → redirect) - 링크명 정규화: 깨진 링크 수정·redirect 직결·개념 매핑 ~2,400건 - 카테고리 MOC 6개 신규 생성 - Graph 섹션 미해결 related-keyword 링크 10,058건 제거 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Antigravity Agent
@@ -2,135 +2,26 @@
|
||||
id: wiki-2026-0508-locality-sensitive-hashing
|
||||
title: Locality Sensitive Hashing
|
||||
category: 10_Wiki/Topics
|
||||
status: verified
|
||||
canonical_id: self
|
||||
aliases: [LSH, Locality-Sensitive-Hashing, MinHash, SimHash]
|
||||
duplicate_of: none
|
||||
status: duplicate
|
||||
canonical_id: wiki-2026-0508-locality-sensitive-hashing-lsh
|
||||
duplicate_of: "[[Locality-Sensitive Hashing (LSH)]]"
|
||||
aliases: []
|
||||
source_trust_level: A
|
||||
confidence_score: 0.93
|
||||
verification_status: applied
|
||||
tags: [hashing, ann, similarity-search, embeddings, retrieval]
|
||||
raw_sources: []
|
||||
last_reinforced: 2026-05-10
|
||||
confidence_score: 0.9
|
||||
verification_status: redirected
|
||||
tags: [duplicate]
|
||||
last_reinforced: 2026-05-20
|
||||
github_commit: pending
|
||||
tech_stack:
|
||||
language: python
|
||||
framework: datasketch-faiss
|
||||
---
|
||||
|
||||
# Locality Sensitive Hashing
|
||||
|
||||
## 매 한 줄
|
||||
> **"매 가까운 점은 매 같은 bucket에 떨어진다"**. LSH는 매 metric similarity를 hash collision probability로 변환하여 매 sub-linear ANN(approximate nearest neighbor) search를 가능케 한다. 2026 vector DB 시대에 IVF-PQ·HNSW에 밀렸지만, 매 streaming dedup·document near-dup detection에서 매 dominant.
|
||||
|
||||
## 매 핵심
|
||||
|
||||
### 매 family들
|
||||
- **MinHash (Jaccard)**: 매 set similarity — shingled documents.
|
||||
- **SimHash (cosine)**: 매 random hyperplane projection.
|
||||
- **p-stable LSH (L2)**: 매 random Gaussian projection + bucketing.
|
||||
- **Cross-polytope LSH**: 매 angular distance, 매 unit sphere 위.
|
||||
|
||||
### 매 amplification
|
||||
- AND-construction: 매 k hashes 모두 일치 → false positive ↓.
|
||||
- OR-construction: 매 L tables 중 하나라도 collision → recall ↑.
|
||||
- (k, L) tuning이 매 핵심.
|
||||
|
||||
### 매 응용
|
||||
1. Plagiarism / near-duplicate detection (web, code).
|
||||
2. Streaming deduplication (logs, training data).
|
||||
3. Genomic sequence matching.
|
||||
4. Pre-filter for vector DBs at billion scale.
|
||||
|
||||
## 💻 패턴
|
||||
|
||||
### MinHash with datasketch
|
||||
```python
|
||||
from datasketch import MinHash, MinHashLSH
|
||||
def shingles(text, k=3):
|
||||
return {text[i:i+k] for i in range(len(text)-k+1)}
|
||||
m = MinHash(num_perm=128)
|
||||
for s in shingles(doc):
|
||||
m.update(s.encode())
|
||||
lsh = MinHashLSH(threshold=0.7, num_perm=128)
|
||||
lsh.insert("doc1", m)
|
||||
matches = lsh.query(m_query)
|
||||
```
|
||||
|
||||
### SimHash (64-bit)
|
||||
```python
|
||||
import numpy as np
|
||||
def simhash(features, bits=64):
|
||||
v = np.zeros(bits)
|
||||
for f, w in features:
|
||||
h = hash(f)
|
||||
for i in range(bits):
|
||||
v[i] += w if (h >> i) & 1 else -w
|
||||
return sum(1 << i for i in range(bits) if v[i] > 0)
|
||||
```
|
||||
|
||||
### Random projection LSH (cosine)
|
||||
```python
|
||||
class CosineLSH:
|
||||
def __init__(self, dim, n_planes=16):
|
||||
self.planes = np.random.randn(n_planes, dim)
|
||||
def hash(self, x):
|
||||
return tuple((self.planes @ x > 0).astype(int))
|
||||
```
|
||||
|
||||
### p-stable LSH (Euclidean)
|
||||
```python
|
||||
def l2_hash(x, a, b, w):
|
||||
return int((a @ x + b) // w)
|
||||
# a ~ N(0, I), b ~ U(0, w)
|
||||
```
|
||||
|
||||
### Banding for MinHash
|
||||
```python
|
||||
def bands(sig, b, r):
|
||||
return [tuple(sig[i*r:(i+1)*r]) for i in range(b)]
|
||||
# threshold ~ (1/b)^(1/r)
|
||||
```
|
||||
|
||||
### FAISS LSH index (for L2)
|
||||
```python
|
||||
import faiss
|
||||
index = faiss.IndexLSH(dim, n_bits=256)
|
||||
index.add(X)
|
||||
D, I = index.search(query, k=10)
|
||||
```
|
||||
|
||||
## 매 결정 기준
|
||||
| Metric | LSH family |
|
||||
|---|---|
|
||||
| Jaccard | MinHash |
|
||||
| Cosine | SimHash / random hyperplane |
|
||||
| Euclidean | p-stable / IndexLSH |
|
||||
| Hamming | Bit sampling |
|
||||
|
||||
**기본값**: HNSW 우선, billion-scale dedup만 LSH.
|
||||
> **이 문서는 [[Locality-Sensitive Hashing (LSH)]] 의 중복본입니다.** Canonical 문서로 redirect.
|
||||
|
||||
## 🔗 Graph
|
||||
- 부모: [[Hash-Functions-and-Maps]] · [[Approximate-Nearest-Neighbor]]
|
||||
- 변형: [[MinHash]] · [[SimHash]] · [[Cross-Polytope-LSH]]
|
||||
- 응용: [[Near-Duplicate-Detection]] · [[Vector-Database]] · [[Streaming-Dedup]]
|
||||
- Adjacent: [[HNSW]] · [[Product-Quantization]] · [[Bloom-Filters in Search]]
|
||||
- 부모: [[Locality-Sensitive Hashing (LSH)]] (canonical)
|
||||
|
||||
## 🤖 LLM 활용
|
||||
**언제**: Billion-scale dedup, Jaccard-based near-dup detection, streaming pipeline.
|
||||
**언제 X**: High-recall ANN with dense embeddings (use HNSW/IVF-PQ).
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **단일 hash table**: recall 낮음 — 매 L tables 필수.
|
||||
- **k 너무 큼**: false neg 폭증.
|
||||
- **Dense embedding에 MinHash**: 매 잘못된 family 선택.
|
||||
|
||||
## 🧪 검증 / 중복
|
||||
- Verified (Indyk & Motwani 1998; Mining of Massive Datasets ch.3).
|
||||
- 신뢰도 A.
|
||||
|
||||
## 🕓 Changelog
|
||||
## 🕓 변경 이력
|
||||
| 날짜 | 변경 |
|
||||
|---|---|
|
||||
| 2026-05-08 | Phase 1 |
|
||||
| 2026-05-10 | Manual cleanup — LSH families + datasketch/FAISS examples |
|
||||
| 2026-05-20 | 중복 병합 — canonical 문서로 redirect |
|
||||
|
||||
Reference in New Issue
Block a user