Files

T

Antigravity Agent 504fd5fb42 [G1-Sync] Manual knowledge update

2026-05-10 22:08:15 +09:00

4.0 KiB

Raw Blame History

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

title

Locality Sensitive Hashing

매 한 줄

"매 가까운 점은 매 같은 bucket에 떨어진다". LSH는 매 metric similarity를 hash collision probability로 변환하여 매 sub-linear ANN(approximate nearest neighbor) search를 가능케 한다. 2026 vector DB 시대에 IVF-PQ·HNSW에 밀렸지만, 매 streaming dedup·document near-dup detection에서 매 dominant.

매 핵심

매 family들

MinHash (Jaccard): 매 set similarity — shingled documents.
SimHash (cosine): 매 random hyperplane projection.
p-stable LSH (L2): 매 random Gaussian projection + bucketing.
Cross-polytope LSH: 매 angular distance, 매 unit sphere 위.

매 amplification

AND-construction: 매 k hashes 모두 일치 → false positive ↓.
OR-construction: 매 L tables 중 하나라도 collision → recall ↑.
(k, L) tuning이 매 핵심.

매 응용

Plagiarism / near-duplicate detection (web, code).
Streaming deduplication (logs, training data).
Genomic sequence matching.
Pre-filter for vector DBs at billion scale.

💻 패턴

MinHash with datasketch

from datasketch import MinHash, MinHashLSH
def shingles(text, k=3):
    return {text[i:i+k] for i in range(len(text)-k+1)}
m = MinHash(num_perm=128)
for s in shingles(doc):
    m.update(s.encode())
lsh = MinHashLSH(threshold=0.7, num_perm=128)
lsh.insert("doc1", m)
matches = lsh.query(m_query)

SimHash (64-bit)

import numpy as np
def simhash(features, bits=64):
    v = np.zeros(bits)
    for f, w in features:
        h = hash(f)
        for i in range(bits):
            v[i] += w if (h >> i) & 1 else -w
    return sum(1 << i for i in range(bits) if v[i] > 0)

Random projection LSH (cosine)

class CosineLSH:
    def __init__(self, dim, n_planes=16):
        self.planes = np.random.randn(n_planes, dim)
    def hash(self, x):
        return tuple((self.planes @ x > 0).astype(int))

p-stable LSH (Euclidean)

def l2_hash(x, a, b, w):
    return int((a @ x + b) // w)
# a ~ N(0, I), b ~ U(0, w)

Banding for MinHash

def bands(sig, b, r):
    return [tuple(sig[i*r:(i+1)*r]) for i in range(b)]
# threshold ~ (1/b)^(1/r)

FAISS LSH index (for L2)

import faiss
index = faiss.IndexLSH(dim, n_bits=256)
index.add(X)
D, I = index.search(query, k=10)

매 결정 기준

Metric	LSH family
Jaccard	MinHash
Cosine	SimHash / random hyperplane
Euclidean	p-stable / IndexLSH
Hamming	Bit sampling

기본값: HNSW 우선, billion-scale dedup만 LSH.

🔗 Graph

부모: Hash-Functions-and-Maps · Approximate-Nearest-Neighbor
변형: MinHash · SimHash · Cross-Polytope-LSH
응용: Near-Duplicate-Detection · Vector-Database · Streaming-Dedup
Adjacent: HNSW · Product-Quantization · Bloom-Filters in Search

🤖 LLM 활용

언제: Billion-scale dedup, Jaccard-based near-dup detection, streaming pipeline. 언제 X: High-recall ANN with dense embeddings (use HNSW/IVF-PQ).

❌ 안티패턴

단일 hash table: recall 낮음 — 매 L tables 필수.
k 너무 큼: false neg 폭증.
Dense embedding에 MinHash: 매 잘못된 family 선택.

🧪 검증 / 중복

Verified (Indyk & Motwani 1998; Mining of Massive Datasets ch.3).
신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — LSH families + datasketch/FAISS examples

4.0 KiB Raw Blame History