2nd/10_Wiki/Topics/Computer_Science_and_Theory/Sampling-Techniques.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-sampling-techniques	Sampling Techniques	10_Wiki/Topics	verified	self	Statistical Sampling Survey Sampling MCMC Sampling	none	A	0.9	applied	statistics sampling monte-carlo mcmc survey		2026-05-10	pending	language framework python numpy / numpyro / scikit-learn

Sampling Techniques

매 한 줄

"매 sampling 은 population 의 representative slice 를 얻는 art". Neyman (1934) 의 stratified sampling formalization 부터 매 modern MCMC (NUTS, HMC, SMC) 까지 — 매 statistics, ML 의 LLM-RLHF preference dataset 까지 backbone. 2026 에 Anthropic, OpenAI 의 RLAIF pipeline 도 매 stratified human-preference sampling 의 위에 동작.

매 핵심

매 Probability sampling family

• Simple Random: 매 uniform pick — baseline 이지만 small subgroup 누락 위험.
• Stratified: population 을 stratum 으로 partition, 매 stratum 당 SRS — variance reduction.
• Cluster: cluster 단위 pick (e.g. zipcode) — 매 cost↓ but design-effect↑.
• Systematic: every k-th — 매 list 가 random 일 때 유효, periodicity 시 bias.
• Multistage: cluster → stratified → SRS — 매 national survey 의 표준.

매 Non-probability (use with care)

• Convenience: 매 quickest, biased.
• Snowball: 매 hidden population (e.g. underground community).
• Quota: 매 demographic match 강제 — Online panel 에서 흔함.

매 Monte Carlo / MCMC

• Importance: q(x) 에서 sample 후 w=p/q reweight.
• Rejection: accept iff u·M·q(x) ≤ p(x).
• MH/HMC/NUTS: 매 high-dim posterior — Stan, NumPyro, BlackJAX 의 default.
• SMC: 매 sequential Bayes (particle filter generalization).

매 응용

1. RLHF preference data — 매 stratified by task family.
2. A/B test bucketing — 매 hash-mod stratify on user_id.
3. Image diffusion — 매 latent prior sampling (DDIM, EDM2).
4. NN training mini-batch — 매 weighted sampler for class imbalance.

💻 패턴

Stratified split (sklearn)

from sklearn.model_selection import StratifiedShuffleSplit

sss = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_idx, test_idx in sss.split(X, y):
    X_tr, X_te = X[train_idx], X[test_idx]
    y_tr, y_te = y[train_idx], y[test_idx]

Reservoir sampling (streaming, k items)

import random

def reservoir(stream, k):
    res = []
    for i, item in enumerate(stream):
        if i < k:
            res.append(item)
        else:
            j = random.randint(0, i)
            if j < k:
                res[j] = item
    return res

Weighted sampling (alias method, O(1) per draw)

import numpy as np

class AliasSampler:
    def __init__(self, weights):
        n = len(weights)
        p = np.asarray(weights, dtype=np.float64)
        p = p * n / p.sum()
        self.prob = np.zeros(n); self.alias = np.zeros(n, dtype=np.int64)
        small = [i for i in range(n) if p[i] < 1.0]
        large = [i for i in range(n) if p[i] >= 1.0]
        while small and large:
            s, l = small.pop(), large.pop()
            self.prob[s] = p[s]; self.alias[s] = l
            p[l] -= 1.0 - p[s]
            (small if p[l] < 1.0 else large).append(l)
        for i in large + small:
            self.prob[i] = 1.0
        self.n = n
    def draw(self, k=1):
        i = np.random.randint(0, self.n, size=k)
        u = np.random.random(k)
        return np.where(u < self.prob[i], i, self.alias[i])

NUTS posterior sampling (NumPyro 0.16)

import numpyro, numpyro.distributions as dist
from numpyro.infer import MCMC, NUTS

def model(x, y):
    a = numpyro.sample("a", dist.Normal(0, 1))
    b = numpyro.sample("b", dist.Normal(0, 1))
    sigma = numpyro.sample("sigma", dist.HalfNormal(1.0))
    numpyro.sample("obs", dist.Normal(a + b * x, sigma), obs=y)

mcmc = MCMC(NUTS(model), num_warmup=1000, num_samples=2000)
mcmc.run(jax.random.PRNGKey(0), x, y)

Importance sampling (with effective sample size)

def importance(samples, log_p, log_q):
    log_w = log_p - log_q
    log_w -= log_w.max()
    w = np.exp(log_w); w /= w.sum()
    ess = 1.0 / np.sum(w ** 2)
    return w, ess

Hash-mod A/B bucketing (deterministic)

import hashlib

def bucket(user_id, salt, n_buckets):
    h = hashlib.blake2b(f"{salt}:{user_id}".encode(), digest_size=8).digest()
    return int.from_bytes(h, "big") % n_buckets

Class-balanced PyTorch sampler

from torch.utils.data import WeightedRandomSampler
import numpy as np

def balanced_sampler(labels):
    counts = np.bincount(labels)
    weights = 1.0 / counts[labels]
    return WeightedRandomSampler(weights, len(labels), replacement=True)

매 결정 기준

상황	Approach
Population homogeneous	SRS
Subgroup variance 다름	Stratified
Geographic cost 제약	Cluster / Multistage
Streaming / unknown size	Reservoir
Heavy-tail Bayes posterior	NUTS / HMC
Sequential filtering	SMC / Particle Filter
Class imbalance training	Weighted / Balanced sampler

기본값: training 은 stratified split, eval 은 i.i.d. test set, Bayes 는 NUTS, RLHF 는 task-stratified preference sampling.

🔗 Graph

• 부모: Statistics · Probability Theory
• 변형: Monte-Carlo-Integration · Particle-Filter-Algorithms
• 응용: 데이터 사이언스 및 ML 엔지니어링 · Information Retrieval (IR)
• Adjacent: Posterior-and-Prior-Probability · Statistical-Power

🤖 LLM 활용

언제: dataset construction, eval design, RLHF/RLAIF preference collection, posterior inference, Monte-Carlo estimation 의 framing 때. 언제 X: full-population accessible (census), deterministic computation 으로 충분한 case.

❌ 안티패턴

• Convenience sampling 으로 production decision: 매 selection bias 직격탄.
• stratified 변수 leakage: stratify-by-target 후 train/test split 시 target 정보 누설.
• MCMC 의 single-chain 만 신뢰: 매 multi-chain + R-hat / ESS 검증 필수.
• Reservoir sampling 의 reproducibility 무시: seed pin 안 하면 매 재현 불가.

🧪 검증 / 검토

• Verified (Lohr 2021 "Sampling: Design and Analysis"; Gelman BDA3 ch.10–12; NumPyro docs 0.16).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — probability/non-probability/MCMC families, alias-method/NUTS/reservoir patterns, RLHF context