2nd/10_Wiki/Topics/AI_and_ML/Experience-Replay.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-experience-replay	Experience Replay	10_Wiki/Topics	verified	self	experience replay replay buffer prioritized replay hindsight HER	none	A	0.97	applied	reinforcement-learning dqn replay-buffer prioritized her off-policy		2026-05-10	pending	language framework Python PyTorch / Stable-Baselines3 / Tianshou

Experience Replay

매 한 줄

"매 RL transition 의 buffer 의 store + sample → 매 i.i.d.-like train". Lin 1992, Mnih DQN 2013. 매 off-policy 의 backbone. 매 modern: 매 Prioritized (PER), Hindsight (HER), distributed (Ape-X), n-step.

매 핵심

매 motivation

• Sequential correlation: 매 trajectory 의 i.i.d. X.
• Sample efficiency: 매 reuse.
• Stability: 매 catastrophic forgetting 의 reduce.

매 variant

• Uniform: 매 random sample.
• Prioritized (PER): 매 TD error.
• HER: 매 sparse reward 의 goal relabel.
• n-step: 매 multi-step return.
• Reservoir (Pham): 매 lifelong.
• Distributed (Ape-X, R2D2): 매 actor 의 parallel.

매 응용

• DQN / DDQN: 매 standard.
• DDPG / TD3 / SAC: 매 continuous.
• HER: 매 robotics goal.
• Recurrent: 매 sequence.

💻 패턴

Uniform replay buffer

import numpy as np
from collections import deque

class ReplayBuffer:
    def __init__(self, capacity=100_000):
        self.buf = deque(maxlen=capacity)
    
    def push(self, s, a, r, s_next, done):
        self.buf.append((s, a, r, s_next, done))
    
    def sample(self, batch_size):
        idx = np.random.choice(len(self.buf), batch_size, replace=False)
        batch = [self.buf[i] for i in idx]
        return [np.array(x) for x in zip(*batch)]
    
    def __len__(self): return len(self.buf)

Prioritized Experience Replay (PER, Schaul 2016)

class PER:
    def __init__(self, capacity, alpha=0.6, beta=0.4):
        self.tree = SumTree(capacity)
        self.alpha = alpha
        self.beta = beta
        self.eps = 1e-6
        self.max_p = 1.0
    
    def push(self, transition):
        self.tree.add(self.max_p, transition)
    
    def sample(self, batch_size):
        batch, idxs, priorities = [], [], []
        seg = self.tree.total() / batch_size
        for i in range(batch_size):
            s = np.random.uniform(seg * i, seg * (i + 1))
            idx, p, data = self.tree.get(s)
            batch.append(data); idxs.append(idx); priorities.append(p)
        
        sampling_p = np.array(priorities) / self.tree.total()
        weights = (len(self.tree) * sampling_p) ** -self.beta
        weights /= weights.max()
        return batch, idxs, weights
    
    def update_priorities(self, idxs, td_errors):
        for idx, err in zip(idxs, td_errors):
            p = (abs(err) + self.eps) ** self.alpha
            self.tree.update(idx, p)
            self.max_p = max(self.max_p, p)

SumTree (PER 의 efficient sample)

class SumTree:
    def __init__(self, cap):
        self.cap = cap
        self.tree = np.zeros(2 * cap - 1)
        self.data = np.zeros(cap, dtype=object)
        self.ptr = 0
    
    def add(self, p, data):
        idx = self.ptr + self.cap - 1
        self.data[self.ptr] = data
        self.update(idx, p)
        self.ptr = (self.ptr + 1) % self.cap
    
    def update(self, idx, p):
        change = p - self.tree[idx]
        self.tree[idx] = p
        while idx != 0:
            idx = (idx - 1) // 2
            self.tree[idx] += change
    
    def get(self, s):
        idx = 0
        while idx < self.cap - 1:
            l, r = 2 * idx + 1, 2 * idx + 2
            if s <= self.tree[l]: idx = l
            else: s -= self.tree[l]; idx = r
        data_idx = idx - self.cap + 1
        return idx, self.tree[idx], self.data[data_idx]
    
    def total(self): return self.tree[0]

Hindsight Experience Replay (HER)

def her_relabel(trajectory, k=4, strategy='future'):
    """매 sparse-reward goal-conditioned RL."""
    augmented = []
    for t, (s, a, r, s_next, done) in enumerate(trajectory):
        augmented.append((s, a, r, s_next, done))
        # 매 future strategy: 매 future state 의 새 goal
        for _ in range(k):
            future_t = np.random.randint(t, len(trajectory))
            new_goal = trajectory[future_t][3]  # 매 future s_next
            new_r = compute_reward(s_next, new_goal)
            new_s = augment_with_goal(s, new_goal)
            new_s_next = augment_with_goal(s_next, new_goal)
            augmented.append((new_s, a, new_r, new_s_next, done))
    return augmented

N-step return

def n_step_buffer(buffer, n=3, gamma=0.99):
    samples = buffer.sample(batch_size)
    s, a, r, s_next, done = samples
    n_step_r = np.zeros_like(r)
    for i in range(len(s)):
        cum = 0
        for k in range(n):
            cum += gamma**k * r[i + k] if i + k < len(r) else 0
            if i + k < len(done) and done[i + k]: break
        n_step_r[i] = cum
    return s, a, n_step_r, s_next, done

Distributed (Ape-X)

# 매 multiple actors → central learner
class DistributedReplay:
    def __init__(self, capacity, n_actors):
        self.local_buffers = [ReplayBuffer(capacity // n_actors) for _ in range(n_actors)]
        self.priorities = []  # 매 from learner
    
    def actor_push(self, actor_id, transition, td_estimate):
        self.local_buffers[actor_id].push(transition)
    
    def learner_sample(self, batch_size):
        # 매 weighted sample across local buffers
        return self._distributed_sample(batch_size)

Recurrent replay (R2D2)

class RecurrentReplay:
    """매 store sequences for LSTM."""
    def push_sequence(self, sequence_of_transitions):
        self.buf.append(sequence_of_transitions)
    
    def sample(self, batch_size, seq_len=80, burn_in=40):
        seqs = []
        for _ in range(batch_size):
            traj = self.buf[np.random.randint(len(self.buf))]
            start = np.random.randint(0, len(traj) - seq_len + 1)
            seqs.append(traj[start:start + seq_len])
        return seqs

DQN training (with replay)

def dqn_step(q_net, target_net, batch, optim, gamma=0.99):
    s, a, r, s_next, done = batch
    q = q_net(s).gather(1, a.unsqueeze(1)).squeeze()
    with torch.no_grad():
        q_next = target_net(s_next).max(1).values
    target = r + gamma * q_next * (1 - done)
    loss = F.smooth_l1_loss(q, target)
    optim.zero_grad(); loss.backward(); optim.step()
    return (target - q).abs().detach()  # 매 TD error for PER

Beta annealing (PER)

def anneal_beta(step, total_steps, start=0.4, end=1.0):
    return start + (end - start) * (step / total_steps)

매 결정 기준

상황	Approach
Standard DQN	Uniform
Sample-efficient	PER
Sparse reward + goal	HER
Long-horizon	n-step
LSTM policy	Recurrent (R2D2)
Massive scale	Distributed (Ape-X)
Lifelong	Reservoir

기본값: 매 PER + n-step (n=3) + double DQN. 매 sparse + goal: HER. 매 production: distributed.

🔗 Graph

• 부모: Reinforcement-Learning
• 변형: Experience-Replay
• 응용: 데이터 사이언스 및 ML 엔지니어링
• Adjacent: Eligibility-Traces · Catastrophic-Forgetting

🤖 LLM 활용

언제: 매 off-policy RL. 매 sparse reward. 매 distributed. 언제 X: 매 on-policy (PPO, A2C).

❌ 안티패턴

• No buffer: 매 correlation collapse.
• PER without weight correction: 매 biased.
• Tiny buffer: 매 forget.
• HER without symmetric reward: 매 fail.
• Sample on-policy: 매 method 의 mismatch.

🧪 검증 / 중복

• Verified (Mnih DQN 2013, Schaul PER 2016, Andrychowicz HER 2017, Horgan Ape-X 2018).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-04-26	RL-REPLAY auto
2026-05-08	Phase 1
2026-05-10	Manual cleanup — uniform / PER / HER / n-step / distributed code