2nd/10_Wiki/Topics/AI_and_ML/Experience-Replay.md

---
id: wiki-2026-0508-experience-replay
title: Experience Replay
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [experience replay, replay buffer, prioritized replay, hindsight, HER]
duplicate_of: none
source_trust_level: A
confidence_score: 0.97
verification_status: applied
tags: [reinforcement-learning, dqn, replay-buffer, prioritized, her, off-policy]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: PyTorch / Stable-Baselines3 / Tianshou
---

# Experience Replay

## 매 한 줄
> **"매 RL transition 의 buffer 의 store + sample → 매 i.i.d.-like train"**. Lin 1992, Mnih DQN 2013. 매 off-policy 의 backbone. 매 modern: 매 Prioritized (PER), Hindsight (HER), distributed (Ape-X), n-step.

## 매 핵심

### 매 motivation
- **Sequential correlation**: 매 trajectory 의 i.i.d. X.
- **Sample efficiency**: 매 reuse.
- **Stability**: 매 catastrophic forgetting 의 reduce.

### 매 variant
- **Uniform**: 매 random sample.
- **Prioritized (PER)**: 매 TD error.
- **HER**: 매 sparse reward 의 goal relabel.
- **n-step**: 매 multi-step return.
- **Reservoir** (Pham): 매 lifelong.
- **Distributed** (Ape-X, R2D2): 매 actor 의 parallel.

### 매 응용
- **DQN / DDQN**: 매 standard.
- **DDPG / TD3 / SAC**: 매 continuous.
- **HER**: 매 robotics goal.
- **Recurrent**: 매 sequence.

## 💻 패턴

### Uniform replay buffer
```python
import numpy as np
from collections import deque

class ReplayBuffer:
    def __init__(self, capacity=100_000):
        self.buf = deque(maxlen=capacity)
    
    def push(self, s, a, r, s_next, done):
        self.buf.append((s, a, r, s_next, done))
    
    def sample(self, batch_size):
        idx = np.random.choice(len(self.buf), batch_size, replace=False)
        batch = [self.buf[i] for i in idx]
        return [np.array(x) for x in zip(*batch)]
    
    def __len__(self): return len(self.buf)
```

### Prioritized Experience Replay (PER, Schaul 2016)
```python
class PER:
    def __init__(self, capacity, alpha=0.6, beta=0.4):
        self.tree = SumTree(capacity)
        self.alpha = alpha
        self.beta = beta
        self.eps = 1e-6
        self.max_p = 1.0
    
    def push(self, transition):
        self.tree.add(self.max_p, transition)
    
    def sample(self, batch_size):
        batch, idxs, priorities = [], [], []
        seg = self.tree.total() / batch_size
        for i in range(batch_size):
            s = np.random.uniform(seg * i, seg * (i + 1))
            idx, p, data = self.tree.get(s)
            batch.append(data); idxs.append(idx); priorities.append(p)
        
        sampling_p = np.array(priorities) / self.tree.total()
        weights = (len(self.tree) * sampling_p) ** -self.beta
        weights /= weights.max()
        return batch, idxs, weights
    
    def update_priorities(self, idxs, td_errors):
        for idx, err in zip(idxs, td_errors):
            p = (abs(err) + self.eps) ** self.alpha
            self.tree.update(idx, p)
            self.max_p = max(self.max_p, p)
```

### SumTree (PER 의 efficient sample)
```python
class SumTree:
    def __init__(self, cap):
        self.cap = cap
        self.tree = np.zeros(2 * cap - 1)
        self.data = np.zeros(cap, dtype=object)
        self.ptr = 0
    
    def add(self, p, data):
        idx = self.ptr + self.cap - 1
        self.data[self.ptr] = data
        self.update(idx, p)
        self.ptr = (self.ptr + 1) % self.cap
    
    def update(self, idx, p):
        change = p - self.tree[idx]
        self.tree[idx] = p
        while idx != 0:
            idx = (idx - 1) // 2
            self.tree[idx] += change
    
    def get(self, s):
        idx = 0
        while idx < self.cap - 1:
            l, r = 2 * idx + 1, 2 * idx + 2
            if s <= self.tree[l]: idx = l
            else: s -= self.tree[l]; idx = r
        data_idx = idx - self.cap + 1
        return idx, self.tree[idx], self.data[data_idx]
    
    def total(self): return self.tree[0]
```

### Hindsight Experience Replay (HER)
```python
def her_relabel(trajectory, k=4, strategy='future'):
    """매 sparse-reward goal-conditioned RL."""
    augmented = []
    for t, (s, a, r, s_next, done) in enumerate(trajectory):
        augmented.append((s, a, r, s_next, done))
        # 매 future strategy: 매 future state 의 새 goal
        for _ in range(k):
            future_t = np.random.randint(t, len(trajectory))
            new_goal = trajectory[future_t][3]  # 매 future s_next
            new_r = compute_reward(s_next, new_goal)
            new_s = augment_with_goal(s, new_goal)
            new_s_next = augment_with_goal(s_next, new_goal)
            augmented.append((new_s, a, new_r, new_s_next, done))
    return augmented
```

### N-step return
```python
def n_step_buffer(buffer, n=3, gamma=0.99):
    samples = buffer.sample(batch_size)
    s, a, r, s_next, done = samples
    n_step_r = np.zeros_like(r)
    for i in range(len(s)):
        cum = 0
        for k in range(n):
            cum += gamma**k * r[i + k] if i + k < len(r) else 0
            if i + k < len(done) and done[i + k]: break
        n_step_r[i] = cum
    return s, a, n_step_r, s_next, done
```

### Distributed (Ape-X)
```python
# 매 multiple actors → central learner
class DistributedReplay:
    def __init__(self, capacity, n_actors):
        self.local_buffers = [ReplayBuffer(capacity // n_actors) for _ in range(n_actors)]
        self.priorities = []  # 매 from learner
    
    def actor_push(self, actor_id, transition, td_estimate):
        self.local_buffers[actor_id].push(transition)
    
    def learner_sample(self, batch_size):
        # 매 weighted sample across local buffers
        return self._distributed_sample(batch_size)
```

### Recurrent replay (R2D2)
```python
class RecurrentReplay:
    """매 store sequences for LSTM."""
    def push_sequence(self, sequence_of_transitions):
        self.buf.append(sequence_of_transitions)
    
    def sample(self, batch_size, seq_len=80, burn_in=40):
        seqs = []
        for _ in range(batch_size):
            traj = self.buf[np.random.randint(len(self.buf))]
            start = np.random.randint(0, len(traj) - seq_len + 1)
            seqs.append(traj[start:start + seq_len])
        return seqs
```

### DQN training (with replay)
```python
def dqn_step(q_net, target_net, batch, optim, gamma=0.99):
    s, a, r, s_next, done = batch
    q = q_net(s).gather(1, a.unsqueeze(1)).squeeze()
    with torch.no_grad():
        q_next = target_net(s_next).max(1).values
    target = r + gamma * q_next * (1 - done)
    loss = F.smooth_l1_loss(q, target)
    optim.zero_grad(); loss.backward(); optim.step()
    return (target - q).abs().detach()  # 매 TD error for PER
```

### Beta annealing (PER)
```python
def anneal_beta(step, total_steps, start=0.4, end=1.0):
    return start + (end - start) * (step / total_steps)
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Standard DQN | Uniform |
| Sample-efficient | PER |
| Sparse reward + goal | HER |
| Long-horizon | n-step |
| LSTM policy | Recurrent (R2D2) |
| Massive scale | Distributed (Ape-X) |
| Lifelong | Reservoir |

**기본값**: 매 PER + n-step (n=3) + double DQN. 매 sparse + goal: HER. 매 production: distributed.

## 🔗 Graph
- 부모: [[Reinforcement-Learning]]
- 변형: [[Experience-Replay|Prioritized-Experience-Replay]]
- 응용: [[데이터_사이언스_및_ML_엔지니어링|DQN]]
- Adjacent: [[Eligibility-Traces]] · [[Catastrophic-Forgetting]]

## 🤖 LLM 활용
**언제**: 매 off-policy RL. 매 sparse reward. 매 distributed.
**언제 X**: 매 on-policy (PPO, A2C).

## ❌ 안티패턴
- **No buffer**: 매 correlation collapse.
- **PER without weight correction**: 매 biased.
- **Tiny buffer**: 매 forget.
- **HER without symmetric reward**: 매 fail.
- **Sample on-policy**: 매 method 의 mismatch.

## 🧪 검증 / 중복
- Verified (Mnih DQN 2013, Schaul PER 2016, Andrychowicz HER 2017, Horgan Ape-X 2018).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-04-26 | RL-REPLAY auto |
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — uniform / PER / HER / n-step / distributed code |