refactor(topics): 멀티 에이전트용 지식 재편 — _Common(공통 기본기) + Domain_* 구조

에이전트 8종(대화형/프로그래머 C·S/디자이너/설계자/기획자/QA/PD/PM)에게 [공통 기본 능력 + 롤별 Specialty] 2층으로 지식을 주입하기 위한 재분류. 문서 내용·포맷은 무수정, 폴더 이동만 (6,372개 문서 수 보존 확인). - Topic_Programming → Domain_Programming (내부 구조 보존) - Topic_Graphic → Domain_Design - Topic_Business → Domain_Product - Topic_General → Domain_General - _Common 신설: Math(구 Topic_Math_Specialty), Reasoning(구 General/From_Thinking & Reasoning), Reasoning_Creativity(구 General/From_창의성), Communication(Poetic_Blog_Writing + From_writing) - 타 도메인의 From_* 폴더는 유지 (출처 표기일 뿐, 이미 도메인에 맞게 분류된 문서) - 빈 폴더 정리 (memory/procedures) - 에이전트→폴더 매핑은 workspace의 .astra/agent-knowledge-map.json (9개 에이전트) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 11:05:56 +09:00
parent 6549ead309
commit c24165b8bc
6193 changed files with 1717 additions and 31 deletions
@@ -0,0 +1,205 @@
+---
+id: wiki-2026-0508-pomdp
+title: POMDP
+category: 10_Wiki/Topics
+status: verified
+canonical_id: self
+aliases: [Partially-Observable-MDP, Partially-Observable-Markov-Decision-Process]
+duplicate_of: none
+source_trust_level: A
+confidence_score: 0.95
+verification_status: applied
+tags: [reinforcement-learning, planning, belief-state, pomdp, decision-making]
+raw_sources: []
+last_reinforced: 2026-05-10
+github_commit: pending
+tech_stack:
+  language: python
+  framework: pytorch-pomdp_py
+---
+
+# POMDP
+
+## 매 한 줄
+> **"매 MDP + observation noise"**. POMDP 는 agent 가 state 를 직접 관측하지 못하고 noisy observation 만 받는 경우의 decision-making 수학 framework — tuple `<S, A, T, R, Ω, O, γ>`. 매 belief state (state 위 distribution) 를 유지하며 행동, dialogue / robotics / medical / game-AI 의 standard model.
+
+## 매 핵심
+
+### 매 정의
+- **S**: state space (hidden).
+- **A**: action space.
+- **T(s'|s,a)**: transition.
+- **R(s,a)**: reward.
+- **Ω**: observation space.
+- **O(o|s',a)**: observation model.
+- **γ ∈ [0,1)**: discount.
+
+### 매 belief state
+- `b(s) = P(s | history)`, sufficient statistic of history.
+- update: `b'(s') ∝ O(o|s',a) Σ_s T(s'|s,a) b(s)`.
+- POMDP = MDP on belief space (continuous, high-dim).
+
+### 매 solver family
+1. **Exact**: value iteration on belief (PWLC), tractable only for tiny S.
+2. **Point-based** (PBVI, SARSOP, Perseus): sample beliefs, backup.
+3. **Online MCTS**: POMCP (Silver 2010), DESPOT — 매 large state, online planning.
+4. **Deep RL**: DRQN, R2D2, Dreamer (latent belief = RNN state) — 매 modern default.
+5. **Bayes-Adaptive**: BAMCP, learn dynamics in addition.
+
+### 매 vs MDP
+- MDP: full observability, policy `π(s) → a`.
+- POMDP: policy `π(b) → a` or `π(history) → a`.
+- **매 함정**: training MDP policy on observations directly = wrong (Markov violation).
+
+### 매 응용
+1. dialogue system — user goal hidden.
+2. robotics — sensor noise, occlusion.
+3. medical treatment — patient state from labs/symptoms.
+4. game AI — fog-of-war (StarCraft, Poker, [[Operation- Western Sun]]).
+5. autonomous driving — pedestrian intent.
+
+## 💻 패턴
+
+### Tiger problem (canonical POMDP)
+```python
+# States: tiger_left, tiger_right
+# Actions: open_left, open_right, listen
+# Obs: hear_left, hear_right (85% accurate after listen)
+import numpy as np
+
+S = ["TL", "TR"]
+A = ["OL", "OR", "LISTEN"]
+O = ["HL", "HR"]
+
+def T(s, a):
+    if a in ("OL", "OR"):
+        return {"TL": 0.5, "TR": 0.5}   # reset
+    return {s: 1.0}
+
+def R(s, a):
+    return {"LISTEN": -1,
+            "OL": -100 if s == "TL" else 10,
+            "OR": -100 if s == "TR" else 10}[a]
+
+def O_model(o, s, a):
+    if a != "LISTEN":
+        return 0.5
+    correct = (o == "HL" and s == "TL") or (o == "HR" and s == "TR")
+    return 0.85 if correct else 0.15
+```
+
+### Belief update (Bayes filter)
+```python
+def update_belief(b, a, o, S, T, O_model):
+    b_new = {}
+    for sp in S:
+        prior = sum(T(s, a).get(sp, 0) * b[s] for s in S)
+        b_new[sp] = O_model(o, sp, a) * prior
+    Z = sum(b_new.values())
+    return {s: p / Z for s, p in b_new.items()}
+```
+
+### Particle filter (continuous / large S)
+```python
+import numpy as np
+
+class ParticleBelief:
+    def __init__(self, particles): self.p = list(particles)
+    def update(self, a, o, sample_T, O_model):
+        new = []
+        for s in self.p:
+            sp = sample_T(s, a)
+            w = O_model(o, sp, a)
+            new.append((sp, w))
+        # resample
+        ws = np.array([w for _, w in new])
+        ws = ws / ws.sum()
+        idx = np.random.choice(len(new), len(new), p=ws)
+        self.p = [new[i][0] for i in idx]
+```
+
+### POMCP (online MCTS on history)
+```python
+import math, random
+from collections import defaultdict
+
+class POMCP:
+    def __init__(self, gen, c=1.0, gamma=0.95):
+        self.gen = gen      # generator: (s, a) -> (s', o, r)
+        self.c, self.gamma = c, gamma
+        self.N = defaultdict(int); self.V = defaultdict(float)
+    def search(self, belief, depth=20, sims=500):
+        for _ in range(sims):
+            s = random.choice(belief)
+            self._sim(s, (), depth)
+        return max(actions, key=lambda a: self.V[((), a)])
+    def _sim(self, s, h, d):
+        if d == 0: return 0
+        a = self._ucb(h)
+        sp, o, r = self.gen(s, a)
+        R = r + self.gamma * self._sim(sp, h + (a, o), d - 1)
+        self.N[(h, a)] += 1
+        self.V[(h, a)] += (R - self.V[(h, a)]) / self.N[(h, a)]
+        return R
+```
+
+### DRQN (Deep RL with recurrent belief)
+```python
+import torch, torch.nn as nn
+
+class DRQN(nn.Module):
+    def __init__(self, obs_dim, n_act, hidden=128):
+        super().__init__()
+        self.enc = nn.Linear(obs_dim, hidden)
+        self.gru = nn.GRU(hidden, hidden, batch_first=True)
+        self.q = nn.Linear(hidden, n_act)
+    def forward(self, obs_seq, h0=None):
+        x = self.enc(obs_seq).relu()
+        h, hN = self.gru(x, h0)
+        return self.q(h), hN
+```
+
+### pomdp_py (library)
+```python
+import pomdp_py
+# Define PomdpProblem, then:
+planner = pomdp_py.POMCP(max_depth=20, num_sims=1000,
+                        discount_factor=0.95, exploration_const=50)
+action = planner.plan(agent)
+```
+
+## 매 결정 기준
+| 문제 크기 | Solver |
+|---|---|
+| 매 |S| < 20 | exact / SARSOP |
+| 매 |S| < 10⁴, offline | point-based (SARSOP) |
+| 매 large S, online | POMCP / DESPOT |
+| 매 raw obs (image) | DRQN / Dreamer |
+| 매 unknown dynamics | Bayes-Adaptive / model-based RL |
+
+**기본값**: SARSOP for tabular, Dreamer-V3 for pixel.
+
+## 🔗 Graph
+- 부모: [[MDP]] · [[Reinforcement-Learning]] · [[Decision Theory]]
+- 응용: [[Robotics]] · [[Operation- Western Sun]]
+- Adjacent: [[MCTS]]
+
+## 🤖 LLM 활용
+**언제**: 매 partial observability 문제 framing, belief-state design, solver 추천.
+**언제 X**: 매 fully-observable env — MDP 면 충분.
+
+## ❌ 안티패턴
+- **Treat obs as state**: Markov violation, policy 가 frame stacking 으로 hack 만 가능.
+- **Forget belief in test**: training 시 belief, deployment 시 raw obs 전달.
+- **Exact solver on large S**: PWLC explosion — point-based 로.
+- **No exploration in POMCP**: c=0 → greedy, belief 가 collapse.
+
+## 🧪 검증 / 중복
+- Verified (Kaelbling 1998, Silver 2010 POMCP, Hafner 2023 Dreamer-V3).
+- 신뢰도 A.
+
+## 🕓 Changelog
+| 날짜 | 변경 |
+|---|---|
+| 2026-05-08 | Phase 1 |
+| 2026-05-10 | Manual cleanup — definition + solver family + Tiger/POMCP/DRQN |