docs(10_Wiki): 위키 전체 재구성 — Topic_* 폴더를 4개 카테고리로 통합 + 대규모 중복 제거

Topic_Agent/Topic_Blog/Topics/Topics_Biz/Topics_Meeting/Topics_Rag의 마크다운 지식 문서를 Topic_General/Topic_Programming/Topic_Graphic/Topic_Business 4개 카테고리로 재분류. - 중복 제거: frontmatter의 status:duplicate/merged + duplicate_of/redirect_to 필드로 자기 자신을 중복으로 선언한 리다이렉트 stub 1032개 제거, 완전 동일 내용 파일 472개 제거, 동일 파일명·다른 내용 충돌 시 더 큰(완전한) 버전만 유지(162개 제거) — 총 1639개 중복 제거. - 분류: 폴더 단위로 명확한 항목(AI_and_ML/Coding/Architecture 등 → Programming, Comfyui/Visual_Effects → Graphic, Topics_Biz/Topics_Meeting/사업 등 → Business, Poetic_Blog_Writing/창의성/Game_Design 등 → General)은 폴더 우선순위로, 나머지 혼재 폴더(Topic_Agent/Topic_Blog/Topics 루트/Thinking & Reasoning/Other/UI_UX_Assets)는 title/tags 키워드 스코어링으로 파일 단위 분류(불명확한 경우 General로 폴백). 원본 폴더명은 "From_*" 서브폴더로 보존해 추적 가능성 유지. - 최종 배치: Programming 2784 / General 1608 / Graphic 285 / Business 249 = 4926개 문서. - 에이전트 운영 상태(.astra/.agent/.obsidian/sessions/memory/_company/docs/lessons/_shared/src)는 지식 콘텐츠가 아니므로 재분류 대상에서 제외하고 원위치 유지. - Topics/Topic_email(상위 보호 폴더 Topic_email과 파일명 100% 중복) 삭제 — 보호 폴더 자체는 미변경. - 완전히 비게 된 Topic_Agent/Topic_Blog/Topics_Biz/Topics_Rag 폴더 제거.
2026-07-05 00:33:48 +09:00
parent 1cfd3bbb56
commit 9148c358d0
6455 changed files with 1 additions and 86875 deletions
@@ -0,0 +1,167 @@
+---
+id: wiki-2026-0508-mapreduce
+title: MapReduce
+category: 10_Wiki/Topics
+status: verified
+canonical_id: self
+aliases: [맵리듀스, Hadoop MR, Map-Reduce]
+duplicate_of: none
+source_trust_level: A
+confidence_score: 0.9
+verification_status: applied
+tags: [distributed, big-data, parallel, hadoop, batch]
+raw_sources: []
+last_reinforced: 2026-05-10
+github_commit: pending
+tech_stack:
+  language: python
+  framework: hadoop-spark
+---
+
+# MapReduce
+
+## 매 한 줄
+> **"매 split → map → shuffle → reduce"**. MapReduce (Dean & Ghemawat, Google 2004) 는 대규모 batch 처리 의 functional programming 모델. 2026 perspective 에서 raw Hadoop MR 은 legacy, Spark / Flink / BigQuery / Beam 이 후속 표준.
+
+## 매 핵심
+
+### 매 4 phase
+- **Split**: input → fixed-size shards (HDFS block 64-128MB).
+- **Map**: (k1, v1) → list[(k2, v2)]. Stateless, parallelizable.
+- **Shuffle/Sort**: same k2 grouped to same reducer.
+- **Reduce**: (k2, list[v2]) → list[(k3, v3)].
+
+### 매 design principles
+- **Data locality**: code → data, not data → code.
+- **Fault tolerance**: re-execute failed tasks (idempotent map/reduce).
+- **Speculative execution**: slow tasks 의 backup copy.
+- **Immutable inputs**: re-runnable.
+
+### 매 응용
+1. Log analysis / web indexing (original use case).
+2. ETL pipelines.
+3. ML feature aggregation.
+4. Data warehouse build.
+
+## 💻 패턴
+
+### Word count (canonical)
+```python
+from collections import defaultdict
+from itertools import groupby
+
+def map_phase(doc_id, text):
+    for word in text.split():
+        yield (word.lower(), 1)
+
+def reduce_phase(word, counts):
+    yield (word, sum(counts))
+
+def mapreduce(docs):
+    # Map
+    pairs = [kv for did, t in docs for kv in map_phase(did, t)]
+    # Shuffle
+    pairs.sort(key=lambda x: x[0])
+    grouped = {k: [v for _, v in g] for k, g in groupby(pairs, key=lambda x: x[0])}
+    # Reduce
+    return dict(kv for k, vs in grouped.items() for kv in reduce_phase(k, vs))
+```
+
+### Combiner (local reduce)
+```python
+def map_with_combiner(doc_id, text):
+    local = defaultdict(int)
+    for word in text.split():
+        local[word.lower()] += 1
+    for w, c in local.items():
+        yield (w, c)
+# 매 network shuffle 양 감소
+```
+
+### Spark RDD equivalent
+```python
+from pyspark import SparkContext
+sc = SparkContext()
+
+counts = (sc.textFile("hdfs:///logs/*.txt")
+            .flatMap(lambda line: line.split())
+            .map(lambda w: (w.lower(), 1))
+            .reduceByKey(lambda a, b: a + b))
+counts.saveAsTextFile("hdfs:///out/wc")
+```
+
+### Inverted index
+```python
+def map_idx(doc_id, text):
+    for word in set(text.split()):
+        yield (word.lower(), doc_id)
+
+def reduce_idx(word, doc_ids):
+    yield (word, sorted(set(doc_ids)))
+```
+
+### Secondary sort
+```python
+# Composite key for sort-within-group
+def map_temp(line):
+    parts = line.split(",")
+    year, temp = parts[0], int(parts[1])
+    yield ((year, temp), None)  # negative temp for desc
+
+def partitioner(key):
+    return hash(key[0]) % num_reducers  # group by year only
+
+def grouping_comparator(a, b):
+    return (a[0] > b[0]) - (a[0] < b[0])  # year only
+```
+
+### Join (reduce-side)
+```python
+def map_users(row):
+    yield (row["user_id"], ("user", row))
+
+def map_orders(row):
+    yield (row["user_id"], ("order", row))
+
+def reduce_join(uid, tagged):
+    user = next(r for tag, r in tagged if tag == "user")
+    for tag, r in tagged:
+        if tag == "order":
+            yield {**user, **r}
+```
+
+## 매 결정 기준
+| 상황 | Approach |
+|---|---|
+| Batch ETL on TB+ | Spark (Hadoop MR 은 legacy) |
+| Streaming | Flink / Spark Structured Streaming |
+| SQL-shaped query | BigQuery / Athena / Presto |
+| Cross-cloud portability | Apache Beam |
+| Educational | Raw MR pseudocode |
+
+**기본값**: Spark for new projects; Hadoop MR 은 legacy 유지보수만.
+
+## 🔗 Graph
+- 부모: [[Distributed Systems]] · [[Parallel-Computing]]
+- 변형: [[Spark]] · [[Apache Flink]]
+- 응용: [[Data Pipeline]] · [[ETL]]
+
+## 🤖 LLM 활용
+**언제**: pipeline design review, Spark migration 가이드, query optimization.
+**언제 X**: real-time low-latency — wrong paradigm.
+
+## ❌ 안티패턴
+- **Many small files**: HDFS namenode 폭발. 매 compaction 필수.
+- **Skewed keys**: 한 reducer 가 hotspot — salting / combiner 로 완화.
+- **Stateful map**: 매 idempotency 깨짐 → fault recovery 실패.
+- **Re-implementing SQL**: 매 BigQuery / Spark SQL 사용.
+
+## 🧪 검증 / 중복
+- Verified (Dean & Ghemawat OSDI 2004, Spark NSDI 2012, Hadoop docs 3.x).
+- 신뢰도 A.
+
+## 🕓 Changelog
+| 날짜 | 변경 |
+|---|---|
+| 2026-05-08 | Phase 1 |
+| 2026-05-10 | Manual cleanup — word-count + Spark + secondary-sort + join 패턴 |