refactor(topics): 멀티 에이전트용 지식 재편 — _Common(공통 기본기) + Domain_* 구조
에이전트 8종(대화형/프로그래머 C·S/디자이너/설계자/기획자/QA/PD/PM)에게 [공통 기본 능력 + 롤별 Specialty] 2층으로 지식을 주입하기 위한 재분류. 문서 내용·포맷은 무수정, 폴더 이동만 (6,372개 문서 수 보존 확인). - Topic_Programming → Domain_Programming (내부 구조 보존) - Topic_Graphic → Domain_Design - Topic_Business → Domain_Product - Topic_General → Domain_General - _Common 신설: Math(구 Topic_Math_Specialty), Reasoning(구 General/From_Thinking & Reasoning), Reasoning_Creativity(구 General/From_창의성), Communication(Poetic_Blog_Writing + From_writing) - 타 도메인의 From_* 폴더는 유지 (출처 표기일 뿐, 이미 도메인에 맞게 분류된 문서) - 빈 폴더 정리 (memory/procedures) - 에이전트→폴더 매핑은 workspace의 .astra/agent-knowledge-map.json (9개 에이전트) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Antigravity Agent
@@ -0,0 +1,185 @@
|
||||
---
|
||||
id: wiki-2026-0508-rouge-metrics
|
||||
title: ROUGE Metrics
|
||||
category: 10_Wiki/Topics
|
||||
status: verified
|
||||
canonical_id: self
|
||||
aliases: [ROUGE, ROUGE-1, ROUGE-2, ROUGE-L, Recall-Oriented Understudy for Gisting Evaluation]
|
||||
duplicate_of: none
|
||||
source_trust_level: A
|
||||
confidence_score: 0.9
|
||||
verification_status: applied
|
||||
tags: [nlp, evaluation, summarization, metric]
|
||||
raw_sources: []
|
||||
last_reinforced: 2026-05-10
|
||||
github_commit: pending
|
||||
tech_stack:
|
||||
language: python
|
||||
framework: rouge-score
|
||||
---
|
||||
|
||||
# ROUGE Metrics
|
||||
|
||||
## 매 한 줄
|
||||
> **"매 n-gram overlap recall 의 summary"**. Lin 2004 의 summarization eval 의 classic. 매 R-1 (unigram), R-2 (bigram), R-L (longest common subsequence) 의 trio. 2026 현재 매 still-default reference baseline, 매 supplemented by BERTScore / BARTScore / LLM-judge for semantic eval.
|
||||
|
||||
## 매 핵심
|
||||
|
||||
### 매 ROUGE variants
|
||||
- **ROUGE-N**: n-gram overlap recall. R-1 (unigram), R-2 (bigram).
|
||||
- **ROUGE-L**: Longest Common Subsequence (LCS) — 매 capture sentence-level structure, allow gap.
|
||||
- **ROUGE-W**: weighted LCS (consecutive 의 prefer).
|
||||
- **ROUGE-S / ROUGE-SU**: skip-bigram (with unigram) — 매 capture skip patterns.
|
||||
- **ROUGE-Lsum**: 매 summary-level — sentence-tokenize 후 매 sentence 의 LCS sum.
|
||||
|
||||
### 매 formula
|
||||
- Recall = matches / |reference n-grams|
|
||||
- Precision = matches / |candidate n-grams|
|
||||
- F1 = 2·P·R / (P+R)
|
||||
- 매 original paper 의 recall focused, 매 modern usage 의 F1 reported.
|
||||
|
||||
### 매 vs BLEU
|
||||
- BLEU: precision-oriented, machine translation. ROUGE: recall-oriented, summarization.
|
||||
- BLEU 의 brevity penalty, ROUGE 의 X (recall handles).
|
||||
- BLEU 의 corpus-level geometric mean, ROUGE 의 typically per-example.
|
||||
|
||||
### 매 limitations
|
||||
- 매 surface-level: synonym / paraphrase 의 penalize.
|
||||
- 매 fluency / factuality 의 capture X.
|
||||
- 매 tokenization sensitivity (BPE vs word).
|
||||
- 매 reference-dependent: 1 reference 의 high variance.
|
||||
|
||||
### 매 응용
|
||||
1. Summarization eval (CNN/DM, XSum, Gigaword).
|
||||
2. Long-doc summarization (arXiv, GovReport, BookSum 2026).
|
||||
3. RAG answer eval (vs gold answer).
|
||||
4. LLM eval reporting (still common alongside BERTScore + LLM-judge).
|
||||
|
||||
## 💻 패턴
|
||||
|
||||
### rouge-score basic
|
||||
```python
|
||||
from rouge_score import rouge_scorer
|
||||
|
||||
scorer = rouge_scorer.RougeScorer(
|
||||
["rouge1", "rouge2", "rougeL", "rougeLsum"],
|
||||
use_stemmer=True,
|
||||
)
|
||||
ref = "the cat sat on the mat in the morning"
|
||||
hyp = "a cat was sitting on the mat"
|
||||
scores = scorer.score(ref, hyp)
|
||||
# {'rouge1': Score(precision=0.71, recall=0.55, fmeasure=0.62), ...}
|
||||
print(scores["rougeL"].fmeasure)
|
||||
```
|
||||
|
||||
### HuggingFace evaluate
|
||||
```python
|
||||
import evaluate
|
||||
rouge = evaluate.load("rouge")
|
||||
results = rouge.compute(
|
||||
predictions=["the cat sat on the mat"],
|
||||
references=["a cat was sitting on the mat"],
|
||||
use_stemmer=True,
|
||||
use_aggregator=True,
|
||||
)
|
||||
# {'rouge1': 0.66, 'rouge2': 0.33, 'rougeL': 0.66, 'rougeLsum': 0.66}
|
||||
```
|
||||
|
||||
### Batch eval on dataset
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
import evaluate
|
||||
|
||||
ds = load_dataset("cnn_dailymail", "3.0.0", split="test[:100]")
|
||||
rouge = evaluate.load("rouge")
|
||||
|
||||
preds = [model.summarize(x["article"]) for x in ds]
|
||||
refs = [x["highlights"] for x in ds]
|
||||
|
||||
results = rouge.compute(predictions=preds, references=refs, use_stemmer=True)
|
||||
print(f"R-1: {results['rouge1']:.3f} R-2: {results['rouge2']:.3f} R-L: {results['rougeL']:.3f}")
|
||||
```
|
||||
|
||||
### Tokenization-aware (multilingual)
|
||||
```python
|
||||
# 매 default rouge-score 의 English-only stemmer
|
||||
# 매 multilingual: pre-tokenize with sentencepiece / language tokenizer
|
||||
from rouge_score.tokenizers import Tokenizer
|
||||
|
||||
class JaTokenizer(Tokenizer):
|
||||
def tokenize(self, text):
|
||||
import fugashi
|
||||
tagger = fugashi.Tagger()
|
||||
return [w.surface for w in tagger(text)]
|
||||
|
||||
scorer = rouge_scorer.RougeScorer(["rouge1", "rougeL"], tokenizer=JaTokenizer())
|
||||
```
|
||||
|
||||
### ROUGE alongside semantic metrics (2026 best practice)
|
||||
```python
|
||||
import evaluate
|
||||
rouge = evaluate.load("rouge")
|
||||
bertscore = evaluate.load("bertscore")
|
||||
|
||||
r = rouge.compute(predictions=preds, references=refs, use_stemmer=True)
|
||||
b = bertscore.compute(predictions=preds, references=refs, lang="en", model_type="microsoft/deberta-xlarge-mnli")
|
||||
|
||||
print(f"R-L: {r['rougeL']:.3f} BERTScore-F1: {sum(b['f1'])/len(b['f1']):.3f}")
|
||||
```
|
||||
|
||||
### LLM-as-judge supplement
|
||||
```python
|
||||
from anthropic import Anthropic
|
||||
client = Anthropic()
|
||||
|
||||
def judge(article, summary):
|
||||
msg = client.messages.create(
|
||||
model="claude-opus-4-7",
|
||||
max_tokens=200,
|
||||
messages=[{"role": "user", "content": f"""
|
||||
Rate this summary 1-5 on faithfulness and coverage.
|
||||
Article: {article}
|
||||
Summary: {summary}
|
||||
Output JSON: {{"faithfulness": int, "coverage": int}}
|
||||
"""}],
|
||||
)
|
||||
return msg.content[0].text
|
||||
```
|
||||
|
||||
## 매 결정 기준
|
||||
| 상황 | Approach |
|
||||
|---|---|
|
||||
| Quick lexical baseline | ROUGE-1 + ROUGE-L F1 |
|
||||
| Summarization paper | report R-1/R-2/R-L all |
|
||||
| Semantic eval needed | BERTScore + ROUGE both |
|
||||
| Factuality matters | LLM-judge or QAGS, not ROUGE |
|
||||
| Multilingual | language-specific tokenizer 의 plug |
|
||||
| Production monitoring | ROUGE-L + BERTScore + sample LLM-judge |
|
||||
|
||||
**기본값**: 매 ROUGE-L F1 + ROUGE-1 F1 의 report, 매 supplement BERTScore for semantic, 매 LLM-judge for nuance. 매 ROUGE alone 의 1980-style — 2026 의 multi-metric.
|
||||
|
||||
## 🔗 Graph
|
||||
- 부모: [[Summarization]]
|
||||
- 변형: [[ROUGE-L]]
|
||||
- Adjacent: [[LLM-as-Judge]]
|
||||
|
||||
## 🤖 LLM 활용
|
||||
**언제**: write eval scripts, explain ROUGE variants, generate reference summaries for testing.
|
||||
**언제 X**: as the metric itself — 매 ROUGE 의 deterministic, no LLM needed. 매 use LLM-judge as separate complementary metric.
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **ROUGE-only eval**: synonym 의 penalize, miss semantic equivalence.
|
||||
- **No stemming**: "running" vs "runs" 의 false negative.
|
||||
- **Single reference**: high variance, prefer multi-reference (CNN/DM 의 1, but XSum 의 1, GovReport 의 1 — limitation).
|
||||
- **ROUGE for QA / dialog**: 매 not summarization-shaped — use task-specific metric.
|
||||
- **ROUGE for factuality**: 매 ROUGE 의 surface-only, 매 hallucinated summary 의 still high R if word-overlap.
|
||||
|
||||
## 🧪 검증 / 중복
|
||||
- Verified (Lin 2004 ACL "ROUGE: A Package for Automatic Evaluation of Summaries", rouge-score 0.1.x, HF evaluate 2026).
|
||||
- 신뢰도 A.
|
||||
|
||||
## 🕓 Changelog
|
||||
| 날짜 | 변경 |
|
||||
|---|---|
|
||||
| 2026-05-08 | Phase 1 |
|
||||
| 2026-05-10 | Manual cleanup — ROUGE variants, rouge-score patterns, 2026 multi-metric guidance |
|
||||
Reference in New Issue
Block a user