---
id: wiki-2026-0508-information-retrieval-evaluation
title: Information Retrieval Evaluation Metrics
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [IR Metrics, Search Evaluation, NDCG, MRR]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [ir, evaluation, metrics, ranking]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: pytrec_eval
---

# Information Retrieval Evaluation Metrics

## 매 한 줄
> **"매 metric 매 system behavior 의 directs"**. IR evaluation 매 ranked list 의 quality measure — 매 binary (precision, recall) vs graded (NDCG). 매 2026 LLM-as-judge, 매 BEIR/MTEB benchmark 매 standard, 매 nDCG@10 매 default.

## 매 핵심

### 매 binary relevance metrics
- **Precision@k**: 매 top-k 중 relevant 비율.
- **Recall@k**: 매 전체 relevant 중 retrieved 비율.
- **F1**: 매 harmonic mean.
- **MAP** (Mean Average Precision): 매 query별 AP 의 평균.
- **MRR** (Mean Reciprocal Rank): 매 first relevant 의 1/rank 평균.

### 매 graded relevance metrics
- **DCG@k**: $\sum_{i=1}^{k} \frac{rel_i}{\log_2(i+1)}$
- **NDCG@k**: DCG / IDCG — 매 [0,1] normalize.
- **ERR** (Expected Reciprocal Rank): 매 cascade user model.

### 매 응용
1. 매 search engine A/B (Google, Bing).
2. 매 RAG retrieval quality (Ragas).
3. 매 recommender systems.
4. 매 LLM benchmark (BEIR, MTEB, MS MARCO).

## 💻 패턴

### 1. Precision/Recall
```python
def precision_at_k(retrieved, relevant, k):
    return len(set(retrieved[:k]) & set(relevant)) / k

def recall_at_k(retrieved, relevant, k):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)
```

### 2. NDCG (graded)
```python
import numpy as np

def dcg(rels, k=10):
    rels = np.asarray(rels)[:k]
    return np.sum(rels / np.log2(np.arange(2, len(rels)+2)))

def ndcg(rels, ideal_rels, k=10):
    return dcg(rels, k) / (dcg(ideal_rels, k) or 1)
```

### 3. MRR
```python
def mrr(retrieved_lists, relevant_sets):
    total = 0
    for retrieved, relevant in zip(retrieved_lists, relevant_sets):
        for i, doc in enumerate(retrieved, 1):
            if doc in relevant:
                total += 1 / i
                break
    return total / len(retrieved_lists)
```

### 4. MAP
```python
def average_precision(retrieved, relevant):
    hits = 0; sum_p = 0
    for i, doc in enumerate(retrieved, 1):
        if doc in relevant:
            hits += 1
            sum_p += hits / i
    return sum_p / len(relevant) if relevant else 0
```

### 5. pytrec_eval (Standard Tool)
```python
import pytrec_eval
qrel = {'q1': {'d1': 1, 'd3': 2}}
run = {'q1': {'d1': 0.9, 'd2': 0.5, 'd3': 0.3}}
ev = pytrec_eval.RelevanceEvaluator(qrel, {'ndcg_cut.10', 'map', 'recip_rank'})
print(ev.evaluate(run))
```

### 6. LLM-as-Judge (2026 trend)
```python
from anthropic import Anthropic
client = Anthropic()
prompt = f"Query: {q}\nDoc: {doc}\nRate relevance 0-3:"
score = client.messages.create(model="claude-opus-4-7",
    max_tokens=10, messages=[{"role":"user","content":prompt}])
# 매 graded relevance — human-aligned, 매 expensive
```

### 7. Statistical Significance (paired t-test)
```python
from scipy.stats import ttest_rel
t, p = ttest_rel(ndcg_system_A, ndcg_system_B)
# 매 p < 0.05 → 매 significant difference
```

## 매 결정 기준
| 상황 | Metric |
|---|---|
| Web search top result | MRR |
| Long-tail recall | Recall@1000 |
| Graded relevance | NDCG@10 |
| Multi-relevant per query | MAP |
| RAG eval | NDCG@5 + LLM-judge |

**기본값**: NDCG@10, 매 paired-t test 매 statistical significance.

## 🔗 Graph
- 부모: [[Information-Retrieval]] · [[Statistics]]
- 변형: [[Ranking-Algorithms]] · [[Relevance-Feedback]]
- 응용: [[Keyword Search]] · [[Knowledge Graph]]
- Adjacent: [[Statistical-Power]] · [[Cross-Validation]]

## 🤖 LLM 활용
**언제**: 매 search system A/B 설계, 매 RAG eval pipeline, 매 metric trade-off 분석.
**언제 X**: 매 single-doc QA (use exact-match), 매 generative output (use BLEU/ROUGE/judge).

## ❌ 안티패턴
- **Precision@1 only**: 매 long-tail blind spot.
- **No statistical test**: 매 noise-driven conclusion.
- **Mismatched qrels**: 매 partial relevance judgment.
- **Self-reported benchmark**: 매 trec_eval/standard tool 사용 권장.

## 🧪 검증 / 중복
- Verified (Manning et al., *IR Book*; TREC standards; BEIR paper 2021).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — full metric coverage + LLM-judge pattern |
