2nd/10_Wiki/Topics/AI_and_ML/Assessment.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-assessment	Assessment (Educational + ML Evaluation)	10_Wiki/Topics	verified	self	평가 evaluation formative summative validity reliability rubric ml-evaluation	none	B	0.88	applied	assessment evaluation education validity reliability fairness rubric ml-eval llm-judge		2026-05-10	pending	language applicable_to education / ML Educational Tech ML Evaluation Performance Review

Assessment

📌 한 줄 통찰

"매 성장 의 거울". 매 current 의 measure + 매 gap → 매 direction. 매 selection 의 X — 매 growth 의 support. 매 modern AI 의 ML evaluation 의 same principle (validity / reliability / fairness).

📖 핵심

매 timing 의 분류

1. Diagnostic (진단): 매 시작 전 의 수준.
2. Formative (형성): 매 진행 중 의 feedback.
3. Summative (총괄): 매 final 의 성취.
4. Authentic: 매 real-world task.

매 quality criteria

• Validity (타당도): 매 measure 의 right thing?
  • Construct: 매 construct 의 capture.
  • Content: 매 domain 의 cover.
  • Predictive: 매 future 의 predict.
  • Face: 매 looks-like-it.
• Reliability (신뢰도): 매 consistent?
  • Test-retest: 매 시간 의 stable.
  • Inter-rater: 매 rater 의 agree.
  • Internal consistency (Cronbach's α).
• Fairness: 매 equal opportunity.
• Authenticity: 매 real-world ≈.

매 educational paradigm

Behaviorist (전통)

• 매 multiple choice.
• 매 right/wrong.

Cognitivist

• 매 understanding.
• 매 short answer / explain.

Constructivist

• 매 portfolio.
• 매 project.
• 매 self/peer reflection.

매 ML evaluation 의 parallel

Education	ML
Validity	매 construct 의 measure
Reliability	매 consistent across runs
Fairness	매 group equity
Diagnostic	매 capability profiling
Formative	매 dev set
Summative	매 test set
Authentic	매 real-world deploy

매 modern issue

LLM-as-judge

• 매 fast + 매 cheap.
• 매 self-bias (GPT-4 가 GPT-4 의 favor).
• 매 calibration 필요.

Multi-dimensional

• 매 single metric 의 X.
• 매 quality + safety + cost + latency.

Adaptive

• 매 IRT (Item Response Theory).
• 매 difficulty 의 adapt.
• 매 GRE / 매 personalized education.

Continuous

• 매 portfolio.
• 매 logging-based.
• 매 longitudinal.

매 rubric (good)

• 매 specific criteria.
• 매 levels (4-6).
• 매 anchored example.
• 매 actionable feedback.

💻 패턴

Rubric (educational)

# 매 essay rubric
criteria:
  - name: Argument
    levels:
      4: "Sophisticated argument with nuance and counter-evidence"
      3: "Clear argument with relevant support"
      2: "Argument present but weakly supported"
      1: "No clear argument or off-topic"
  - name: Evidence
    levels:
      4: "Multiple high-quality sources, integrated"
      3: "Adequate sources cited"
      2: "Few or weak sources"
      1: "No evidence or invented"
  - name: Writing
    levels:
      4: "Polished, varied, error-free"
      3: "Clear, mostly correct"
      2: "Comprehensible but error-laden"
      1: "Incomprehensible"

scoring: weighted_sum  # 매 levels[criterion] * weight

LLM-as-judge (educational)

def judge_essay(essay, rubric):
    prompt = f"""Score this essay against the rubric. Return JSON.

Rubric: {rubric}

Essay:
{essay}

Format:
{{
  "argument": {{ "score": 1-4, "evidence": "..." }},
  "evidence": {{ "score": 1-4, "evidence": "..." }},
  "writing": {{ "score": 1-4, "evidence": "..." }},
  "feedback": "actionable feedback in 3 sentences"
}}"""
    
    response = llm.generate(prompt)
    return json.loads(response)

# 매 calibration
# 매 N=3 judge → 매 average. 매 disagreement → 매 human review.

Inter-rater agreement (Cohen's kappa)

from sklearn.metrics import cohen_kappa_score

def measure_reliability(rater1_scores, rater2_scores):
    kappa = cohen_kappa_score(rater1_scores, rater2_scores)
    if kappa < 0.4: return 'poor'
    if kappa < 0.6: return 'fair'
    if kappa < 0.8: return 'good'
    return 'excellent'

IRT (adaptive testing)

import numpy as np

def irt_3pl(theta, a, b, c):
    """매 3-parameter logistic.
    theta: ability, a: discrimination, b: difficulty, c: guessing."""
    return c + (1 - c) / (1 + np.exp(-a * (theta - b)))

def adaptive_next_item(theta_estimate, item_pool, answered_ids):
    # 매 information 의 maximum 의 item.
    candidates = [item for item in item_pool if item.id not in answered_ids]
    info = lambda item: item.a**2 * irt_3pl(theta_estimate, item.a, item.b, item.c) * \
                        (1 - irt_3pl(theta_estimate, item.a, item.b, item.c))
    return max(candidates, key=info)

Fairness check (group)

def fairness_check(scores, group_labels):
    by_group = collections.defaultdict(list)
    for score, group in zip(scores, group_labels):
        by_group[group].append(score)
    
    means = {g: np.mean(s) for g, s in by_group.items()}
    
    # 매 disparate impact
    max_mean = max(means.values())
    min_mean = min(means.values())
    if min_mean / max_mean < 0.8:
        return f'WARN: disparate impact: {min_mean/max_mean:.2f} < 0.8'
    return 'OK'

Portfolio assessment

class Portfolio:
    def __init__(self, student_id):
        self.student_id = student_id
        self.artifacts = []
    
    def add(self, artifact):
        self.artifacts.append({
            'id': artifact.id,
            'date': artifact.date,
            'type': artifact.type,  # essay, code, image
            'reflection': artifact.reflection,
        })
    
    def progression(self):
        # 매 시간 의 growth 의 visualize
        scores_over_time = [(a.date, a.score) for a in self.artifacts]
        return scores_over_time

ML evaluation suite (multi-dim)

def evaluate_model(model, eval_set):
    return {
        'accuracy': accuracy(model, eval_set),
        'fairness': fairness_check(model, eval_set, sensitive='gender'),
        'safety': safety_score(model, harm_set),
        'calibration': ece(model, eval_set),
        'latency_p95': latency(model),
        'cost_per_1k': cost(model),
        'human_pref': pairwise_human(model, baseline, n=100),
    }

🤔 결정 기준

상황	Approach
Standardized test	Summative + IRT
Personalized learning	Diagnostic + adaptive
Skill development	Formative + portfolio
LLM evaluation	Multi-metric + LLM-judge + human
Hiring	Authentic + rubric + structured
Performance review	360° + portfolio

기본값: Multi-method + rubric + inter-rater check + fairness audit.

🔗 Graph

• 부모: Evaluation
• 응용: Rubric
• ML parallel: ML-Evaluation · Benchmarks · LLM-as-Judge · Bias-Correction-Algorithm
• Adjacent: Algorithmic Fairness · Validity · Reliability

🤖 LLM 활용

언제: 매 educational system design. 매 ML evaluation suite. 매 performance review framework. 매 rubric 작성. 언제 X: 매 single high-stakes metric (Goodhart). 매 fairness 의 ignore.

❌ 안티패턴

• Single-metric: 매 saturate / game.
• No rubric: 매 inter-rater disagreement.
• Stale benchmark: 매 contamination.
• No fairness check: 매 disparate impact.
• Diagnostic 의 stigma: 매 student labeling.
• LLM judge 의 single: 매 self-bias.
• No validation 의 construct: 매 wrong thing measured.

🧪 검증 / 중복

• Verified (educational psychology + ML evaluation literature).
• 신뢰도 B.
• Related: Benchmarks · Bias-Correction-Algorithm · Algorithmic Fairness · LLM-as-Judge.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — type + criteria + ML parallel + rubric / IRT / fairness code