2nd/10_Wiki/Topic_Programming/From_Other/Automation-Paradox.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-automation-paradox	Automation Paradox	10_Wiki/Topics	verified	self	Paradox of Automation Bainbridge's Ironies Lights-Out Fallacy	none	A	0.92	applied	human-factors automation ai-safety ergonomics system-design		2026-05-10	pending	language framework n/a human-factors / HRO

Automation Paradox

매 한 줄

"매 자동화 의 better 일수록, human operator 의 role 의 critical 의 increase — skill atrophy 와 vigilance decrement 의 통한 catastrophic edge-case 의 amplification". Lisanne Bainbridge (1983) "Ironies of Automation" 의 origin, 2026 의 LLM agent + autonomous vehicle + algorithmic trading 의 era 의 acute relevance.

매 핵심

매 4 ironies (Bainbridge)

• Designer 의 error: automation 의 design 의 bug 의 operator 의 inherit
• Skill atrophy: routine-task 의 takeover 의 인해 human skill 의 decay → emergency 의 unable
• Monitoring task: vigilance 의 인해 의 unsuited 의 task 의 human 의 assign
• Trust calibration: under-trust (rejection) 또는 over-trust (complacency) 의 binary failure

매 mechanisms

• Out-of-the-loop unfamiliarity (OOTLUF) — automation handover 의 시 의 operator 의 context 의 lack
• Mode confusion — automation 의 current state 의 mismatch (Air France 447, Tesla autopilot)
• Skill decay curve — manual skill 의 disuse 의 인해 의 exponential degradation (~6 months)
• Calibration drift — automation 의 reliability 의 over-extrapolation

매 응용

1. Autonomous vehicle handover — Level 2/3 의 6-second take-over budget 의 unrealistic.
2. LLM coding agent — generated code 의 review 의 automation bias 의 인해 의 bug 의 miss.
3. Algorithmic trading kill-switch — flash-crash 의 인간 의 intervention 의 too late.
4. Aviation glass cockpit — Air France 447 (2009) 의 stall 의 mode confusion.

💻 패턴

Trust calibration metric (TLX-derived)

from dataclasses import dataclass

@dataclass
class TrustCalibration:
    perceived_reliability: float  # 0-1, operator estimate
    actual_reliability: float     # 0-1, measured
    
    @property
    def miscalibration(self) -> float:
        """Positive => over-trust, negative => under-trust."""
        return self.perceived_reliability - self.actual_reliability
    
    def risk_class(self) -> str:
        gap = self.miscalibration
        if gap > 0.15:  return "OVER_TRUST_DANGER"
        if gap < -0.15: return "REJECTION_DANGER"
        return "CALIBRATED"

Vigilance decrement model

import numpy as np

def vigilance_curve(t_minutes: np.ndarray, base_hit_rate: float = 0.95) -> np.ndarray:
    """Mackworth clock — 30-min decrement of ~0.2 in detection."""
    decay = 0.2 * (1 - np.exp(-t_minutes / 15))
    return np.clip(base_hit_rate - decay, 0, 1)

# Recommendation: rotate operators every 20 min on monitoring tasks

Handover protocol (autonomous vehicle)

class L3HandoverManager:
    def __init__(self, min_takeover_seconds: float = 12.0):
        self.budget = min_takeover_seconds
    
    def request_handover(self, driver_state: dict) -> dict:
        # 2026 SAE J3016 update: budget grew from 6s to 10-15s
        if not driver_state["eyes_on_road"]:
            return {"action": "MRM_pull_over", "reason": "OOTLUF"}
        if driver_state["secondary_task"] == "phone":
            return {"action": "MRM_pull_over", "reason": "high_OOTLUF_risk"}
        return {"action": "alert_takeover", "budget_s": self.budget}

LLM agent guardrail (skill-atrophy aware)

class CodeReviewWithAutomationParadox:
    """Force human active review on high-stakes diff to prevent atrophy."""
    
    def __init__(self, llm_client):
        self.llm = llm_client
    
    def review(self, diff: str, stakes: str) -> str:
        ai_review = self.llm.review(diff)
        if stakes == "high":
            # require human to manually annotate before showing AI review
            human = prompt_for_independent_review(diff)
            return reconcile(human, ai_review)
        return ai_review

Skill maintenance schedule

def manual_practice_schedule(automation_uptime_pct: float) -> dict:
    """Recommend periodic manual mode to combat skill decay."""
    if automation_uptime_pct > 0.9:
        return {"frequency": "weekly", "duration_min": 30}
    if automation_uptime_pct > 0.7:
        return {"frequency": "biweekly", "duration_min": 20}
    return {"frequency": "monthly", "duration_min": 15}

매 결정 기준

상황	Approach
High-stakes + rare edge case	Keep human in loop, force periodic manual mode
Routine + low risk	Full automation OK
L2/L3 autonomy	Long handover budget (12s+), DMS (driver monitoring)
LLM agent	Active review on critical paths, not passive accept
HRO (high-reliability)	Multiple redundant operators, rotation

기본값: high-stakes automation 의 default — human-in-the-loop + periodic manual practice + miscalibration monitoring.

🔗 Graph

• 부모: Neuroergonomics · Complex Systems
• 변형: Neuromuscular-Control
• 응용: AI_Safety_and_Alignment · Multi-agent-System
• Adjacent: Burnout · Continuous Obsolescence

🤖 LLM 활용

언제: AI agent system design, code review automation, autonomous system handover protocol. 언제 X: stateless automation 의 인간 의 involvement 의 unnecessary 인 case.

❌ 안티패턴

• "Lights-out" fallacy: full automation 의 human 의 unnecessary 의 assume.
• 6-second handover budget: empirically insufficient — 12-15s baseline.
• Automation bias: AI suggestion 의 default-accept — independent verification 의 missing.
• Skill decay 의 ignore: emergency-only manual training 의 too late.

🧪 검증 / 중복

• Verified (Bainbridge 1983 Ironies of Automation; Parasuraman & Manzey 2010 Complacency and Bias).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — Bainbridge ironies, trust calibration, L3 handover, LLM agent guardrail