---
id: wiki-2026-0508-automation-paradox
title: Automation Paradox
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Paradox of Automation, Bainbridge's Ironies, Lights-Out Fallacy]
duplicate_of: none
source_trust_level: A
confidence_score: 0.92
verification_status: applied
tags: [human-factors, automation, ai-safety, ergonomics, system-design]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: n/a
  framework: human-factors / HRO
---

# Automation Paradox

## 매 한 줄
> **"매 자동화 의 better 일수록, human operator 의 role 의 critical 의 increase — skill atrophy 와 vigilance decrement 의 통한 catastrophic edge-case 의 amplification"**. Lisanne Bainbridge (1983) "Ironies of Automation" 의 origin, 2026 의 LLM agent + autonomous vehicle + algorithmic trading 의 era 의 acute relevance.

## 매 핵심

### 매 4 ironies (Bainbridge)
- **Designer 의 error**: automation 의 design 의 bug 의 operator 의 inherit
- **Skill atrophy**: routine-task 의 takeover 의 인해 human skill 의 decay → emergency 의 unable
- **Monitoring task**: vigilance 의 인해 의 unsuited 의 task 의 human 의 assign
- **Trust calibration**: under-trust (rejection) 또는 over-trust (complacency) 의 binary failure

### 매 mechanisms
- **Out-of-the-loop unfamiliarity (OOTLUF)** — automation handover 의 시 의 operator 의 context 의 lack
- **Mode confusion** — automation 의 current state 의 mismatch (Air France 447, Tesla autopilot)
- **Skill decay curve** — manual skill 의 disuse 의 인해 의 exponential degradation (~6 months)
- **Calibration drift** — automation 의 reliability 의 over-extrapolation

### 매 응용
1. Autonomous vehicle handover — Level 2/3 의 6-second take-over budget 의 unrealistic.
2. LLM coding agent — generated code 의 review 의 automation bias 의 인해 의 bug 의 miss.
3. Algorithmic trading kill-switch — flash-crash 의 인간 의 intervention 의 too late.
4. Aviation glass cockpit — Air France 447 (2009) 의 stall 의 mode confusion.

## 💻 패턴

### Trust calibration metric (TLX-derived)
```python
from dataclasses import dataclass

@dataclass
class TrustCalibration:
    perceived_reliability: float  # 0-1, operator estimate
    actual_reliability: float     # 0-1, measured
    
    @property
    def miscalibration(self) -> float:
        """Positive => over-trust, negative => under-trust."""
        return self.perceived_reliability - self.actual_reliability
    
    def risk_class(self) -> str:
        gap = self.miscalibration
        if gap > 0.15:  return "OVER_TRUST_DANGER"
        if gap < -0.15: return "REJECTION_DANGER"
        return "CALIBRATED"
```

### Vigilance decrement model
```python
import numpy as np

def vigilance_curve(t_minutes: np.ndarray, base_hit_rate: float = 0.95) -> np.ndarray:
    """Mackworth clock — 30-min decrement of ~0.2 in detection."""
    decay = 0.2 * (1 - np.exp(-t_minutes / 15))
    return np.clip(base_hit_rate - decay, 0, 1)

# Recommendation: rotate operators every 20 min on monitoring tasks
```

### Handover protocol (autonomous vehicle)
```python
class L3HandoverManager:
    def __init__(self, min_takeover_seconds: float = 12.0):
        self.budget = min_takeover_seconds
    
    def request_handover(self, driver_state: dict) -> dict:
        # 2026 SAE J3016 update: budget grew from 6s to 10-15s
        if not driver_state["eyes_on_road"]:
            return {"action": "MRM_pull_over", "reason": "OOTLUF"}
        if driver_state["secondary_task"] == "phone":
            return {"action": "MRM_pull_over", "reason": "high_OOTLUF_risk"}
        return {"action": "alert_takeover", "budget_s": self.budget}
```

### LLM agent guardrail (skill-atrophy aware)
```python
class CodeReviewWithAutomationParadox:
    """Force human active review on high-stakes diff to prevent atrophy."""
    
    def __init__(self, llm_client):
        self.llm = llm_client
    
    def review(self, diff: str, stakes: str) -> str:
        ai_review = self.llm.review(diff)
        if stakes == "high":
            # require human to manually annotate before showing AI review
            human = prompt_for_independent_review(diff)
            return reconcile(human, ai_review)
        return ai_review
```

### Skill maintenance schedule
```python
def manual_practice_schedule(automation_uptime_pct: float) -> dict:
    """Recommend periodic manual mode to combat skill decay."""
    if automation_uptime_pct > 0.9:
        return {"frequency": "weekly", "duration_min": 30}
    if automation_uptime_pct > 0.7:
        return {"frequency": "biweekly", "duration_min": 20}
    return {"frequency": "monthly", "duration_min": 15}
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| High-stakes + rare edge case | Keep human in loop, force periodic manual mode |
| Routine + low risk | Full automation OK |
| L2/L3 autonomy | Long handover budget (12s+), DMS (driver monitoring) |
| LLM agent | Active review on critical paths, not passive accept |
| HRO (high-reliability) | Multiple redundant operators, rotation |

**기본값**: high-stakes automation 의 default — human-in-the-loop + periodic manual practice + miscalibration monitoring.

## 🔗 Graph
- 부모: [[Neuroergonomics]] · [[Complex Systems]]
- 변형: [[Neuromuscular-Control]]
- 응용: [[AI_Safety_and_Alignment|AI Safety]] · [[Multi-agent-System]]
- Adjacent: [[Burnout]] · [[Continuous Obsolescence]]

## 🤖 LLM 활용
**언제**: AI agent system design, code review automation, autonomous system handover protocol.
**언제 X**: stateless automation 의 인간 의 involvement 의 unnecessary 인 case.

## ❌ 안티패턴
- **"Lights-out" fallacy**: full automation 의 human 의 unnecessary 의 assume.
- **6-second handover budget**: empirically insufficient — 12-15s baseline.
- **Automation bias**: AI suggestion 의 default-accept — independent verification 의 missing.
- **Skill decay 의 ignore**: emergency-only manual training 의 too late.

## 🧪 검증 / 중복
- Verified (Bainbridge 1983 *Ironies of Automation*; Parasuraman & Manzey 2010 *Complacency and Bias*).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — Bainbridge ironies, trust calibration, L3 handover, LLM agent guardrail |