2nd/10_Wiki/Topics/Coding/Testing_Stryker_Mutation.md

id, title, category, status, source_trust_level, verification_status, created_at, updated_at, tags, tech_stack, applied_in, aliases

id	title	category	status	source_trust_level	verification_status	created_at	updated_at	tags	tech_stack	applied_in	aliases
testing-stryker-mutation	Mutation Testing — Stryker / PIT / Mutmut	Coding	draft	B	conceptual	2026-05-09	2026-05-09	testing mutation vibe-coding	language applicable_to TS / Java / Python Testing		mutation testing Stryker PIT Mutmut mutation score killed mutant survived mutant

Mutation Testing

Code coverage 100% ≠ test 가 좋다. Mutation: 코드 의 작은 변경 → test 가 잡으면 OK. Stryker (JS/TS), PIT (Java), Mutmut (Python).

📖 핵심 개념

• Coverage: 실행된 line 비율.
• Mutation: 의도적 bug 주입 후 test pass = bad test.
• Killed mutant: test 가 잡음.
• Survived: test 가 못 잡음 → bad test.

💻 코드 패턴

Stryker (JS / TS)

yarn add -D @stryker-mutator/core @stryker-mutator/jest-runner

// stryker.conf.js
module.exports = {
  testRunner: 'jest',
  coverageAnalysis: 'perTest',
  mutate: ['src/**/*.ts', '!src/**/*.test.ts'],
  thresholds: { high: 80, low: 60, break: 50 },
};

yarn stryker run

Output

Mutation testing report
=======================
Files: src/utils/math.ts
Mutants tested: 25
- Killed:    18 (72%)
- Survived:   5 (20%)
- Timeout:    1
- No coverage: 1
- Total time: 45 sec

Mutation score: 72% (warning, threshold 80%)

Mutator 종류

// Original
if (x > 5) return 'big';

// Mutants:
if (x > 5) return 'big';     // ❌ → if (x >= 5)
if (x > 5) return 'big';     // ❌ → if (x < 5)
if (x > 5) return 'big';     // ❌ → if (true)
if (x > 5) return 'big';     // ❌ → return null

→ 매 statement 가 N variant. Test 가 매 variant 잡아야.

Stryker mutators

- Arithmetic: + → -, * → /
- Boolean: true → false
- Equality: === → !==
- Comparison: > → >=, < → <=
- Conditional: if → if (true)
- Logical: && → ||
- Update: i++ → i--
- Block: return X → return null
- ...

Survived mutant 의 의미

// src/math.ts
export function add(a: number, b: number) {
  return a + b;
}

// test
test('add', () => {
  expect(add(2, 2)).toBe(4);
});

// Stryker mutates: a + b → a - b
// Test: 2 - 2 = 0, not 4 → killed ✓

// Mutates: a + b → a * b
// Test: 2 * 2 = 4 → SURVIVED ❌ (test 가 multiplication 도 통과)

→ 더 다양 input 필요: expect(add(2, 3)).toBe(5).

Coverage vs mutation score

Coverage 100% + Mutation 30% = test 가 부드러움.
- "Code 가 실행" 만 검증
- "Code 가 정확" 검증 X

→ Mutation score 가 진짜 quality.

Performance

Stryker = N x test 시간 (N = mutant 수).
1000 mutant × 10s test = 10000s (3 hour).

→ Incremental + parallel.

{
  incremental: true,    // 변경 file 만
  concurrency: 4,
}

CI integration

- run: yarn stryker run --incremental
- run: |
    if [[ $MUTATION_SCORE -lt 70 ]]; then
      echo "Mutation score too low"
      exit 1
    fi

→ 매 PR (incremental) 검증.

Threshold

break: 60   — 미만 = CI fail
low: 70     — warning
high: 80    — green

매 PR 기준: 변경 file 의 mutation score.

Equivalent mutant

// Original
for (let i = 0; i < n; i++) { ... }

// Mutant
for (let i = 0; i <= n - 1; i++) { ... }   // 같은 동작

→ Mutant 가 same behavior = test 가 못 잡아도 OK. False positive. Manually mark.

Mutation 의 한계

- 큰 codebase = 매우 느림.
- Equivalent mutant.
- Configuration overhead.
- Sometimes test 보다 코드 변경 가 더 단순.

→ Critical code 만 mutation test.
일반 = unit test + coverage 충분.

When?

✓ Critical algorithm (math, encryption, finance)
✓ Library author (모든 use case)
✓ Test suite quality 검증
✓ Onboarding (신규 사람 가 test quality 봄)

✗ UI / E2E
✗ Glue code
✗ 매일 (cost)

Subset (file 별)

yarn stryker run --mutate "src/critical/**/*.ts"

→ 핵심만 매주. 전체 = 매월.

PIT (Java)

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <configuration>
    <targetClasses><param>com.example.*</param></targetClasses>
    <mutationThreshold>80</mutationThreshold>
  </configuration>
</plugin>

mvn org.pitest:pitest-maven:mutationCoverage

Mutmut (Python)

pip install mutmut
mutmut run --paths-to-mutate src/
mutmut results

Cosmic Ray (Python alt)

cosmic-ray init config.toml session
cosmic-ray exec session
cosmic-ray dump session > result.json

Mutation testing 의 비판

"Test 의 quality > 정량 measure".
-1 character variation 이 의미 가 다양:
- "잘못 잡음" 가 useless test 의미일 수.
- "잡음" 가 over-specification 일 수.

→ 100% mutation = brittle test.

→ Sweet spot: 70-80%.

Practical workflow

1. Coverage 80%+ 먼저.
2. Mutation 가 baseline 측정.
3. Survived mutant 분석 → 매 mutant 가:
   - 진짜 missing test? → 추가.
   - Equivalent? → mark + ignore.
   - Over-test 가능? → review.
4. Score 가 80%+ 가 목표.

IDE integration

- Stryker VS Code extension
- Survived mutant 가 file 에 highlight
- 매 mutation 의 specific change visible

Visual report

Stryker → HTML report:
- File 별 mutation score.
- Survived mutant 의 정확 line.
- Test 가 호출 한 mutant 추적.

Combined with property-based test

import { fc } from 'fast-check';

test('add commutative', () => {
  fc.assert(fc.property(fc.integer(), fc.integer(), (a, b) => {
    expect(add(a, b)).toBe(add(b, a));
  }));
});

→ Property test + mutation = 매우 강.

LLM 활용

Survived mutant 가 LLM 에:
"Test add should kill: a + b → a * b. 새 test case 작성"

→ Test gap 가 LLM 가 자동 채움.

Cost analysis

1 시간 = $2 (CI).
매월 10 hour = $20.
→ Bug 가 prod 에 1 = $1000s.

→ Critical code 면 가치.

🤔 의사결정 기준

상황	추천
Critical algorithm	Mutation
Library author	Mutation
일반 app	Coverage 만
큰 codebase	Subset / incremental
정확 검증	Property + mutation
CI gate	Threshold 70-80%
작은 PR	Incremental

❌ 안티패턴

• Coverage 100% = 좋아 가정: mutation 가 진실.
• 모든 file mutation: 너무 느림.
• Equivalent mutant 무시: false positive.
• 매일 mutation: cost 폭발.
• Threshold 100%: brittle.
• Survived 안 분석: 의미 없음.
• 만 mutation, no coverage: 기본 안 됨.

🤖 LLM 활용 힌트

• Mutation > coverage 의 quality measure.
• Stryker (JS), PIT (Java), Mutmut (Python).
• Critical code 만 — full = 비용.
• Property + mutation 가 강력 조합.

🔗 관련 문서

• Testing_Test_Pyramid
• Testing_Property_Based
• Testing_Mocking_Boundaries