--- id: testing-stryker-mutation title: Mutation Testing — Stryker / PIT / Mutmut category: Coding status: draft source_trust_level: B verification_status: conceptual created_at: 2026-05-09 updated_at: 2026-05-09 tags: [testing, mutation, vibe-coding] tech_stack: { language: "TS / Java / Python", applicable_to: ["Testing"] } applied_in: [] aliases: [mutation testing, Stryker, PIT, Mutmut, mutation score, killed mutant, survived mutant] --- # Mutation Testing > Code coverage 100% ≠ test 가 좋다. **Mutation: 코드 의 작은 변경 → test 가 잡으면 OK**. Stryker (JS/TS), PIT (Java), Mutmut (Python). ## 📖 핵심 개념 - Coverage: 실행된 line 비율. - Mutation: 의도적 bug 주입 후 test pass = bad test. - Killed mutant: test 가 잡음. - Survived: test 가 못 잡음 → bad test. ## 💻 코드 패턴 ### Stryker (JS / TS) ```bash yarn add -D @stryker-mutator/core @stryker-mutator/jest-runner ``` ```js // stryker.conf.js module.exports = { testRunner: 'jest', coverageAnalysis: 'perTest', mutate: ['src/**/*.ts', '!src/**/*.test.ts'], thresholds: { high: 80, low: 60, break: 50 }, }; ``` ```bash yarn stryker run ``` ### Output ``` Mutation testing report ======================= Files: src/utils/math.ts Mutants tested: 25 - Killed: 18 (72%) - Survived: 5 (20%) - Timeout: 1 - No coverage: 1 - Total time: 45 sec Mutation score: 72% (warning, threshold 80%) ``` ### Mutator 종류 ```ts // Original if (x > 5) return 'big'; // Mutants: if (x > 5) return 'big'; // ❌ → if (x >= 5) if (x > 5) return 'big'; // ❌ → if (x < 5) if (x > 5) return 'big'; // ❌ → if (true) if (x > 5) return 'big'; // ❌ → return null ``` → 매 statement 가 N variant. Test 가 매 variant 잡아야. ### Stryker mutators ``` - Arithmetic: + → -, * → / - Boolean: true → false - Equality: === → !== - Comparison: > → >=, < → <= - Conditional: if → if (true) - Logical: && → || - Update: i++ → i-- - Block: return X → return null - ... ``` ### Survived mutant 의 의미 ```ts // src/math.ts export function add(a: number, b: number) { return a + b; } // test test('add', () => { expect(add(2, 2)).toBe(4); }); // Stryker mutates: a + b → a - b // Test: 2 - 2 = 0, not 4 → killed ✓ // Mutates: a + b → a * b // Test: 2 * 2 = 4 → SURVIVED ❌ (test 가 multiplication 도 통과) ``` → 더 다양 input 필요: `expect(add(2, 3)).toBe(5)`. ### Coverage vs mutation score ``` Coverage 100% + Mutation 30% = test 가 부드러움. - "Code 가 실행" 만 검증 - "Code 가 정확" 검증 X → Mutation score 가 진짜 quality. ``` ### Performance ``` Stryker = N x test 시간 (N = mutant 수). 1000 mutant × 10s test = 10000s (3 hour). → Incremental + parallel. ``` ```js { incremental: true, // 변경 file 만 concurrency: 4, } ``` ### CI integration ```yaml - run: yarn stryker run --incremental - run: | if [[ $MUTATION_SCORE -lt 70 ]]; then echo "Mutation score too low" exit 1 fi ``` → 매 PR (incremental) 검증. ### Threshold ``` break: 60 — 미만 = CI fail low: 70 — warning high: 80 — green 매 PR 기준: 변경 file 의 mutation score. ``` ### Equivalent mutant ```ts // Original for (let i = 0; i < n; i++) { ... } // Mutant for (let i = 0; i <= n - 1; i++) { ... } // 같은 동작 ``` → Mutant 가 same behavior = test 가 못 잡아도 OK. False positive. Manually mark. ### Mutation 의 한계 ``` - 큰 codebase = 매우 느림. - Equivalent mutant. - Configuration overhead. - Sometimes test 보다 코드 변경 가 더 단순. → Critical code 만 mutation test. 일반 = unit test + coverage 충분. ``` ### When? ``` ✓ Critical algorithm (math, encryption, finance) ✓ Library author (모든 use case) ✓ Test suite quality 검증 ✓ Onboarding (신규 사람 가 test quality 봄) ✗ UI / E2E ✗ Glue code ✗ 매일 (cost) ``` ### Subset (file 별) ```bash yarn stryker run --mutate "src/critical/**/*.ts" ``` → 핵심만 매주. 전체 = 매월. ### PIT (Java) ```xml org.pitest pitest-maven com.example.* 80 ``` ```bash mvn org.pitest:pitest-maven:mutationCoverage ``` ### Mutmut (Python) ```bash pip install mutmut mutmut run --paths-to-mutate src/ mutmut results ``` ### Cosmic Ray (Python alt) ```bash cosmic-ray init config.toml session cosmic-ray exec session cosmic-ray dump session > result.json ``` ### Mutation testing 의 비판 ``` "Test 의 quality > 정량 measure". -1 character variation 이 의미 가 다양: - "잘못 잡음" 가 useless test 의미일 수. - "잡음" 가 over-specification 일 수. → 100% mutation = brittle test. ``` → Sweet spot: 70-80%. ### Practical workflow ``` 1. Coverage 80%+ 먼저. 2. Mutation 가 baseline 측정. 3. Survived mutant 분석 → 매 mutant 가: - 진짜 missing test? → 추가. - Equivalent? → mark + ignore. - Over-test 가능? → review. 4. Score 가 80%+ 가 목표. ``` ### IDE integration ``` - Stryker VS Code extension - Survived mutant 가 file 에 highlight - 매 mutation 의 specific change visible ``` ### Visual report ``` Stryker → HTML report: - File 별 mutation score. - Survived mutant 의 정확 line. - Test 가 호출 한 mutant 추적. ``` ### Combined with property-based test ```ts import { fc } from 'fast-check'; test('add commutative', () => { fc.assert(fc.property(fc.integer(), fc.integer(), (a, b) => { expect(add(a, b)).toBe(add(b, a)); })); }); ``` → Property test + mutation = 매우 강. ### LLM 활용 ``` Survived mutant 가 LLM 에: "Test add should kill: a + b → a * b. 새 test case 작성" → Test gap 가 LLM 가 자동 채움. ``` ### Cost analysis ``` 1 시간 = $2 (CI). 매월 10 hour = $20. → Bug 가 prod 에 1 = $1000s. → Critical code 면 가치. ``` ## 🤔 의사결정 기준 | 상황 | 추천 | |---|---| | Critical algorithm | Mutation | | Library author | Mutation | | 일반 app | Coverage 만 | | 큰 codebase | Subset / incremental | | 정확 검증 | Property + mutation | | CI gate | Threshold 70-80% | | 작은 PR | Incremental | ## ❌ 안티패턴 - **Coverage 100% = 좋아 가정**: mutation 가 진실. - **모든 file mutation**: 너무 느림. - **Equivalent mutant 무시**: false positive. - **매일 mutation**: cost 폭발. - **Threshold 100%**: brittle. - **Survived 안 분석**: 의미 없음. - **만 mutation, no coverage**: 기본 안 됨. ## 🤖 LLM 활용 힌트 - Mutation > coverage 의 quality measure. - Stryker (JS), PIT (Java), Mutmut (Python). - Critical code 만 — full = 비용. - Property + mutation 가 강력 조합. ## 🔗 관련 문서 - [[Testing_Test_Pyramid]] - [[Testing_Property_Based]] - [[Testing_Mocking_Boundaries]]