Mutation testing is an advanced testing technique that evaluates the effectiveness of your test suite by making small changes (mutations) to your source code and checking if your tests can detect these changes. Traditional code coverage tells you whether your code ran during tests. Mutation testing tells you whether your tests would actually catch a bug.
How Mutation Testing Works
The process of mutation testing follows these steps:
- Create Mutants: The mutation testing tool automatically creates "mutants" by making small, controlled changes to your source code (like changing a "+" to a "-" or a ">" to a ">=")
- Run Tests Against Mutants: The tool runs your existing test suite against each mutant
- Analyze Results: If your tests fail when run against a mutant, the mutant is considered "killed" (good). If the tests still pass, the mutant "survived" (bad, indicates a weakness in your tests)
Mutation Score
The primary metric in mutation testing is the mutation score:
Mutation Score = (Number of Killed Mutants / Total Number of Mutants) × 100%A higher mutation score indicates a more effective test suite that's better at detecting potential bugs.
Common Types of Mutations
Mutation testing tools apply various types of mutations, including:
- Statement Deletion: Remove a statement entirely
- Operator Replacement: Change arithmetic, logical, or comparison operators
- Constant Replacement: Replace constants with different values
- Conditional Boundary Changes: Modify boundary conditions in if statements
- Return Value Modifications: Change the values returned from functions
Benefits of Mutation Testing
While more resource-intensive than basic code coverage, mutation testing offers real benefits:
- Identifies gaps in test assertions that simple execution coverage misses
- Encourages writing more meaningful tests that assert correct behavior
- Highlights which parts of your code lack thorough testing
- Provides a more accurate measure of test quality than simple code coverage
Popular Mutation Testing Tools
Some widely used mutation testing tools include:
- For PHP: Infection, Humbug
- For JavaScript: Stryker
- For Java: PIT
- For Python: Mutmut, Cosmic Ray
Code coverage shows whether your tests ran. Mutation testing shows whether they'd catch a bug. Used together, they give a fuller picture of test quality than either metric alone.
