We all know the feeling. Writing tests for hours, running them and they all went red. Disappointment, debug the tests , start all over again. But what if the test is successful if it is failing, what if the color red make us happy?
This is what mutation testing is. Although the goal is not to make our test red but a failing test is a successful test from the point of Mutation testing. So what is it really?
Mutation testing gives us some insights in not only the line coverage but also the quality of our tests. In other words it takes care of the edge cases in our tests. For example we have a function: function isAdult(user) { return user.age >= 18;} We write a test to assert if the the age is 19, run the code, test succeed and we have 100% coverage. Great work !
Then one of our developers is going to refactor some other part of the code and replace all the >= operators by > so the code we just tested is changed. New code, so we run our tests again, 19 is still above 18, test passes and 100% coverage. Great work again !
But wait, the code change and our test is still successful ? Did our test notice the change or is it always passing ? Line coverage is 100% but are we sure that we cover other situations as well? In other words is the quality of the test as good as we expected?
I this case we only have a single line of code, but imagine an application with 1000 lines or even a million lines. In such cases Mutation testing makes our life a little bit easier by examine if the testcode we use is able to detect softwarechanges or in other words is the quality of our test code that what we expect.
Mutation testing looks through the code ( in memory not the real source code ) and find operator like greater then, less then and so on. It can detect other operators too, but later more. For each operator it found it makes a mutant. A mutant is a change of the operator. For example it change the > in a >= , = or < and run the test again. If the tests fail, the mutant is found and we call it we killed the mutant. If the test passed the mutant survived. A mutant which will survive is a gap in our tests. The test didn’t find the change in the code, so we can say that the quality of our testcode is not at the level we want. So we have to change our tests or make more tests.
Reading articles about Mutation testing is a nice way to get some impression of what is possible with this kind of testing. But how does it really work? Is it easy to setup? How do I use it in my projects? An answer to those kind of questions is trying it myself.
Last week I was at the Testnet Najaarsevent in Nieuwegein ( Netherlands). One of the workshops was about Mutation testing. Bas Dijkstra, a well known speaker and Test Automation Expert, gave a nice workshop about mutation testing. A little theory and a lot of hands on, the ideal way to start.
After a few hours of listening, doing and struggling with the code (Bas used the Java variant : PiTest ) the overall picture is clear. Since I am not very familiar with Java and mainly work with JavaScript applications at this moment I choose for this series the JavaScript ( C# and Scala) variant called Stryker.