If you’ve just changed or created a test the fact it passes doesn’t mean SH*T…
At this stage you’ve no idea whether you’ve done something stupid in your test and are getting a false negative and your code really does have a bug. A lot of developers seem to forget that tests are still code and they can and do regularly contain bugs just like regular app code… no matter how simple they are.
This is why we need to…
1) treat tests like regular app code and include them in review processes like pair programming, pull-requests, code reviews, static analysis etc.
and…
2) change tests as little as possible. A test is like a fine wine, its gets better with age… assuming you don’t touch it 😂. Ideally we want to be able to do as much refactoring (which we can consider everything except changing the signature/API of a method under test) to our code as possible without having to change our tests.
To support this we need to keep tests high level and focused on behaviour, inputs and outputs.. and not the specifics of how something is implemented. In practice this means things like not directly testing sub methods, private methods, lower level methods etc.
A great talk on this topic which I’ve shared before a couple of times is below…
Consider mutation testing
Mutation testing frameworks like Pitest or Stryker change the production code of your app during testing to introduce bugs. The framework will then have a look whether your tests catch these bugs, i.e. a test should fail if the production code has bugs. This will let you know whether your tests catch the bugs in the production code. Essentially mutation testing tests your tests.
Original topic on LinkedIn
I originally posted a variation of this post on LinkedIn. There’s over 60 comments on the thread with many different helpful views.
