Are we a band-aid?
Jay Fields recently wrote something interesting:
“Problems with tests are often handled by creating band-aids such as your own test case subclass that hides an underlying problem, testing frameworks that run tests in parallel, etc. To be clear, running tests in parallel is a good thing. However, if you have a long running build because of underlying issues and you solve it by running the tests in parallel.. that’s a band-aid, not a solution. The poorly written tests may take 10 minutes right now. If you run the tests in parallel it might take 2 minutes today, but when you are back to 10 minutes you now have ~5 times as many problematic tests. That’s not a good position to be in.”
This is really relevant to Devver since our first tool will be a easy-to-use, distributed unit test runner.
So, if you have long-running test suite, is using Devver a solution or a band-aid? It depends on the reason your tests take a long time to execute.
The key phrase Jay uses is “if you have a long running build because of underlying issues.” Clearly, in some cases, having a long test suite is justified. On my machine, the Rubinius suite takes about one minute to run. That’s not a bad thing – they have tons and tons of specs (5675 examples, 20924 expectations, to be exact).
Another example is Rails tests – your integration and functional tests may be slow because they hit the DB and the full Rails stack. It doesn’t make sense to speed them up using stubs, as you should in Rails unit tests, because the whole point of those tests it to make sure your entire application works together.
In these cases, your suite would truly provide less value if you cut or changed tests, but using a distributed test runner like Devver is a huge win – you get feedback much, much more quickly.
On the other hand, as Jay mentions, there are cases where running your tests in parallel might give you a temporary speed up, but ultimately you’re just hiding real problems with your suite. For instance, your unit tests might be too high-level or you might be depending on some slow external system that could be stubbed out.
Can Devver be both a band-aid and a solution?
Let’s say that you have a test suite that really has underlying problems. What can Devver do for you?
Well, in the short term, we can be that band-aid. As long as you realize that using Devver’s test runner isn’t a fix for your suite’s underlying issues, there’s nothing wrong with getting faster feedback.
However, I think we can do more. As time goes on, we can build Devver tools that actually help you fix the underlying problems by measuring the tracking the quality of your tests over time. For instance, in the future we could integrate with RCov and Heckle and track metrics like application LOC to test LOC ratio, test execution speed, and average number of assertions/expectations per test. If you have ideas as to what other metrics and tools might help you fix the quality of your tests, let us know.