The Devver Blog

A Boulder startup improving the way developers work.

Archive for the ‘Testing’ Category

Tips for Unit Testing

For the past few weeks, I’ve been doing a series of posts on my thoughts on unit testing. Although I originally published them in little, bite-sized posts, I wanted to collect them all here in one massive post for those of you with bigger reading appetites.

I also wanted to add one thought to sort of tie all these tips together. Unit testing is all about improving productivity. It’s important to realize that the ROI for testing looks something like this:

this graph is very exact

A very professional-looking graph. I guess ROI should really be ‘benefit’, but whatever, you get it.

If you are just getting started with unit testing, you’re at the bottom of the curve, so you’re going to sink a lot of time into testing without much benefit. Similarly, once you’ve done a lot of testing on a project, trying to test that last little bit may require more time than it’s worth. The goal of these tips is to help you maximize the benefit-to-time ratio, wherever you may be in this curve.


We’re big on automated testing here at Devver, but I know a lot of companies aren’t as into it. There’s been plenty written about all the reasons you should be writing tests, but over the next week or so, I’ll give you some tips on how to get started (and if you’ve already got some tests, how to improve and expand your test suite).

I can’t claim to have come up with these best practices, so I’ll litter this post with links to those resources that have taught me something.

A quick word about terminology. When I say “tests” I mean any type of automated tests, which may include unit, functional, integration or any other types of tests. When I say “production code” I simply mean the code goes into the actual product – i.e. the code being tested.

Tip 1: You’ll probably suck at testing
Writing tests can be frustrating at first. It is usually a lot harder and more time consuming than you’d expect. Unfortunately, some developers assume that the cost of writing tests is fixed and conclude that the benefits can’t possible justify the time spent – so they quit writing tests.

Writing test code is an art unto itself. There are a whole new set of tricks and skills to learn and it’s difficult to do correcty right away. Stick with it. The better you get, the faster you’ll write tests, and the more your tests will pay off.

Tip 2: Most code is not written to be tested
Another surprising thing you’ll find when you start testing is that your production code is not very testable. This isn’t surprising – if there were no tests previously, there was no reason to design for testability. This will make your first tests way harder to write and less valuable (i.e. they are less likely to catch real bugs)

There are a few tricks to get around this. First, try testing only new code or just test a smaller side project to start to get the hang of it. When you’re ready to start testing your legacy application, try the following.

1. Write a few very high-level tests. These tests will likely exercise almost the whole system and will interact with the application at the highest-level interface.
2. Refactor out one component of the application so it is more decoupled and testable
3. Continually run your high-level tests to make sure you haven’t broken anything major
4. Write more focused tests for the component you pulled out in step #2
5. Go back to step #2

If you need more help with this, pick up a copy of Working Effectively w/ Legacy Code. There is also some additional information here.

Again, stick with it. As you write more tests, your application will be more testable (bonus: it’s likely be easier to understand, more loosely coupled, easier to refactor, and more DRY as well!). As it becomes more testable, it’ll be easier to write additional tests. This creates a positive loop where things get better and easier as you go.

Tip 3: Test code isn’t production code
Another common mistake is to treat test code just like production code. For instance, you’d like your code production code to be as dry as possible. But in test code, it’s actually more important for tests to be readable and independent than to be dry. As a result, you’ll want your tests to be more “moist” than dry. Specifically, you’ll want to use literals a lot more in test code than you would in production.

In general, the most important properties of good tests are:

Independent – No test should affect the outcome of any other test. Put another way, you should be able to run your tests in any order and always have the same outcome. A corollary of this is that setup/teardown methods are evil (both because they increase dependence and they decrease readability)
Readable – The intent of each test should be immediately obvious (both by it’s name and by its code).
Fast – Each test should run as quickly as possible, so the entire suite is also fast. The faster the suite, the more you’ll run the tests, and the greater benefit you’ll get (because you’ll catch regressions quickly)
Precise – Each test should focus on testing one thing (and only one thing) well*. Ideally, if a test fails, you should know exactly what part of your production code broke by just glancing at the name of the test. Also, if your tests are precise, it’s less likely that a change in your code will require you to change many different tests. In practice, precise tests are short and only have one assertion or expectation per test.

*Note: this doesn’t apply to integration tests, which should make sure all components play nicely together.

Tip 4: Always write one test

When writing new code, it’s easy to avoid testing because it seems so daunting to test all the functionality. Rather than thinking of testing as an all-or-nothing proposition, try to write just one good test for the new functionality.

You’ll find that having just one test is much, much better than having no tests at all. Why? First of all, it’ll catch catastrophic errors, even if it doesn’t catch bugs in edge cases. Secondly, writing even one test may force you to refactor your production code slightly to make it more testable (which in turn, makes future tests easier to write). Finally, it gives you “test momentum”. If you have no tests, you’ll be inclined to delay testing, since there is more overhead to get started. But if you already have just one test in place, it’ll be much easier to add tests as you think of them (and to write regression tests as you find bugs).

By the way, don’t worry about testing at exactly the right level. Having one functional test is way better than having no tests at all. You can always come back and break the “bigger” test down into more targeted, precise tests.

Tip 5: Improve your tests over time

Here’s a terrible idea – decide you are going to spend a whole week building a test suite for your project. First of all, you’ll likely just get frustrated and burn out on testing. Secondly, you’ll probably write bad tests at first, so even if you get a bunch of tests written, you’re going to need to go back and rewrite them one you figure out how slow, brittle, or unreadable they are.

As they say, the best writing is rewriting. You should try out new techniques (and rewrite) old test code. But it’s OK to have patchwork tests.

You just found out fixtures suck? (they do). Or that those ‘setup’ methods make your tests less readable? Are you excited about using mocks? Great, apply your new technique to some new tests, rewrite a few old tests, and call it a day. Don’t try to rewrite your whole suite, because you’ll be kicking yourself when you rewrite your suite again after you decide technique X isn’t perfect in all cases.

Just like in production code, good practices take awhile to bake and prove themselves. See how maintainable, easy to understand, easy to read a new technique is. You can always move more tests over.

Tip 6: Don’t be dogmatic

There are a lot of best practices for testing that may or may not apply to your situation. Should you have one assertion per test? Should you use mocks and stubs? Should you use Test Driven Development? Or Behavior Driven Development? Should you do interaction or state-based testing? While all of these practices have real benefits, remember that their applicability and value depends largely on your project, schedule, and team.

Don’t be afraid to play, but don’t feel like you need to convert everything to the one, true way to test. It’s fine to have a suite that mixes and matches these best practices. In other words, context is king.

Tip 7: Be reasonable

There are lots of reasons why tests are great, but if your practices aren’t ultimately making your code better and you more productive, it’s not worth it. You have to always think about the return on your time investment.

There are domains in which automated testing is very difficult and doesn’t provide a lot of value, like GUI testing. I would recommend writing tests for the interface that the GUI calls, but actually testing that things show up correctly is quite tricky and error prone.

Also, 100% code coverage shouldn’t necessarily be your goal. As you get better at writing tests, I think you’ll find they provide a lot of value, but at some point, covering that last small percentage of code may require way more effort than it’s worth.

Tip 8: Keep learning!

Just like learning new programming languages makes you a better developer, learning about new testing approaches, libraries, and tools will make you a better tester. The state of the art of testing is changing very rapidly these days – new frameworks and techniques are released almost every month. Keep looking at example code and trying out new stuff.

For instance, here’s a few tools that you may not be using but are very cool: Heckle and RushCheck

Finally, if you want to learn more, subscribe to Jay Field’s blog – he has lots of good (if sometimes controversial) thoughts about testing.

And with that, I’ll wrap up this series on testing. If you have your own testing tips, please share them!


At Devver, we’re building some awesome, cloud-based tools for Ruby hackers. If you’re interested, sign up for our mailing list.

Written by Ben

July 7, 2008 at 4:31 pm

Posted in Testing

Tips for Testing: Tip #8

Tip 8: Keep learning!

Just like learning new programming languages makes you a better developer, learning about new testing approaches, libraries, and tools will make you a better tester. The state of the art of testing is changing very rapidly these days – new frameworks and techniques are released almost every month. Keep looking at example code and trying out new stuff.

For instance, here’s a few tools that you may not be using but are very cool: Heckle and RushCheck

Finally, if you want to learn more, subscribe to Jay Field’s blog – he has lots of good (if sometimes controversial) thoughts about testing.

And with that, I’ll wrap up this series on testing. If you have your own testing tips, please share them!

Written by Ben

July 2, 2008 at 6:46 am

Posted in Testing

Tips for Testing: Tip #7

Tip 7: Be reasonable

There are lots of reasons why tests are great, but if your practices aren’t ultimately making your code better and you more productive, it’s not worth it. You have to always think about the return on your time investment.

There are domains in which automated testing is very difficult and doesn’t provide a lot of value, like GUI testing. I would recommend writing tests for the interface that the GUI calls, but actually testing that things show up correctly is quite tricky and error prone.

Also, 100% code coverage shouldn’t necessarily be your goal. As you get better at writing tests, I think you’ll find they provide a lot of value, but at some point, covering that last small percentage of code may require way more effort than it’s worth.

Written by Ben

July 1, 2008 at 3:06 pm

Posted in Testing

Tips for Testing: Tip #6

Tip 6: Don’t be dogmatic

There are a lot of best practices for testing that may or may not apply to your situation. Should you have one assertion per test? Should you use mocks and stubs? Should you use Test Driven Development? Or Behavior Driven Development? Should you do interaction or state-based testing? While all of these practices have real benefits, remember that their applicability and value depends largely on your project, schedule, and team.

Don’t be afraid to play, but don’t feel like you need to convert everything to the one, true way to test. It’s fine to have a suite that mixes and matches these best practices. In other words, context is king.

Written by Ben

June 30, 2008 at 8:36 am

Posted in Testing

Tips for Testing: Tip #5

Tip 5: Improve your tests over time

Here’s a terrible idea – decide you are going to spend a whole week building a test suite for your project. First of all, you’ll likely just get frustrated and burn out on testing. Secondly, you’ll probably write bad tests at first, so even if you get a bunch of tests written, you’re going to need to go back and rewrite them one you figure out how slow, brittle, or unreadable they are.

As they say, the best writing is rewriting. You should try out new techniques (and rewrite) old test code. But it’s OK to have patchwork tests.

You just found out fixtures suck? (they do). Or that those ‘setup’ methods make your tests less readable? Are you excited about using mocks? Great, apply your new technique to some new tests, rewrite a few old tests, and call it a day. Don’t try to rewrite your whole suite, because you’ll be kicking yourself when you rewrite your suite again after you decide technique X isn’t perfect in all cases.

Just like in production code, good practices take awhile to bake and prove themselves. See how maintainable, easy to understand, easy to read a new technique is. You can always move more tests over.

Written by Ben

June 27, 2008 at 10:09 am

Posted in Testing

Tips for Testing: Tip #4

Tip 4: Always write one test

When writing new code, it’s easy to avoid testing because it seems so daunting to test all the functionality. Rather than thinking of testing as an all-or-nothing proposition, try to write just one good test for the new functionality.

You’ll find that having just one test is much, much better than having no tests at all. Why? First of all, it’ll catch catastrophic errors, even if it doesn’t catch bugs in edge cases. Secondly, writing even one test may force you to refactor your production code slightly to make it more testable (which in turn, makes future tests easier to write). Finally, it gives you “test momentum”. If you have no tests, you’ll be inclined to delay testing, since there is more overhead to get started. But if you already have just one test in place, it’ll be much easier to add tests as you think of them (and to write regression tests as you find bugs).

By the way, don’t worry about testing at exactly the right level. Having one functional test is way better than having no tests at all. You can always come back and break the “bigger” test down into more targeted, precise tests.

Written by Ben

June 26, 2008 at 8:09 am

Posted in Testing

Tips for Testing: Tip #3

Tip 3: Test code isn’t production code
Another common mistake is to treat test code just like production code. For instance, you’d like your code production code to be as dry as possible. But in test code, it’s actually more important for tests to be readable and independent than to be dry. As a result, you’ll want your tests to be more “moist” than dry. Specifically, you’ll want to use literals a lot more in test code than you would in production.

In general, the most important properties of good tests are:

Independent – No test should affect the outcome of any other test. Put another way, you should be able to run your tests in any order and always have the same outcome. A corollary of this is that setup/teardown methods are evil (both because they increase dependence and they decrease readability)
Readable – The intent of each test should be immediately obvious (both by it’s name and by its code).
Fast – Each test should run as quickly as possible, so the entire suite is also fast. The faster the suite, the more you’ll run the tests, and the greater benefit you’ll get (because you’ll catch regressions quickly)
Precise – Each test should focus on testing one thing (and only one thing) well*. Ideally, if a test fails, you should know exactly what part of your production code broke by just glancing at the name of the test. Also, if your tests are precise, it’s less likely that a change in your code will require you to change many different tests. In practice, precise tests are short and only have one assertion or expectation per test.

*Note: this doesn’t apply to integration tests, which should make sure all components play nicely together.

Written by Ben

June 25, 2008 at 11:50 am

Posted in Testing

Tips for Testing: Tip #2

Tip 2: Most code is not written to be tested
Another surprising thing you’ll find when you start testing is that your production code is not very testable. This isn’t surprising – if there were no tests previously, there was no reason to design for testability. This will make your first tests way harder to write and less valuable (i.e. they are less likely to catch real bugs)

There are a few tricks to get around this. First, try testing only new code or just test a smaller side project to start to get the hang of it. When you’re ready to start testing your legacy application, try the following.

1. Write a few very high-level tests. These tests will likely exercise almost the whole system and will interact with the application at the highest-level interface.
2. Refactor out one component of the application so it is more decoupled and testable
3. Continually run your high-level tests to make sure you haven’t broken anything major
4. Write more focused tests for the component you pulled out in step #2
5. Go back to step #2

If you need more help with this, pick up a copy of Working Effectively w/ Legacy Code. There is also some additional information here.

Again, stick with it. As you write more tests, your application will be more testable (bonus: it’s likely be easier to understand, more loosely coupled, easier to refactor, and more DRY as well!). As it becomes more testable, it’ll be easier to write additional tests. This creates a positive loop where things get better and easier as you go.

Written by Ben

June 24, 2008 at 7:29 am

Posted in Testing

Tips for Testing: Tip #1

We’re big on automated testing here at Devver, but I know a lot of companies aren’t as into it. There’s been plenty written about all the reasons you should be writing tests, but over the next week or so, I’ll give you some tips on how to get started (and if you’ve already got some tests, how to improve and expand your test suite).

I can’t claim to have come up with these best practices, so I’ll litter this post with links to those resources that have taught me something.

A quick word about terminology. When I say “tests” I mean any type of automated tests, which may include unit, functional, integration or any other types of tests. When I say “production code” I simply mean the code goes into the actual product – i.e. the code being tested.

Tip 1: You’ll probably suck at testing
Writing tests can be frustrating at first. It is usually a lot harder and more time consuming than you’d expect. Unfortunately, some developers assume that the cost of writing tests is fixed and conclude that the benefits can’t possible justify the time spent – so they quit writing tests.

Writing test code is an art unto itself. There are a whole new set of tricks and skills to learn and it’s difficult to do correcty right away. Stick with it. The better you get, the faster you’ll write tests, and the more your tests will pay off.

Written by Ben

June 23, 2008 at 9:35 am

Posted in Testing

Are we a band-aid?

Jay Fields recently wrote something interesting:

“Problems with tests are often handled by creating band-aids such as your own test case subclass that hides an underlying problem, testing frameworks that run tests in parallel, etc. To be clear, running tests in parallel is a good thing. However, if you have a long running build because of underlying issues and you solve it by running the tests in parallel.. that’s a band-aid, not a solution. The poorly written tests may take 10 minutes right now. If you run the tests in parallel it might take 2 minutes today, but when you are back to 10 minutes you now have ~5 times as many problematic tests. That’s not a good position to be in.”

This is really relevant to Devver since our first tool will be a easy-to-use, distributed unit test runner.

So, if you have long-running test suite, is using Devver a solution or a band-aid? It depends on the reason your tests take a long time to execute.

The key phrase Jay uses is “if you have a long running build because of underlying issues.” Clearly, in some cases, having a long test suite is justified. On my machine, the Rubinius suite takes about one minute to run. That’s not a bad thing – they have tons and tons of specs (5675 examples, 20924 expectations, to be exact).

Another example is Rails tests – your integration and functional tests may be slow because they hit the DB and the full Rails stack. It doesn’t make sense to speed them up using stubs, as you should in Rails unit tests, because the whole point of those tests it to make sure your entire application works together.

In these cases, your suite would truly provide less value if you cut or changed tests, but using a distributed test runner like Devver is a huge win – you get feedback much, much more quickly.

On the other hand, as Jay mentions, there are cases where running your tests in parallel might give you a temporary speed up, but ultimately you’re just hiding real problems with your suite. For instance, your unit tests might be too high-level or you might be depending on some slow external system that could be stubbed out.

Can Devver be both a band-aid and a solution?

Let’s say that you have a test suite that really has underlying problems. What can Devver do for you?

Well, in the short term, we can be that band-aid. As long as you realize that using Devver’s test runner isn’t a fix for your suite’s underlying issues, there’s nothing wrong with getting faster feedback.

However, I think we can do more. As time goes on, we can build Devver tools that actually help you fix the underlying problems by measuring the tracking the quality of your tests over time. For instance, in the future we could integrate with RCov and Heckle and track metrics like application LOC to test LOC ratio, test execution speed, and average number of assertions/expectations per test. If you have ideas as to what other metrics and tools might help you fix the quality of your tests, let us know.

Written by Ben

June 19, 2008 at 3:39 pm

Posted in Devver, Ruby, Testing

Follow

Get every new post delivered to your Inbox.