The Devver Blog

A Boulder startup improving the way developers work.

Shutting down the Devver Test Accelerator

After much deliberation, we’ve decided to shut down our Test Accelerator product on January 22nd, 2010. (the Accelerator was previously just known as “Devver”, but to clarify between our old product and Caliper, we’re referring to it as the “Accelerator” or the “Test Accelerator”).

This has been a difficult decision for us. Understandably, some people have been confused and disappointed by this news, so I wanted to take some time to explain our decision.

For over a year, we worked to build our vision of accelerated Ruby testing in the cloud. We made our share of mistakes and learned a lot in the process. Ultimately, we found the demand was too small to warrant overcoming all the technical challenges that were harder than we anticipated.

We had many different technical challenges, but it boils down to this: we underestimated the difficulty of making it simple to run Ruby/Rails tests in the cloud.

When we began working on the Accelerator, we knew we would need many features: automatic installation of gems, automatic setup of common databases, and support for Test:Unit, Shoulda, and RSpec. However, other features caught us by surprise. For instance, many projects required running daemons, such as memcached or Solr to be running when the tests executed. Also, it was common for projects to work fine on a user’s machine, but fail in the cloud due to undeclared RubyGem dependencies (or dependencies that lacked sufficient version information). Some problems were extra difficult because, due to costs, we needed to be able to run many projects on a single EC2 instance (and it’s currently not possible to create virtual machines within EC2 Xen instances).

We worked hard to overcome these challenges, but it was still too common for users to discover that running tests in the cloud required additional work on their part (or, in some cases, that their project was not supported). And even worse, users had to jump through these hoops before they could assess whether or not the Accelerator would speed up their tests enough to make the process worth it. Because the potential benefit and possible configuration time were both unknown, many users understandably gave up before they experienced the time savings of running tests in parallel. And unfortunately, the projects that were most likely to need acceleration (those with large test suites), were also the most likely to have complex dependencies and require the most setup work.

While we may have been able to solve these problems with a lot of additional engineering, we also found that only a small percentage of teams really needed test acceleration. Some teams loved our solution, because they had test suites that took 10, 20, or even 45 minutes to run. However, most projects had smaller test suites. We found that many teams didn’t mind waiting for suites that took 3-5 minutes (or, more accurately, didn’t mind enough to justify the considerable time investment of setting up tests on the Accelerator).

So, after much discussion, we decided that we needed to create a product that would help a larger set of Ruby teams and require significantly less setup to provide value. Those discussions led to Caliper, a tool to identify risky code and technical debt in Ruby code.

Focus is key in a startup, especially a small one like ours. While we considered keeping the Accelerator running in maintenance mode, we realized we could not provide the level of support our users deserve while executing effectively on Caliper. As a result, we’ve decided to shut down the Accelerator indefinitely.

We deeply appreciate each and every person who helped us with the Accelerator, either by using it, or giving us ideas, feedback, and encouragement. We could not have gotten as far as we did without the help from users, friends, and mentors.

It pains us to shut down the service and we apologize for any inconvenience this causes our users. We hope that you understand our decision and we look forward to helping you improve your code with Caliper.

(Accelerator users: if you need to speed up your tests, here are some possible alternatives to Devver: multitest, parallel_specs, Testjour, and DeepTest. I don’t have experience with these tools, so I can’t comment on how well they work myself, but you may want to check them out.)

Written by Ben

January 6, 2010 at 2:17 pm

Posted in Devver

Tagged with , , ,

4 Responses

Subscribe to comments with RSS.

  1. Thanks for the shoutout on multitest! Multitest is still quite young but we use it on a few of our current projects.

    Will you be open-sourcing the Test Accelerator code? It's something we'd be interested in setting up internally.

    -Nick Gauthier
    SmartLogic Solutions

    Nick Gauthier

    January 6, 2010 at 2:32 pm

  2. Hey Ben, this type of decision can be very tough to make. I hope this pivot turns out to be a winner for Devver. Good luck with the new direction guys!

    John Trupiano

    January 6, 2010 at 2:38 pm

  3. Sorry, but we aren't planning on open-sourcing that work in the near future because part or all of the technology may eventually return in some form.

    bhb

    January 6, 2010 at 3:55 pm

  4. Thanks!

    bhb

    January 6, 2010 at 3:56 pm


Comments are closed.

%d bloggers like this: