The Devver Blog

A Boulder startup improving the way developers work.

Posts Tagged ‘lessons

Lessons Learned

As we’ve begun to wrap things up here at Devver, we’ve had the chance to reflect a bit on our experience. Although shutting down is not the outcome we wanted, it’s clear to both Dan and I that doing a startup has been an amazing learning experience. While we still have a lot to learn, we wanted to share some of the most important lessons we’ve learned during this time.

The community

When we started Devver, we were hesitant to ask for feedback and help. We quickly found that people are incredibly helpful and generous with their time. Users were willing to take a chance and use our products while giving us valuable feedback. Fellow Rubyists gave us ideas and helped us with technical problems. Mentors made time for meetings and introduced us to others who could assist us. And other entrepreneurs, both new and seasoned, were happy to share stories, compare experiences, and offer support.

If you are working on a startup, don’t be afraid to ask for help! The vast majority of people want to help you succeed, provided that you respect them and their time. That means you need to prepare adequately (do your research and ask good questions), figure out their preferred methods of communication (e.g. don’t call if they prefer email), show up on time, don’t overburden them, and thank them. And when other people need your help, give back!

You can build awesome relationships with various communities on your own, but we strongly recommend joining a mentorship program like TechStars. The program accelerated the process of connecting with mentors, users, and other entrepreneurs by providing an amazing community during the summer (and to this day). The advice, introductions, and support have been simply incredible.

Founding team

Dan and I are both technical founders. Looking back, it would have been to our advantage to have a third founder who really loved the business aspect of running a startup.

There is a belief (among technical founders) that technical founders are sufficient for a successful startup. Or, put more harshly, that you can teach a hacker business, but you can’t teach a businessman how to hack“. I don’t want to argue whether that’s true or not. Clearly there are examples of technical founders being sufficient to get a company going, but my point is that having solely technical founders is non-optimal. You can teach a hacker business, but you can’t make him or her get excited about it, which means it may not get the time or attention it deserves.

Hackers are passionate about, well, hacking. And so we tend to measure progress in terms of features completed or lines of code written. Clearly, code needs to be written, but ideally a startup would have a founder who is working on important non-technical tasks: talking with customers, measuring key metrics, developing distribution channels, etc. I’m not advocating that only one founder works on these tasks while technical founders ignore customer development – everyone needs to get involved. Rather, I’m pointing out that given a choice, technical founders will tend to solve problems technically and having a founder who has the opposite default is valuable.

Remote teams

We embraced working remotely: we hired Avdi to work in Pennsylvania while Dan and I lived in Boulder and later on, Dan moved to Washington, DC. There are many benefits to having a distributed team, but two stood out in our experience. First, we could hire top talent without having to worry about location (in fact, our flexibility regarding location was very attractive to most candidates we interviewed). Secondly, being in different locations allowed every team member to work with minimal distractions, which is invaluable when it comes to efficiently writing good code.

That said, communication was a challenge. To ensure we were all synced up, we had a daily standup as well as a weekly review. When Dan moved to DC, he and I scheduled another weekly meeting with no set agenda to just bring up all the issues, large and small, that were on our minds. We also all got together in the same location every few months to work in the same room and rekindle our team energy.

Also, pair programming was difficult to do remotely and we never came up with a great solution. As a result, we spent less than a day pairing a week on average.

The most significant drawback to a remote team is the administrative hassle. It’s a pain to manage payroll, unemployment, insurance, etc in one state. It’s a freaking nightmare to manage in three states (well, two states and a district), even though we paid a payroll service to take care of it. Apparently, once your startup gets larger, there are companies that will manage this with minimal hassle, but for a small team, it was a major annoyance and distraction.

Product development

Most of the mistakes we made developing our test accelerator and, later, Caliper boiled down to one thing: we should have focused more on customer development and finding a minimum viable product (MVP).

The first thing we worked on was our Ruby test accelerator. At the time, we thought we had found our MVP: we had made encouraging technical progress and we had talked to several potential customers who were excited about the product we were building. Anything simpler seems “too simple” to be interesting.

Our mistake at that point was to go “heads down” and focus on building the accelerator while minimizing our contact with users and customers (after all, we knew how great it was and time spent talking to customers was time we could be hacking!). We should have asking, “Is there an even simpler version of this product that we can deliver sooner to learn more about pricing, market size, and technical challenges?”

If we had done so, we would have discovered:

  • whether the need was great enough (and if the solution was good enough) to convince people to open their wallets
  • that while a few users acutely felt the pain of slow tests, most didn’t care about acceleration. However, many of those users did want a “simpler” application – non-accelerated Ruby cloud testing.
  • the primary technical challenge was not accelerating tests, it was configuring servers for customers’ Rails applications. Not only did we spend time focusing on the wrong technical challenges, we also made architectural decisions that actually made it harder to solve this core problem.

After eventually discovering that setup and configuration was our primary adoption problem (and after trying and failing to implement various strategies to make it simple and easy), we tried to move to the other end of the spectrum. Caliper was designed to provide value with zero setup or configuration – users just provided a link to source code and instantly got valuable data.

Unfortunately, we again made the mistake of focusing on engineering first and customer development second. We released our first version to some moderate success and then proceeded to continue to churn out features without really understanding customer needs. Only later on, after finally engaging potential customers did we realize that market was too small and price point was to low to have Caliper sustain our company by itself.


This is by no means a comprehensive list, but it is our hope that other startups and founders-to-be can learn from our experiences, both mistakes and successes. Doing a startup has been an incredible learning experience for both Dan and I and we look forward to learning more in the future – both first-hand and from the amazing group of entrepreneurs and hackers that we’ve been privileged enough to know.


Written by Ben

April 26, 2010 at 11:04 am

Shutting down the Devver Test Accelerator

After much deliberation, we’ve decided to shut down our Test Accelerator product on January 22nd, 2010. (the Accelerator was previously just known as “Devver”, but to clarify between our old product and Caliper, we’re referring to it as the “Accelerator” or the “Test Accelerator”).

This has been a difficult decision for us. Understandably, some people have been confused and disappointed by this news, so I wanted to take some time to explain our decision.

For over a year, we worked to build our vision of accelerated Ruby testing in the cloud. We made our share of mistakes and learned a lot in the process. Ultimately, we found the demand was too small to warrant overcoming all the technical challenges that were harder than we anticipated.

We had many different technical challenges, but it boils down to this: we underestimated the difficulty of making it simple to run Ruby/Rails tests in the cloud.

When we began working on the Accelerator, we knew we would need many features: automatic installation of gems, automatic setup of common databases, and support for Test:Unit, Shoulda, and RSpec. However, other features caught us by surprise. For instance, many projects required running daemons, such as memcached or Solr to be running when the tests executed. Also, it was common for projects to work fine on a user’s machine, but fail in the cloud due to undeclared RubyGem dependencies (or dependencies that lacked sufficient version information). Some problems were extra difficult because, due to costs, we needed to be able to run many projects on a single EC2 instance (and it’s currently not possible to create virtual machines within EC2 Xen instances).

We worked hard to overcome these challenges, but it was still too common for users to discover that running tests in the cloud required additional work on their part (or, in some cases, that their project was not supported). And even worse, users had to jump through these hoops before they could assess whether or not the Accelerator would speed up their tests enough to make the process worth it. Because the potential benefit and possible configuration time were both unknown, many users understandably gave up before they experienced the time savings of running tests in parallel. And unfortunately, the projects that were most likely to need acceleration (those with large test suites), were also the most likely to have complex dependencies and require the most setup work.

While we may have been able to solve these problems with a lot of additional engineering, we also found that only a small percentage of teams really needed test acceleration. Some teams loved our solution, because they had test suites that took 10, 20, or even 45 minutes to run. However, most projects had smaller test suites. We found that many teams didn’t mind waiting for suites that took 3-5 minutes (or, more accurately, didn’t mind enough to justify the considerable time investment of setting up tests on the Accelerator).

So, after much discussion, we decided that we needed to create a product that would help a larger set of Ruby teams and require significantly less setup to provide value. Those discussions led to Caliper, a tool to identify risky code and technical debt in Ruby code.

Focus is key in a startup, especially a small one like ours. While we considered keeping the Accelerator running in maintenance mode, we realized we could not provide the level of support our users deserve while executing effectively on Caliper. As a result, we’ve decided to shut down the Accelerator indefinitely.

We deeply appreciate each and every person who helped us with the Accelerator, either by using it, or giving us ideas, feedback, and encouragement. We could not have gotten as far as we did without the help from users, friends, and mentors.

It pains us to shut down the service and we apologize for any inconvenience this causes our users. We hope that you understand our decision and we look forward to helping you improve your code with Caliper.

(Accelerator users: if you need to speed up your tests, here are some possible alternatives to Devver: multitest, parallel_specs, Testjour, and DeepTest. I don’t have experience with these tools, so I can’t comment on how well they work myself, but you may want to check them out.)

Written by Ben

January 6, 2010 at 2:17 pm

Posted in Devver

Tagged with , , ,