The Devver Blog

A Boulder startup improving the way developers work.

Archive for the ‘Devver’ Category

Devver.next

We announced about two weeks ago that Devver and Caliper would be shutting down. The Caliper service will be shut down on April 30th, and Devver will be ceasing operations. We shared some of our thoughts about lessons learned while working on Devver.

Now people have been asking what Ben and I will be doing next. Honestly, at the moment it remains a mystery as much to us as to anyone. Both Ben and I have been working on startups together for over 3 years, something we had talked about doing together since high school. After our experience we both plan to take a bit of time off, to work on open source, personal projects, learn new things, and maybe catch up on some hobbies that have been neglected. Since Devver and the structure around it will be disappearing, we wanted to share our personal contact info in case anyone wants to get in touch with us. We will be looking at new work to get involved with sometime in May. Feel free to contact either of us if there is an opportunity one of us might be interested in.

Ben Brinckerhoff can be found online at bbrinck.com, and his email is ben@bbrinck.com

Dan Mayer can be found online at mayerdan.com, and his email is dan@mayerdan.com

We have learned an amazing amount over the last couple of years. We both feel like this has been an amazing opportunity and one last time want to thank everyone for their support. Thanks to the Ruby community, all the awesome Techstars teams, the startup community, our friends, families, and investors. We never would have made it this far and lasted this long in the startup world without all of you.

Next? Life is a journey, and we are excited to see whatever the future brings us. Thanks for all the good times, knowledge learned, and all the amazing people we met along the way.

Advertisements

Written by DanM

April 30, 2010 at 8:45 am

Posted in Boulder, Devver, Ruby, TechStars

Tagged with ,

Lessons Learned

As we’ve begun to wrap things up here at Devver, we’ve had the chance to reflect a bit on our experience. Although shutting down is not the outcome we wanted, it’s clear to both Dan and I that doing a startup has been an amazing learning experience. While we still have a lot to learn, we wanted to share some of the most important lessons we’ve learned during this time.

The community

When we started Devver, we were hesitant to ask for feedback and help. We quickly found that people are incredibly helpful and generous with their time. Users were willing to take a chance and use our products while giving us valuable feedback. Fellow Rubyists gave us ideas and helped us with technical problems. Mentors made time for meetings and introduced us to others who could assist us. And other entrepreneurs, both new and seasoned, were happy to share stories, compare experiences, and offer support.

If you are working on a startup, don’t be afraid to ask for help! The vast majority of people want to help you succeed, provided that you respect them and their time. That means you need to prepare adequately (do your research and ask good questions), figure out their preferred methods of communication (e.g. don’t call if they prefer email), show up on time, don’t overburden them, and thank them. And when other people need your help, give back!

You can build awesome relationships with various communities on your own, but we strongly recommend joining a mentorship program like TechStars. The program accelerated the process of connecting with mentors, users, and other entrepreneurs by providing an amazing community during the summer (and to this day). The advice, introductions, and support have been simply incredible.

Founding team

Dan and I are both technical founders. Looking back, it would have been to our advantage to have a third founder who really loved the business aspect of running a startup.

There is a belief (among technical founders) that technical founders are sufficient for a successful startup. Or, put more harshly, that you can teach a hacker business, but you can’t teach a businessman how to hack“. I don’t want to argue whether that’s true or not. Clearly there are examples of technical founders being sufficient to get a company going, but my point is that having solely technical founders is non-optimal. You can teach a hacker business, but you can’t make him or her get excited about it, which means it may not get the time or attention it deserves.

Hackers are passionate about, well, hacking. And so we tend to measure progress in terms of features completed or lines of code written. Clearly, code needs to be written, but ideally a startup would have a founder who is working on important non-technical tasks: talking with customers, measuring key metrics, developing distribution channels, etc. I’m not advocating that only one founder works on these tasks while technical founders ignore customer development – everyone needs to get involved. Rather, I’m pointing out that given a choice, technical founders will tend to solve problems technically and having a founder who has the opposite default is valuable.

Remote teams

We embraced working remotely: we hired Avdi to work in Pennsylvania while Dan and I lived in Boulder and later on, Dan moved to Washington, DC. There are many benefits to having a distributed team, but two stood out in our experience. First, we could hire top talent without having to worry about location (in fact, our flexibility regarding location was very attractive to most candidates we interviewed). Secondly, being in different locations allowed every team member to work with minimal distractions, which is invaluable when it comes to efficiently writing good code.

That said, communication was a challenge. To ensure we were all synced up, we had a daily standup as well as a weekly review. When Dan moved to DC, he and I scheduled another weekly meeting with no set agenda to just bring up all the issues, large and small, that were on our minds. We also all got together in the same location every few months to work in the same room and rekindle our team energy.

Also, pair programming was difficult to do remotely and we never came up with a great solution. As a result, we spent less than a day pairing a week on average.

The most significant drawback to a remote team is the administrative hassle. It’s a pain to manage payroll, unemployment, insurance, etc in one state. It’s a freaking nightmare to manage in three states (well, two states and a district), even though we paid a payroll service to take care of it. Apparently, once your startup gets larger, there are companies that will manage this with minimal hassle, but for a small team, it was a major annoyance and distraction.

Product development

Most of the mistakes we made developing our test accelerator and, later, Caliper boiled down to one thing: we should have focused more on customer development and finding a minimum viable product (MVP).

The first thing we worked on was our Ruby test accelerator. At the time, we thought we had found our MVP: we had made encouraging technical progress and we had talked to several potential customers who were excited about the product we were building. Anything simpler seems “too simple” to be interesting.

Our mistake at that point was to go “heads down” and focus on building the accelerator while minimizing our contact with users and customers (after all, we knew how great it was and time spent talking to customers was time we could be hacking!). We should have asking, “Is there an even simpler version of this product that we can deliver sooner to learn more about pricing, market size, and technical challenges?”

If we had done so, we would have discovered:

  • whether the need was great enough (and if the solution was good enough) to convince people to open their wallets
  • that while a few users acutely felt the pain of slow tests, most didn’t care about acceleration. However, many of those users did want a “simpler” application – non-accelerated Ruby cloud testing.
  • the primary technical challenge was not accelerating tests, it was configuring servers for customers’ Rails applications. Not only did we spend time focusing on the wrong technical challenges, we also made architectural decisions that actually made it harder to solve this core problem.

After eventually discovering that setup and configuration was our primary adoption problem (and after trying and failing to implement various strategies to make it simple and easy), we tried to move to the other end of the spectrum. Caliper was designed to provide value with zero setup or configuration – users just provided a link to source code and instantly got valuable data.

Unfortunately, we again made the mistake of focusing on engineering first and customer development second. We released our first version to some moderate success and then proceeded to continue to churn out features without really understanding customer needs. Only later on, after finally engaging potential customers did we realize that market was too small and price point was to low to have Caliper sustain our company by itself.

Conclusion

This is by no means a comprehensive list, but it is our hope that other startups and founders-to-be can learn from our experiences, both mistakes and successes. Doing a startup has been an incredible learning experience for both Dan and I and we look forward to learning more in the future – both first-hand and from the amazing group of entrepreneurs and hackers that we’ve been privileged enough to know.

Written by Ben

April 26, 2010 at 11:04 am

Closing up shop

Today we’re officially announcing that Devver (and our Caliper service) will be shutting down on Friday, April 30th.

This was one of the hardest decisions we’ve ever had to make, but, after careful consideration, we’ve decided it is the right one.

Nearly two years ago, we started Devver with the vision of using the cloud to build tools that would change developers lives. We’ve worked hard to achieve that vision, but ultimately we’ve had to face the reality that the services we’ve built will not generate the revenue necessary to sustain and grow our company (although it could be used to enhance or complement another cloud product or service – contact me if you’re interested).

To all of our users – thank you so much for taking a chance by using our tools, for reporting bugs, and for giving us ideas for the future. We deeply appreciate each and every one of you.

While this is not the end we wanted, we know how fortunate we were to be able to take this journey and thank everyone – friends, family, investors, mentors, users, and the Ruby community – who helped us along the way. You’re all awesome and we couldn’t have gone this far without you. Thank you.

– Ben & Dan

Written by Ben

April 19, 2010 at 5:17 pm

Posted in Devver

Shutting down the Devver Test Accelerator

After much deliberation, we’ve decided to shut down our Test Accelerator product on January 22nd, 2010. (the Accelerator was previously just known as “Devver”, but to clarify between our old product and Caliper, we’re referring to it as the “Accelerator” or the “Test Accelerator”).

This has been a difficult decision for us. Understandably, some people have been confused and disappointed by this news, so I wanted to take some time to explain our decision.

For over a year, we worked to build our vision of accelerated Ruby testing in the cloud. We made our share of mistakes and learned a lot in the process. Ultimately, we found the demand was too small to warrant overcoming all the technical challenges that were harder than we anticipated.

We had many different technical challenges, but it boils down to this: we underestimated the difficulty of making it simple to run Ruby/Rails tests in the cloud.

When we began working on the Accelerator, we knew we would need many features: automatic installation of gems, automatic setup of common databases, and support for Test:Unit, Shoulda, and RSpec. However, other features caught us by surprise. For instance, many projects required running daemons, such as memcached or Solr to be running when the tests executed. Also, it was common for projects to work fine on a user’s machine, but fail in the cloud due to undeclared RubyGem dependencies (or dependencies that lacked sufficient version information). Some problems were extra difficult because, due to costs, we needed to be able to run many projects on a single EC2 instance (and it’s currently not possible to create virtual machines within EC2 Xen instances).

We worked hard to overcome these challenges, but it was still too common for users to discover that running tests in the cloud required additional work on their part (or, in some cases, that their project was not supported). And even worse, users had to jump through these hoops before they could assess whether or not the Accelerator would speed up their tests enough to make the process worth it. Because the potential benefit and possible configuration time were both unknown, many users understandably gave up before they experienced the time savings of running tests in parallel. And unfortunately, the projects that were most likely to need acceleration (those with large test suites), were also the most likely to have complex dependencies and require the most setup work.

While we may have been able to solve these problems with a lot of additional engineering, we also found that only a small percentage of teams really needed test acceleration. Some teams loved our solution, because they had test suites that took 10, 20, or even 45 minutes to run. However, most projects had smaller test suites. We found that many teams didn’t mind waiting for suites that took 3-5 minutes (or, more accurately, didn’t mind enough to justify the considerable time investment of setting up tests on the Accelerator).

So, after much discussion, we decided that we needed to create a product that would help a larger set of Ruby teams and require significantly less setup to provide value. Those discussions led to Caliper, a tool to identify risky code and technical debt in Ruby code.

Focus is key in a startup, especially a small one like ours. While we considered keeping the Accelerator running in maintenance mode, we realized we could not provide the level of support our users deserve while executing effectively on Caliper. As a result, we’ve decided to shut down the Accelerator indefinitely.

We deeply appreciate each and every person who helped us with the Accelerator, either by using it, or giving us ideas, feedback, and encouragement. We could not have gotten as far as we did without the help from users, friends, and mentors.

It pains us to shut down the service and we apologize for any inconvenience this causes our users. We hope that you understand our decision and we look forward to helping you improve your code with Caliper.

(Accelerator users: if you need to speed up your tests, here are some possible alternatives to Devver: multitest, parallel_specs, Testjour, and DeepTest. I don’t have experience with these tools, so I can’t comment on how well they work myself, but you may want to check them out.)

Written by Ben

January 6, 2010 at 2:17 pm

Posted in Devver

Tagged with , , ,

Caliper: Next Steps

We released the first version of Caliper just under two months ago. Since then, we’ve fixed bugs, made improvements, and rolled out new features. The feedback we’ve gotten from the community has been encouraging and your suggestions have helped up improve Caliper each week.

Initially, we simply wanted to take an existing tool (namely, metric_fu) and make it dead simple to set up, so that all open-source projects could get access to metric data. However, our larger goal is to improve the usability, features, and data presentation in Caliper so it becomes an indispensable part of your regular Ruby development process.

To that end, I’m happy to announce that support for private projects is on the road map. We’re still working out the details, but we plan to release this feature within the next thirty days.

At the same time, we’ll work with users to figure out how we can make metrics useful enough to be a regular part of your process (by “regular”, we mean that we hope you use metrics at least once a week).

We recently put out a short survey (results here) to try to assess what features are most important for our users. By looking at the results (somewhat limited, likely due to overly wordy questions on my part) and based on some conversations I had at RubyConf, we see a few possible scenarios for where to take Caliper next.

“Code exploration”
In this scenario, you’d primarily use Caliper to jump into an existing code base, in order to help you better understand where the problematic areas are. This might be used by consultants for a project-level code review, for understanding how to start contributing to an open-source project, or even to periodically do a survey of problems of your own project. The primary features for this scenario would be our Hot Spots feature (which list the worst methods, classes, and files in the project). Important new features include a display that shows the source code with the problems displayed alongside the source.

“Quality monitoring”
In this scenario, the primary usage for Caliper is to set and maintain certain metrics thresholds within your code base. You’d set levels for the various tools and then get alerts when those thresholds are reached. Key features would include an ability to customize thresholds for each tool, a dashboard that displays offending classes and methods, and customizable notifications.

“Refactoring assistant”
In this scenario, a single developer uses Caliper to assist them in refactoring up his/her code, perhaps after reviewing a commit that will be pushed, or because a specific section of code is known to be overly complex. Caliper would both find the biggest problems and provide feedback on how refactorings change the metrics. Key features would include an improved commit-based view of metric changes as well as the ability to push code directly to Caliper before merge that code into a shared ‘master’ branch.

Do any of these scenarios resonate with you? Would you prefer one over another? Is there another scenario that would be more useful? If you are currently using metrics in your project, how do you use them and what features do you want? If you don’t currently use metrics, what features would make you change your mind? Please make your voice heard in the comments, so we can make Caliper a valuable addition to your toolbox.

Written by Ben

November 24, 2009 at 4:57 pm

Posted in Devver

Tagged with , ,

Improving Code using Metric_fu

Often, when people see code metrics they think, “that is interesting, I don’t know what to do with it.” I think metrics are great, but when you can really use them to improve your project’s code, that makes them even more valuable. metric_fu provides a bunch of great metric information, which can be very useful. But if you don’t know what parts of it are actionable it’s merely interesting instead of useful.

One thing when looking at code metrics to keep in mind is that a single metric may not be as interesting. If you look at a metric trends over time it might help give you more meaningful information. Showing this trending information is one of our goals with Caliper. Metrics can be your friend watching over the project and like having a second set of eyes on how the code is progressing, alerting you to problem areas before they get out of control. Working with code over time, it can be hard to keep everything in your head (I know I can’t). As the size of the code base increases it can be difficult to keep track of all the places where duplication or complexity is building up in the code. Addressing the problem areas as they are revealed by code metrics can keep them from getting out of hand, making future additions to the code easier.

I want to show how metrics can drive changes and improve the code base by working on a real project. I figured there was no better place to look than pointing metric_fu at our own devver.net website source and fixing up some of the most notable problem areas. We have had our backend code under metric_fu for awhile, but hadn’t been following the metrics on our Merb code. This, along with some spiked features that ended up turning into Caliper, led to some areas getting a little out of control.

Flay Score before cleanup

When going through metric_fu the first thing I wanted to start to work on was making the code a bit more DRY. The team and I were starting to notice a bit more duplication in the code than we liked. I brought up the Flay results for code duplication and found that four databases models shared some of the same methods.

Flay highlighted the duplication. Since we are planning on making some changes to how we handle timestamps soon, it seemed like a good place to start cleaning up. Below are the methods that existed in all four models. A third method ‘update_time’ existed in two of the four models.

 def self.pad_num(number, max_digits = 15)
    "%%0%di" % max_digits % number.to_i
  end

  def get_time
      Time.at(self.time.to_i)
  end

Nearly all of our DB tables store time in a way that can be sorted with SimpleDB queries. We wanted to change our time to be stored as UTC in the ISO 8601 format. Before changing to the ISO format, it was easy to pull these methods into a helper module and include it in all the database models.

module TimeHelper

  module ClassMethods
    def pad_num(number, max_digits = 15)
      "%%0%di" % max_digits % number.to_i
    end
  end

  def get_time
      Time.at(self.time.to_i)
  end

  def update_time
    self.time = self.class.pad_num(Time.now.to_i)
  end

end

Besides reducing the duplication across the DB models, it also made it much easier to include another time method update_time, which was in two of the DB models. This consolidated all the DB time logic into one file, so changing the time format to UTC ISO 8601 will be a snap. While this is a trivial example of a obvious refactoring it is easy to see how helper methods can often end up duplicated across classes. Flay can come in really handy at pointing out duplication that over time that can occur.

Flog gives a score showing how complex the measured code is. The higher the score the greater the complexity. The more complex code is the harder it is to read and it likely contains higher defect density. After removing some duplication from the DB models I found our worst database model based on Flog scores was our MetricsData model. It included an incredibly bad high flog score of 149 for a single method.

File Total score Methods Average score Highest score
/lib/sdb/metrics_data.rb 327 12 27 149

The method in question was extract_data_from_yaml, and after a little refactoring it was easy to make extract_data_from_yaml drop from a score of 149 to a series of smaller methods with the largest score being extract_flog_data! (33.6). The method was doing too much work and was frequently being changed. The method was extracting the data from 6 different metric tools and creating summary of the data.

The method went from a sprawling 42 lines of code to a cleaner and smaller method of 10 lines and a collection of helper methods that look something like the below code:

  def self.extract_data_from_yaml(yml_metrics_data)
    metrics_data = Hash.new {|hash, key| hash[key] = {}}
    extract_flog_data!(metrics_data, yml_metrics_data)
    extract_flay_data!(metrics_data, yml_metrics_data)
    extract_reek_data!(metrics_data, yml_metrics_data)
    extract_roodi_data!(metrics_data, yml_metrics_data)
    extract_saikuro_data!(metrics_data, yml_metrics_data)
    extract_churn_data!(metrics_data, yml_metrics_data)
    metrics_data
  end

  def self.extract_flog_data!(metrics_data, yml_metrics_data)
    metrics_data[:flog][:description] = 'measures code complexity'
    metrics_data[:flog]["average method score"] = Devver::Maybe(yml_metrics_data)[:flog][:average].value(N_A)
    metrics_data[:flog]["total score"]   = Devver::Maybe(yml_metrics_data)[:flog][:total].value(N_A)
    metrics_data[:flog]["worst file"] = Devver::Maybe(yml_metrics_data)[:flog][:pages].first[:path].fmap {|x| Pathname.new(x)}.value(N_A)
  end

Churn gives you an idea of files that might be in need of a refactoring. Often if a file is changing a lot it means that the code is doing too much, and would be more stable and reliable if broken up into smaller components. Looking through our churn results, it looks like we might need another layout to accommodate some of the different styles on the site. Another thing that jumps out is that both the TestStats and Caliper controller have fairly high churn. The Caliper controller has been growing fairly large as it has been doing double duty for user facing features and admin features, which should be split up. TestStats is admin controller code that also has been growing in size and should be split up into more isolated cases.

churn results

Churn gave me an idea of where might be worth focusing my effort. Diving in to the other metrics made it clear that the Caliper controller needed some attention.

The Flog, Reek, and Roodi Scores for Caliper Controller:

File Total score Methods Average score Highest score
/app/controllers/caliper.rb 214 14 15 42

reek before cleanup

Roodi Report
app/controllers/caliper.rb:34 - Method name "index" has a cyclomatic complexity is 14.  It should be 8 or less.
app/controllers/caliper.rb:38 - Rescue block should not be empty.
app/controllers/caliper.rb:51 - Rescue block should not be empty.
app/controllers/caliper.rb:77 - Rescue block should not be empty.
app/controllers/caliper.rb:113 - Rescue block should not be empty.
app/controllers/caliper.rb:149 - Rescue block should not be empty.
app/controllers/caliper.rb:34 - Method name "index" has 36 lines.  It should have 20 or less.

Found 7 errors.

Roodi and Reek both tell you about design and readability problems in your code. The screenshot of our Reek ‘code smells’ in the Caliper controller should show how it had gotten out of hand. The code smells filled an entire browser page! Roodi similarly had many complaints about the Caliper controller. Flog was also showing the file was getting a bit more complex than it should be. After picking off some of the worst Roodi and Reek complaints and splitting up methods with high Flog scores, the code had become easily readable and understandable at a glance. In fact I nearly cut the Reek complaints in half for the controller.

Reek after cleanup

Refactoring one controller, which had been quickly hacked together and growing out of control, brought it from a dizzying 203 LOC to 138 LOC. The metrics drove me to refactor long methods (52 LOC => 3 methods the largest being 23 LOC), rename unclear variable names (s => stat, p => project), move some helpers methods out of the controller into the helper class where they belong. Yes, all these refactorings and good code designs can be done without metrics, but it can be easy to overlook bad code smells when they start small, metrics can give you an early warning that a section of code is becoming unmanageable and likely prone to higher defect rates. The smaller file was a huge improvement in terms of cyclomatic complexity, LOC, code duplication, and more importantly, readability.

Obviously I think code metrics are cool, and that your projects can be improved by paying attention to them as part of the development lifecycle. I wrote about metric_fu so that anyone can try these metrics out on their projects. I think metric_fu is awesome, and my interest in Ruby tools is part of what drove us to build Caliper, which is really the easiest way try out metrics for your project. Currently, you can think of it as hosted metric_fu, but we are hoping to go even further and make the metrics clearly actionable to users.

In the end, yep, this is a bit of a plug for a product I helped build, but it is really because I think code metrics can be a great tool to help anyone with their development. So submit your repo in and give Caliper hosted Ruby metrics a shot. We are trying to make metrics more actionable and useful for all Ruby developers out, so we would love to here from you with any ideas about how to improve Caliper, please contact us.

Written by DanM

October 27, 2009 at 10:30 pm

Announcing Caliper

If you’re anything like us, you recognize the usefulness of code quality metrics. But configuring metrics tools and linking them to your Continuous Integration system is a chore you may not have found time for. Here at Devver we’re all about taking the pain out of common development chores. Which is why we’re pleased to announce a new Devver service: Caliper.

Caliper is the easy way to get code metrics – like cyclomatic complexity and code duplication – about your project. Just go to http://devver.net/caliper. Enter the public Git repository URL of a Ruby project. In a few moments, we’ll give you a full set of metrics, generated by the awesome metric_fu gem.

Try it out right here:

If your project is hosted on Github, we can even re-generate metrics automatically every time someone makes a commit. Just add the URL

http://api.devver.net/github

to your project’s Post-Receive Hooks.

Caliper is currently an Alpha product. We will be be rapidly adding features and refinements over the next few weeks. If you have a problem, a question, or a feature request, please let us know!

Written by avdi

October 9, 2009 at 8:44 am