The Devver Blog

A Boulder startup improving the way developers work.

Archive for January 2010

Help us improve Caliper by taking a short survey

If you have tried Caliper, it would help us immensely if you filled out this short, eight-question survey at Thanks!

Written by Ben

January 25, 2010 at 1:49 pm

Posted in Uncategorized

Tagged with ,

Avdi Grimm on Confident Code

Recently Avdi, gave a talk on Confident Code at B’More on Rails. I enjoyed the talk and recommend giving it a listen, if you are interested in better structured code. The slides are available here, if you want to follow along.

Avdi Grimm on Confident Code at B’More on Rails from Avdi Grimm on Vimeo.

Written by DanM

January 20, 2010 at 7:02 am

Posted in Uncategorized

Upcoming improvements for Caliper

For the past several weeks, you may have noticed that not much has changed on Caliper. The reason for this is that we’ve been working hard to allow you to use Caliper on your private GitHub repositories (we’re currently in private beta for that feature. If you are interested, you can sign up for a beta invitation).

However, we realize that we have much to improve on Caliper. We’ve been talking with our users to identify the biggest problems and most-requested features. In addition, we’re getting help from the folks at Viget Labs to make Caliper easier to understand, more intuitive to use, and all around better.

In our first batch of work, we’ll be focusing on three improvements.

First, we’ll be developing an improved project dashboard. Our current dashboard isn’t terribly useful. We’d like to better represent the state of the project, important trends, and areas that need the most help. We’re working now to come up with a concrete design that addresses these issues. In the meantime, what data would you find most helpful on the dashboard?

We’ll also be changing the way you navigate to specific tools. Users have told us that the specific tool names (‘Reek’, ‘Flog’) are not, by themselves, very helpful for navigation. As a result, we’ll be increasingly highlighting their function. For example, even if you aren’t familiar with Flog, we’ll make it obvious that it is a tool for measuring complexity.

We’re considering eventually merging tools that have a similar function. So instead of going to two different pages for Reek and Roodi results, we might just have a ‘code smells’ report. Similarly, Flog and Saikuro might be presented within the same page. Would this be helpful?

To be clear, it isn’t our intention to obscure the fact that Caliper is built on great tools like metric_fu, Reek, Flog, etc. We will still make it easy to understand where the data is coming from and highlight the work being done by the dedicated open-source developers who write them. However, we think that grouping by function will make Caliper easier to use and understand.

Did you know that you can set up Caliper to automatically generate metrics using GitHub post-receive hooks? Did you know that, by default, up to 15 of your latest commits are automatically run through Caliper when you set up a project? Or that with one click, you can generate metrics for up to 40 commits throughout the history of your project?

Many users don’t know this, because our UI for these features is currently pretty terrible. The last task in the current batch of work will be to make these features more visible and easier to use. Our goal is to make it simple to see trends in your project over time.

(Incidentally, many users also don’t know that you can view a metrics diff for two commits to see how a commit has changed the metrics, but making this more noticeable may or may not get tackled in the first set of changes).

What’s next for Caliper after we complete these changes? We’re currently talking to users about what features and improvements matter most to them. One possibility is to display some or all of our metric data directly on a view of source code itself, so you don’t have to open up an editor (or click through to GitHub) to see the code. How important is this feature to you?

As always, feedback from our users is hugely helpful. If you have ideas about features that you’d like to see, please let us know in the comments or start a discussion at our support site. Thanks!

Written by Ben

January 14, 2010 at 5:21 pm

Posted in Uncategorized

Tagged with ,

Churn, a gem for class and method level churn rates

There are many metrics that can help identify potentially problematic code. One metric that has a high correlation with buggy code is high-churn code.

Code churn metrics were found to be among the most highly correlated with problem reports

Predicting Bugs from History, Microsoft Research

Through metric_fu, Ruby devs have had access to information about high-churn files, but that is a bit too high level. Often with Rails apps the few controllers you have bubble up to the top, as do your most commonly edited model files. Often this isn’t because old code is being modified and fixes being applied, often it is new features and methods are being added. What you really want to find is the classes and methods that seem to require updates every time the code is changed. Often this means a method needs to be refactored as it is doing too much, and is tightly coupled with many other parts of the code. If code is changing to often it is likely to be violating the Single Responsibility Principle.

Martin defines a responsibility as a reason to change, and concludes that a class or module should have one, and only one, reason to change.

Wikipedia, Single Responsibility Principle

In our own projects we have noticed this and sometimes recognized the problem. Often this leads to a bit of refactoring which separates the code in more logical and easier to debug and test pieces. Often though we haven’t noticed until many bugs keep popping up in the same area of the application. If there were a tool to identify these areas of the code base before a single person on the team notices the problem that would be great. Well, now with our Churn gem you can.

Churn analyzes each commit to your Git repo and notes which files, classes, and methods were changed. If you run Churn over a series of commits it will begin to build up a list of the highest churn classes and methods (it can always give you the high-churn files, since it can get that straight from Git log with no analysis). Currently the gem is just set up to output the information to the command line. Once we finish up a bit more testing it will be integrated into metric_fu, as well as Caliper. On Caliper it will be easy to go back and generate the current high-churn methods and classes for the projects history.

Give it a shot on your projects, and let me know if there are any feature requests, bugs, or problems. I hope this helps identify some places you can improve your projects.


> gem install churn
> churn

example output:

* Revision Changes
 * "lib/churn/churn_calculator.rb"

 * {"klass"=>"ChurnCalculator", "file"=>"lib/churn/churn_calculator.rb"}

 * {"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#calculate_revision_data", "file"=>"lib/churn/churn_calculator.rb"}
 * {"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#filters", "file"=>"lib/churn/churn_calculator.rb"}
 * {"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#display_array", "file"=>"lib/churn/churn_calculator.rb"}
 * {"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#initialize", "file"=>"lib/churn/churn_calculator.rb"}
* Project Churn
 * {:file_path=>"lib/churn/churn_calculator.rb", :times_changed=>13}
 * {:file_path=>"README.rdoc", :times_changed=>7}
 * {:file_path=>"lib/tasks/churn_tasks.rb", :times_changed=>6}
 * {:file_path=>"Rakefile", :times_changed=>5}
 * {:file_path=>"VERSION", :times_changed=>4}
 * {:file_path=>"lib/churn/git_analyzer.rb", :times_changed=>4}
 * {:file_path=>"test/test_helper.rb", :times_changed=>4}
 * {:file_path=>"test/unit/churn_calculator_test.rb", :times_changed=>3}
 * {:file_path=>"test/churn_test.rb", :times_changed=>3}

 * {"klass"=>{"klass"=>"ChurnCalculator", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>9}
 * {"klass"=>{"klass"=>"LocationMapping", "file"=>"lib/churn/location_mapping.rb"}, "times_changed"=>1}
 * {"klass"=>{"klass"=>"GitAnalyzer", "file"=>"lib/churn/git_analyzer.rb"}, "times_changed"=>1}
 * {"klass"=>{"klass"=>"ChurnTest", "file"=>"test/churn_test.rb"}, "times_changed"=>1}
 * {"klass"=>{"klass"=>"ChurnCalculatorTest", "file"=>"test/unit/churn_calculator_test.rb"}, "times_changed"=>1}

 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#initialize", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>3}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#to_h", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>3}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#analyze", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>2}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#report", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>2}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#display_array", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>2}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#calculate_revision_data", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>2}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#emit", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>1}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#calculate_revision_changes", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>1}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#get_changes", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>1}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#calculate_changes!", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>1}
 * {"method"=>{"klass"=>"ChurnCalculator", "method"=>"ChurnCalculator#set_source_control", "file"=>"lib/churn/churn_calculator.rb"}, "times_changed"=>1}
 * {"method"=>{"klass"=>"GitAnalyzer", "method"=>"GitAnalyzer#date_range", "file"=>"lib/churn/git_analyzer.rb"}, "times_changed"=>1}

Visit Churn on github, and view the current Churn metrics.

Written by DanM

January 12, 2010 at 3:30 pm

Posted in Uncategorized

Shutting down the Devver Test Accelerator

After much deliberation, we’ve decided to shut down our Test Accelerator product on January 22nd, 2010. (the Accelerator was previously just known as “Devver”, but to clarify between our old product and Caliper, we’re referring to it as the “Accelerator” or the “Test Accelerator”).

This has been a difficult decision for us. Understandably, some people have been confused and disappointed by this news, so I wanted to take some time to explain our decision.

For over a year, we worked to build our vision of accelerated Ruby testing in the cloud. We made our share of mistakes and learned a lot in the process. Ultimately, we found the demand was too small to warrant overcoming all the technical challenges that were harder than we anticipated.

We had many different technical challenges, but it boils down to this: we underestimated the difficulty of making it simple to run Ruby/Rails tests in the cloud.

When we began working on the Accelerator, we knew we would need many features: automatic installation of gems, automatic setup of common databases, and support for Test:Unit, Shoulda, and RSpec. However, other features caught us by surprise. For instance, many projects required running daemons, such as memcached or Solr to be running when the tests executed. Also, it was common for projects to work fine on a user’s machine, but fail in the cloud due to undeclared RubyGem dependencies (or dependencies that lacked sufficient version information). Some problems were extra difficult because, due to costs, we needed to be able to run many projects on a single EC2 instance (and it’s currently not possible to create virtual machines within EC2 Xen instances).

We worked hard to overcome these challenges, but it was still too common for users to discover that running tests in the cloud required additional work on their part (or, in some cases, that their project was not supported). And even worse, users had to jump through these hoops before they could assess whether or not the Accelerator would speed up their tests enough to make the process worth it. Because the potential benefit and possible configuration time were both unknown, many users understandably gave up before they experienced the time savings of running tests in parallel. And unfortunately, the projects that were most likely to need acceleration (those with large test suites), were also the most likely to have complex dependencies and require the most setup work.

While we may have been able to solve these problems with a lot of additional engineering, we also found that only a small percentage of teams really needed test acceleration. Some teams loved our solution, because they had test suites that took 10, 20, or even 45 minutes to run. However, most projects had smaller test suites. We found that many teams didn’t mind waiting for suites that took 3-5 minutes (or, more accurately, didn’t mind enough to justify the considerable time investment of setting up tests on the Accelerator).

So, after much discussion, we decided that we needed to create a product that would help a larger set of Ruby teams and require significantly less setup to provide value. Those discussions led to Caliper, a tool to identify risky code and technical debt in Ruby code.

Focus is key in a startup, especially a small one like ours. While we considered keeping the Accelerator running in maintenance mode, we realized we could not provide the level of support our users deserve while executing effectively on Caliper. As a result, we’ve decided to shut down the Accelerator indefinitely.

We deeply appreciate each and every person who helped us with the Accelerator, either by using it, or giving us ideas, feedback, and encouragement. We could not have gotten as far as we did without the help from users, friends, and mentors.

It pains us to shut down the service and we apologize for any inconvenience this causes our users. We hope that you understand our decision and we look forward to helping you improve your code with Caliper.

(Accelerator users: if you need to speed up your tests, here are some possible alternatives to Devver: multitest, parallel_specs, Testjour, and DeepTest. I don’t have experience with these tools, so I can’t comment on how well they work myself, but you may want to check them out.)

Written by Ben

January 6, 2010 at 2:17 pm

Posted in Devver

Tagged with , , ,