The Devver Blog

A Boulder startup improving the way developers work.

Archive for November 2009

Caliper: Next Steps

We released the first version of Caliper just under two months ago. Since then, we’ve fixed bugs, made improvements, and rolled out new features. The feedback we’ve gotten from the community has been encouraging and your suggestions have helped up improve Caliper each week.

Initially, we simply wanted to take an existing tool (namely, metric_fu) and make it dead simple to set up, so that all open-source projects could get access to metric data. However, our larger goal is to improve the usability, features, and data presentation in Caliper so it becomes an indispensable part of your regular Ruby development process.

To that end, I’m happy to announce that support for private projects is on the road map. We’re still working out the details, but we plan to release this feature within the next thirty days.

At the same time, we’ll work with users to figure out how we can make metrics useful enough to be a regular part of your process (by “regular”, we mean that we hope you use metrics at least once a week).

We recently put out a short survey (results here) to try to assess what features are most important for our users. By looking at the results (somewhat limited, likely due to overly wordy questions on my part) and based on some conversations I had at RubyConf, we see a few possible scenarios for where to take Caliper next.

“Code exploration”
In this scenario, you’d primarily use Caliper to jump into an existing code base, in order to help you better understand where the problematic areas are. This might be used by consultants for a project-level code review, for understanding how to start contributing to an open-source project, or even to periodically do a survey of problems of your own project. The primary features for this scenario would be our Hot Spots feature (which list the worst methods, classes, and files in the project). Important new features include a display that shows the source code with the problems displayed alongside the source.

“Quality monitoring”
In this scenario, the primary usage for Caliper is to set and maintain certain metrics thresholds within your code base. You’d set levels for the various tools and then get alerts when those thresholds are reached. Key features would include an ability to customize thresholds for each tool, a dashboard that displays offending classes and methods, and customizable notifications.

“Refactoring assistant”
In this scenario, a single developer uses Caliper to assist them in refactoring up his/her code, perhaps after reviewing a commit that will be pushed, or because a specific section of code is known to be overly complex. Caliper would both find the biggest problems and provide feedback on how refactorings change the metrics. Key features would include an improved commit-based view of metric changes as well as the ability to push code directly to Caliper before merge that code into a shared ‘master’ branch.

Do any of these scenarios resonate with you? Would you prefer one over another? Is there another scenario that would be more useful? If you are currently using metrics in your project, how do you use them and what features do you want? If you don’t currently use metrics, what features would make you change your mind? Please make your voice heard in the comments, so we can make Caliper a valuable addition to your toolbox.

Written by Ben

November 24, 2009 at 4:57 pm

Posted in Devver

Tagged with , ,

Announcing Caliper Community Statistics

For the past few months, we’ve been building Caliper to help you easily generate code metrics for your Ruby projects. We’ve recently added another dimension of metrics information: community statistics for all the Ruby projects that are currently in Caliper.

The idea of community statistics is two-fold. From a practical perspective, you can now compare your project’s metrics to the community. For example, Flog measures the complexity of methods. Many people wonder exactly defines a good Flog score for an individual method. In Jake Scruggs’ opinion, a score of 0-10 is “Awesome”, while a score of 11-20 is “Good enough”. That sounds correct, but with Caliper’s community metrics, we can also compare the average Flog scores for entire projects to see what defines a good average score.

To do so, we calculate the average Flog method score for each project and plot those averages on a histogram, like so:

flog_average_method_histogram

Looking at the data, we see that a lot of projects have an average Flog score between 6 and 12 (the mean is 10.3 and the max is is 21.3).

If your project’s average Flog score is 9, does that mean it has only “Awesome” methods, Flog-wise? Well, remember that we’re looking at the average score for each project. I suspect that in most projects, lots of tiny methods are pulling down the average, but there are still plenty of big, nasty methods. It would be interesting to look at the community statistics for maximum Flog score per project or see a histogram of the Flog scores for all methods across all projects (watch this space!).

Since several of the metrics (like Reek, which detects code smells) have scores that grow in proportion to the number of lines of code, we divide the raw score by each project’s lines of code. As a result, we can sensibly compare your project to other projects, no matter what the difference in size.

The second reason we’re calculating community statistics is so we can discover trends across the Ruby community. For example, we can compare the ratio of lines of application code to test code. It’s interesting to note that a significant portion of projects in Caliper have no tests, but that, for the projects that do have tests, most of them have a code:test ratio in the neighborhood of 2:1.

code_to_test_ratio_histogram

Other interesting observations from our initial analysis:
* A lot of projects (mostly small ones) have no Flay duplications.
* Many smaller projects have no Reek smells, but the average project has about 1 smell per 9 lines of code.

Want to do your own analysis? We’ve built a scatter plotter so you can see if any two metrics have any correlation. For instance, note the correlation between code complexity and code smells.

Here’s a scatter plot of that data (zoomed in):

scatter_plot

Over the coming weeks, we’ll improve the graphs we have and add new graphs that expose interesting trends. But we need your help! Please let us know if you spot problems, have ideas for new graphs, or have any questions. Additionally, please add your project to Caliper so it can be included in our community statistics. Finally, feel free to grab the raw stats from our alpha API* and play around yourself!

* Quick summary:

curl http://api.devver.net/metrics

for JSON,

curl -H 'Accept:text/x-yaml' http://api.devver.net/metrics

for YAML. More details. API is under development, so please send us feedback!

Written by Ben

November 19, 2009 at 8:37 am

Posted in Development, Ruby, Tools

Tagged with ,

Announcing DataMapper/SimpleDB Adapter 1.0.0

Today we are pleased to announce the release of dm-adapter-simpledb 1.0.0.

The big new feature is compatibility with DataMapper 0.10.*. The DataMapper adapter API changed significantly in 0.10, necessitating a broad rewrite. This release breaks backwards compatibility with DataMapper 0.9.*.

In addition, the way nils are stored has changed, so datasets created with the 0.9.* adapter may need to be updated. Previously nils were stored in the form of a special nil value (by default, the string “nil”). Starting with version 1.0.0, any attributes set to nil will be removed from the SimpleDB record. This is more in line with the SimpleDB concept of NULL, and enables

IS NOT NULL

predicates to work as expected.

New and existing SimpleDB adapter features include:

  • NEW: Supports all DataMapper query predicates.
  • NEW: DataMapper identity map support for record caching
  • NEW: Lazy-loaded attributes
  • NEW: DataMapper Serial property support via UUIDs.
  • Uses the RightAWS gem for efficient SimpleDB operations.
  • Full set of CRUD operations
  • Can translate many queries into efficient native SELECT operations.
  • Migrations
  • Array properties
  • Basic aggregation support (Model.count(“…”))
  • String “chunking” permits attributes to exceed the 1024-byte limit

If you have Gemcutter in your Gem sources, you can install the gem immediately:

gem install dm-adapter-simpledb

Check out the code at http://github.com/devver/dm-adapter-simpledb. You can see metrics for the code here. Give it a try, and feel free to send us any feedback, patches, and suggestions. The project’s issue tracker is on GitHub, or you can always email us.

Written by avdi

November 16, 2009 at 12:45 pm

Posted in Uncategorized

Tagged with ,