The Devver Blog

A Boulder startup improving the way developers work.

Announcing Caliper Community Statistics

For the past few months, we’ve been building Caliper to help you easily generate code metrics for your Ruby projects. We’ve recently added another dimension of metrics information: community statistics for all the Ruby projects that are currently in Caliper.

The idea of community statistics is two-fold. From a practical perspective, you can now compare your project’s metrics to the community. For example, Flog measures the complexity of methods. Many people wonder exactly defines a good Flog score for an individual method. In Jake Scruggs’ opinion, a score of 0-10 is “Awesome”, while a score of 11-20 is “Good enough”. That sounds correct, but with Caliper’s community metrics, we can also compare the average Flog scores for entire projects to see what defines a good average score.

To do so, we calculate the average Flog method score for each project and plot those averages on a histogram, like so:

flog_average_method_histogram

Looking at the data, we see that a lot of projects have an average Flog score between 6 and 12 (the mean is 10.3 and the max is is 21.3).

If your project’s average Flog score is 9, does that mean it has only “Awesome” methods, Flog-wise? Well, remember that we’re looking at the average score for each project. I suspect that in most projects, lots of tiny methods are pulling down the average, but there are still plenty of big, nasty methods. It would be interesting to look at the community statistics for maximum Flog score per project or see a histogram of the Flog scores for all methods across all projects (watch this space!).

Since several of the metrics (like Reek, which detects code smells) have scores that grow in proportion to the number of lines of code, we divide the raw score by each project’s lines of code. As a result, we can sensibly compare your project to other projects, no matter what the difference in size.

The second reason we’re calculating community statistics is so we can discover trends across the Ruby community. For example, we can compare the ratio of lines of application code to test code. It’s interesting to note that a significant portion of projects in Caliper have no tests, but that, for the projects that do have tests, most of them have a code:test ratio in the neighborhood of 2:1.

code_to_test_ratio_histogram

Other interesting observations from our initial analysis:
* A lot of projects (mostly small ones) have no Flay duplications.
* Many smaller projects have no Reek smells, but the average project has about 1 smell per 9 lines of code.

Want to do your own analysis? We’ve built a scatter plotter so you can see if any two metrics have any correlation. For instance, note the correlation between code complexity and code smells.

Here’s a scatter plot of that data (zoomed in):

scatter_plot

Over the coming weeks, we’ll improve the graphs we have and add new graphs that expose interesting trends. But we need your help! Please let us know if you spot problems, have ideas for new graphs, or have any questions. Additionally, please add your project to Caliper so it can be included in our community statistics. Finally, feel free to grab the raw stats from our alpha API* and play around yourself!

* Quick summary:

curl http://api.devver.net/metrics

for JSON,

curl -H 'Accept:text/x-yaml' http://api.devver.net/metrics

for YAML. More details. API is under development, so please send us feedback!

Advertisements

Written by Ben

November 19, 2009 at 8:37 am

Posted in Development, Ruby, Tools

Tagged with ,

6 Responses

Subscribe to comments with RSS.

  1. […] Announcing Caliper Community Statistics – The Devver Blog […]

  2. Thanks for a very interesting value add service for open source projects.
    What to do with several branches of a project?
    – live with the multiple counting (bias the community profile)
    – eliminate duplicates (how to identify the root/parent)
    – treat branches as a group, report one set of aggergated stats, but allow the branches to be viewed as a sub-community?

    Hedge

    November 24, 2009 at 9:22 pm

  3. Is it possible to filter the data to show only a certain github user's projects (eg agrimm)?

    I'm not sure how hard this would be to implement, but sometimes logging both axes of a graph makes it easier to look at.

    andrewjgrimm

    November 25, 2009 at 3:24 pm

  4. Hedge,

    It's a good question. Right now, forks are double (or triple, etc) counted. That's clearly not ideal. I'll have to think about it more, but my first thought is to aggregate the stats as a group for the purpose of community metrics. I'll make a story to look into this.

    Thanks!
    Ben

    bhb

    November 25, 2009 at 6:14 pm

  5. Andrew,

    Unfortunately, neither of those features are available right now, but I'll make a story for each one. Thanks for the great suggestions!

    Ben

    bhb

    November 25, 2009 at 6:16 pm

  6. From one A. Grimm to another, I agree that being able to look at a subset based on a search would be cool 🙂 I'll put a story in for it.

    Avdi

    November 26, 2009 at 8:14 am


Comments are closed.

%d bloggers like this: