The Devver Blog

A Boulder startup improving the way developers work.

Archive for September 2008

Ruby Test Quality Tools

Update: Devver now offers a hosted metrics service for Ruby developers which can give you useful feedback about your code. Check out Caliper, to get started with metrics for your project.

This is the second post in my series of Ruby tools articles. This time I am focused on Ruby test quality tools. Devver is always really interested in testing, and obviously the quality of a project’s tests is important. We are always looking at ways to add even more value to the investment teams put in with testing. Simply knowing that you are writing higher quality tests helps increase the value returned on the time invested in testing. I haven’t found many tools to help with test quality, but these tools are a great help to any Ruby tester.

Heckle


Heckle is an interesting tool to do mutation testing of your tests. Heckle currently supports Test:Unit and RSpec, but does have a number of issues. I had to run it on a few different files and methods before I got some useful output that helped me improve my testing. The first problem was it crashing when I passed it entire files (crashing the majority of the time). I then began passing it single methods I was curious about, which still occasionally caused Heckle to get into an infinite loop case. This is a noted problem in Heckle, but -T and providing a timeout should solve that issue. In my case it was actually not an infinite loop timing error, but an error when attempting to rewrite the code, which lead to a continual failure loop that wouldn’t time out. When I found a class and method that Heckle could test I got some good results. I found one badly written test case, and one case that was never tested. Lets run through a simple Heckle example.

#install heckle
dmayer$ sudo gem install heckle

#example of the infinite loop Error Heckle run
heckle Syncer should_be_excluded? --tests test/unit/client/syncer_test.rb -v
Setting timeout at 5 seconds.
Initial tests pass. Let's rumble.

**********************************************************************
***  Syncer#should_be_excluded? loaded with 13 possible mutations
**********************************************************************
...
2 mutations remaining...
Replacing Syncer#should_be_excluded? with:

2 mutations remaining...
Replacing Syncer#should_be_excluded? with:
... loops forever ...

#Heckle run against our Client class and the process method
dmayer$ heckle Client process --tests test/unit/client/client_test.rb
Initial tests pass. Let's rumble.

**********************************************************************
***  Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

--- original
+++ mutation

 def process(command)

   case command
   when @buffer.Ready then
     process_ready
-  when @buffer.SetID then
+  when nil then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
     @log.error("client ignoring invalid command #{command}") if @log
   end
 end

--- original
+++ mutation
 def process(command)
   case command
   when @buffer.Ready then
     process_ready
   when @buffer.SetID then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
-    @log.error("client ignoring invalid command #{command}") if @log
+    nil if @log
   end
 end

Heckle Results:

Passed    :   0
Failed    :   1
Thick Skin:   0

Improve the tests and try again.

#Tests added / changed to improve Heckle results
  def test_process_process_loop__random_result
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    client.log.expects(:error).with("client ignoring invalid command this is random")
    client.process("this is random")
  end

  def test_process_process_loop__set_id
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    cmd = DataBuffer.new.create_set_ids_msg("4")
    client.expects(:process_set_id).with(cmd)
    client.process(cmd)
  end

#A final Heckle run, showing successful results
dmayer$ heckle Client process --tests test/unit/client/client_test.rb
Initial tests pass. Let's rumble.

**********************************************************************
*** Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...
No mutants survived. Cool!

Heckle Results:

Passed : 1
Failed : 0
Thick Skin: 0

All heckling was thwarted! YAY!!!

rcov


rcov is a code coverage tool for Ruby. If you are doing testing you should probably be monitoring your coverage with a code coverage tool. I don't know of a better tool for code coverage than rcov. It is simple to use and generates beautiful, easy-to-read HTML charts showing the current coverage broken down by file. An easy way to make you project more stable is to occasionally spend some time increasing the coverage you have on your project. I have always found it a great way to get back into a project if you have been off of it for awhile. You just need to find some weak coverage points and get to work.
Rcov Screenshot
rcov screenshot

Written by DanM

September 30, 2008 at 9:57 am

Posted in Development, Ruby, Testing

Ruby Performance Tools

Update: We now offer a hosted metrics service for Ruby developers which can give you useful feedback about your code. Check out Caliper, to get started with metrics for your project.

I have been interested in all of the tools that exist for Ruby developers. I give you the first of a series of posts where I will be looking at some of the more interesting Ruby tools.

Recently a few different Rail performance tools / web services have been released. They are leaps and bounds better than just watching your developer log and trying to fix performance issues, or just tracking down slow MySQL queries. I went ahead and installed and worked with a few of these tools and wrote a bit about my thoughts and experience with each of them.

ruby-prof


ruby-prof does what every other profiler does, but it is much faster than the one built in to Ruby. It also makes it easy to output the information you are seeking to HTML pages, such as call graphs. If you are just looking for a simple write up to get started with ruby-prof I recommend the previous link. I will talk a little more about the kinds of problems I find and how I have solved them with ruby-prof.

I have used ruby-prof a number of times to isolate the ways to speed up my code. I haven’t used it to identify why an entire Rails application is slow (there are better tools I discuss later for that), but if you have a small but highly important piece of code ruby-prof is often the best way to isolate the problem. I used ruby-prof to identified the two slowest lines of code of a spellchecker, which was rewritten to become twice as fast.

Most recently I used it to identify where the code was spending all of its time in a loop for a file syncer. It turns out that for thousands of files each time through the loop we were continually calling Pathname.new(path).relative_path_from(@dir_path) over and over. Putting a small cache around that call essentially eliminated all delays in our file synchronization. Below is a simple example of how a few lines of code can make all the difference in performance and how easily ruby-prof can help you isolate the problem areas and where to spend your time. I think seeing the code that ruby-prof helped isolate, and the changes made to the code might be useful if you are new to profiling and performance work.

changes in our spellchecker / recommender

 #OLD Way
 alteration = []
    n.times {|i| LETTERS.each_byte {
        |l| alteration << word[0...i].strip+l.chr+word[i+1..-1].strip } }
 insertion = []
     (n+1).times {|i| LETTERS.each_byte {
        |l| insertion << word[0...i].strip+l.chr+word[i..-1].strip } }
 #NEW Way
    #pre-calculate the word breakups
    word_starts = []
    word_missing_ends = []
    word_ends = []
    (n+1).times do |i|
      word_starts << word[0...i]
      word_missing_ends << word[i+1..-1]
      word_ends << word[i..-1]
    end

 alteration = []
    n.times {|i|
      alteration = alteration.concat LETTERS.collect { |l|
        word_starts[i]+l+word_missing_ends[i] } }
 insertion = []
    (n+1).times {|i|
      insertion = insertion.concat LETTERS.collect { |l|

        word_starts[i]+l+word_ends[i] } }

Changes in our file syncer

#OLD
 path_name = Pathname.new(path).relative_path_from(@dir_path).to_s
 #NEW
 path_name = get_relative_path(path)

  def get_relative_path(path)
    return @path_cache[path] if @path_cache.member?(path)
    retval = Pathname.new(path).relative_path_from(@dir_path).to_s
    @path_cache[path] = retval
    return retval
  end

New Relic


New Relic is a performance monitoring tool for Rails apps. It has a great development mode that will help you track down performance issues before they even become a problem, and live monitoring so that you can find any hiccups that are slowing down the production application. The entire performance monitoring space for Ruby/Rails seems to be heating up. I guess it is easy to see why, when scaling has been such an issue for some Rails apps. Just playing around with New Relic was exciting and fun. I could quickly track down the slowest pages, and our most problematic SQL calls, in this case I was testing New Relic on Seekler (an old project of ours) since I didn’t think I would find much interesting on our current Devver site. Seekler had some glaring performance issues and I think if we had New Relic from the beginning we could have avoided many of them. Sounds like I might have a day project involving New Relic and giving Seekler as much of a performance boost as possible. New Relic turned out to be my favorite of the performance monitoring tools. For a much more detailed writeup check out RailsTips New Relic Review.

newrelic screenshot
New Relic screenshot

TuneUp


TuneUp another easy-to-install and use Rails performance monitoring solution. The problem I had with TuneUp was I couldn’t get it working on test app for these sorts of things. I tried running Seekler with TuneUp, but had no luck. I found that many people on the message boards seemed to be having various compatibility issues. I looked at the TuneUp screencast and the kind of information that they give you and I feel like this would be equal to New Relic if it works for you. I am emailing back and forth with FiveRuns support who have been very attentive and helpful, so if I get it working I will update this section.

Ruby Run


RubyRun provides live performance monitoring and debugging tools. I hadn’t ever heard of this product before I started doing some research while writing this blog article. I am sorry to say but this was the hardest to set up, and gave back less valuable information. I think they need a simple screencast on how to get set up and get useful information back. After getting setup and running I could only get ugly CSV reports that didn’t tell me much more than the regular Rails log files. I started reading the RubyRun Manual but it was about as long as Moby Dick and all I wanted was how to view simple easy-to-read reports which is a snap in New Relic and TuneUp. Since the site didn’t mention RubyRun providing better data than New Relic or TuneUp which were much more user friendly, I don’t think I would recommend RubyRun.

UPDATE: After reading about my difficulties with RubyRun the great folks from Rubysophic got in touch with me. They offered to help me get the tool working and posted a RubyRun quick start guide to their site. I got it working in a snap thanks to an email from their dev and the amazingly simple quick start guide. I still didn’t get the same depth of information that I got with New Relic, although RubyRun has a ton of settings so it is likely you can get more depth to the reports. Something worth pointing out is that RubyRun is working on Seekler, which I haven’t been able to get TuneUp running on. So if you have been having problems with TuneUp or New Relic, definitely give RubyRun a look. In the end I think the other offerings are slightly more user friendly (less complex settings), and easier to explore the data (link in the feed to both reports, at least when in developer mode). That being said RubyRun offers some great information and options that the others don’t and with a bit more UI tuning RubyRun would be at the top of the pack. Thanks to the helpful devs at Rubysophic for helping me to get the most out of RubyRun.

RubyRun screenshot
RubyRun screenshot
RubyRun second screen shot
screenshot of a different RubyRun report

If there are any other Ruby performance tools worth checking out let me know. Obviously, if you have experience with any of these tools I would love to hear your thoughts on them.

Written by DanM

September 28, 2008 at 2:35 pm

Boulder.Me: An interesting way to get employees

Boulder has a lot going on in terms of tech startups. A bunch of Boulder startups got together for a pretty interesting plan. A group of startups will be flying 100 interviewees out to Boulder to interview potential employees and let them get a feel for the city. There has already been some pretty good coverage of the plan on TechCrunch and elsewhere.

So if you are interested in getting a great job with some of the awesome startups in Boulder, I highly recommend checking out and applying to Boulder.Me.

Written by DanM

September 27, 2008 at 2:23 pm

Devver has Launched!

We’re happy to announce that on Thursday we officially launched with our first alpha customer!

Dan and I were both pretty nervous, but we also knew that this was an important step forward for Devver. I won’t say it went off without a hitch (we ran into a few bugs in the morning, which was pretty stressful). I don’t think we’ve ever watched log files with such interest and anticipation.

Thankfully, our alpha customers were awesome – patient, forgiving, and funny all at once. We got tons of great feedback from them in just in the first few hours alone. And, by the afternoon, Devver was up and running on their code base – in fact, we brought a test suite that was taking around forty minutes to complete on one machine down to two minutes on Devver.

All in all, it was a successful launch. It was a huge thrill to see a customer using Devver. I’ll leave you with a few real IM quotes from our customers:

“hurray! tests are zipping along.”

“I’m actually really excited to work with this.  I think it will really improve the process around here.”

And my personal favorite: “dots a’flowin’ “

Written by Ben

September 22, 2008 at 8:47 am

Posted in Devver, First Steps

Why Side Projects?

I just read a great post at Ekinoderm that nicely tied together two things Dan and I have been thinking and blogging about recently – the joys of side projects and why you should own your own IP. Check it out.

Written by Ben

September 5, 2008 at 3:46 pm

Posted in Development, Hacking

One Day of TDD

I am am a big believer in software testing. I normally have created tests after writing my code and mostly to ensure that regressions of functionality don’t occur when the code is changed. As I have become more comfortable with testing, and the changes it requires such as writing testable code, I have found even more benefits of testing. Better automated testing, and better understanding of testing has changed my development practices.

I haven’t practiced TDD, but I do follow Test Driven Corrections (TDC) which I might be coining right now. Following TDC means that when you find a bug, you should try to write a test that fails on that bug, then fix bug and make the test pass. I have become a fan of fixing bugs this way because bugs often first appear in code that for one reason or another is brittle or has some unseen dependency. If you just fix the bug even if it was a simple mistake it is still far more likely that there will be another bug with that piece of code than other areas. If you know a section of code is error prone wouldn’t you want to catch the error as fast as possible?

A new tool in my toolbox is exploratory testing. If I was learning a new library or object in the past I would often write simple programs that would print out the state, manipulate that state, and print the result. I would then continue to learn ways to work with the objects and verify that the printed state matched my expectations. Hmmm that seems error prone, and not necessarily repeatable. Now when I am learning something new I tend to write tests against my expectations of how things should work. A great recent example of this was when I was learning to use the RightScale AWS gem so I could access Amazon’s SimpleDB (SDB). I ended up writing tests to create, save, update, and delete objects. After that it was easy to write a real DB layer for some of our objects and write additional tests for that as well.

The more I learned about testing the more useful I have found it in many situations. The issue is that I have still never bought into the idea of TDD for most development. Even though many smart people have written about the benefits of TDD (Adam@Heroku Jay Fields), it just hasn’t ever seemed worth it to me. I decided that I really couldn’t knock it if I really hadn’t ever tried it for any significant amount of time. So I decided to spend one full day completely following TDD.

I was going to be adding some new features to Devver, and thought it would be a good chance to try my TDD challenge. The features I was adding were some small actions on the server that would be triggered remotely by the client. This broke down into a few separate pieces of functionality.

1. The client would send one of three new requests based on user input and existing client state
2. The server would receive and parse these new messages
3. The server would call 3 new handlers with the encoded project data and carry out tasks

Breaking this into tests was very natural and led to nearly no debugging time as almost the first time the client and server connected the interactions all behaved exactly as expected. I didn’t waste any time looking into where a message or a response got lost or wasn’t properly acknowledged in the code. The tests had already simulated the message creation, parsing, routing as well as the event handling and completion. It is nice when you put all the pieces together and it just works, and you know it is very solid from the beginning.

To break down the tasks I wrote the tests in this order.

1. Tested creating messages to store the expected project information
2. Tested parsing messages to get the expect project information
3. Tested client inputs would call my event handlers
4. Tested that client event handlers would send the proper message
5. Tested that the server would receive and parse the expected messages
6. Tested that the server would call my event handlers
7. Tested that event handler would complete the task expected off them

After spending a day and completing all the pieces of functionality I had expected to complete I was happy with my TDD experiment. I came away with a new respect for TDD an while I still don’t think it would be well suited to all programming tasks, I can certainly see a place for it and plan on using TDD more in the future. I do think that it took me slightly longer to complete the features than it normally would have. I freely admit that the more often you do TDD development the better you would get and that likely less time would be wasted trying to think up the proper test cases. I felt that the code I wrote under TDD was of a higher quality than the code I normally write. It forced me to refactor and rework my code as I went as well as break it into small enough pieces that I could write simple tests to verify the next piece of the project. I think that the code I ended up writing will be less brittle and easier to work with in the future. As a developer, I was happier and more sure of the stability of the features I just added to the system.

I think TDD is especially well suited to situations with small communications between systems, as each independent system can be completely tested and have it’s behavior verified while isolated from the other pieces. I was surprised that working in a way that demanded more upfront costs before I wrote anything functional didn’t slow me down more. I was expecting that it would make my development process slower by a factor of two, but the truth is that a single nasty bug halting your forward progress can take up more time than you would have needed to spend initially if working with a TDD approach.

I don’t plan on trying to move over to an entirely TDD approach but by challenging myself to work in a way that seemed unintuitive to me, I ended up learning a lot and likely will use the approach in the future for myself.

Written by DanM

September 4, 2008 at 3:19 pm

Posted in Development, Ruby, Testing

Follow

Get every new post delivered to your Inbox.