The Devver Blog

A Boulder startup improving the way developers work.

Speed with Messaging Matters

Devver started with a good idea (web-based tools for Ruby
hackers) and a working prototype, which couldn’t scale. We started working on making Devver scalable and decided to go with EC2, from Amazon. Unfortunately we quickly learned that a library, Rinda, which we built our messaging system on, couldn’t connect between multiple EC2 instances.

No worries, we thought, Amazon has its own queuing system, which can be used for messaging, SQS. We thought since Devver was on Amazon’s EC2, it maked sense to switch to Amazon’s Queues. We first refactored our code and isolated our old messaging away from the rest of the system. Then we we switched over to SQS which we noticed was slower. Since, Devver takes developer tools into the cloud, we have to run our tools against various projects to verify correct functionality and performance. Running Devver against on our test and toy projects, SQS was slower, but it was working out OK.

After building in a stats collection system allowing us to start charting our progress against real metrics, we started running larger projects on Devver. We quickly learned that Devver suddenly couldn’t handle or deal with any real projects. Using Devver’s metrics to guide us, we isolated the problem down to messaging.

We found out that Devver was spending approximately 60 seconds messaging our distributed machines, for every 6 seconds of work they would do. The round trip of messaging through Devver made the entire project useless. Knowing this wasn’t a problem in our Rinda based prototype, we decided Devver just needed a better messaging system. Since we had just refactored the code, the messaging system part was very isolated from the rest of the system and it was easy to drop in a new solution.

While Amazon SQS is robust and reliable, fast it is not (.4 seconds put/take a message remotely, .04 within AWS (on EC2)). We did some research and found many different messaging systems for Ruby, and held a Ruby messaging shootout. This post isn’t about what system we eventually went with, because I am sure many of the options would have been fine. I mostly want to share a story of using metrics to pinpoint the worst offender, and isolating parts of your application, so that in a single afternoon you can replace a highly critical part of a project. Changing out one component that is heavily relied on , especially if there is a flaw or a bottleneck, can be the difference between success and failure.

A quick example of the performance changes we saw from swapping out our messaging:

Devver processing Mocha with SQS, 63.8-80.8 seconds
Devver processing Mocha with new messaging, 3.9 seconds

That is it for this post, but look forward to some stats and our detailed review of Ruby messaging systems in the future.

About these ads

Written by DanM

June 18, 2008 at 9:25 am

3 Responses

Subscribe to comments with RSS.

  1. [...] have spent a bit of time looking into various Ruby messaging systems. We briefly posted about the speed of Ruby messaging in the past and promised some more detailed numbers. We will share a bit of code to run some basic [...]

  2. I am working with a new Message Queue appliance that is at the far high end of performance (8 Million msg/sec non-persistent, 200,000 msg/sec persistent). It works fine with a Java/JMS client, and has a propritary C API as well as a REST interface (XML over HTTP), but no Ruby support(yet). Any pointers for a Ruby newbie on quick and easy way to wrapper JMS and expose as Ruby Messaging?

    Hans

    July 8, 2008 at 12:28 pm

  3. I haven’t really done much with JMS, but if you have a server REST interface, I would write writing a simple REST client in ruby. Dealing with REST interfaces and simple XML is really easy in Ruby. You could take a look at any of Amazon’s AWS implementations to see how to easily build the REST requests in Ruby.

    Dan

    July 8, 2008 at 4:59 pm


Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: