The Devver Blog

A Boulder startup improving the way developers work.

Archive for the ‘Amazon Web Services’ Category

SimpleDB DataMapper Adapter: Progress Report

From the beginning of Devver, we decided we wanted to work with some new technologies and we wanted to be able to scale easily. After looking at options AWS seemed to have many technologies that could help us build and scale a system like Devver. One of these technologies was SimpleDB. One of the other new things we decided to try was DataMapper (DM) rather than the more familiar ActiveRecord. This eventually let me to work on my own SimpleDB DataMapper adapter.

Searching for ways to work with SDB using Ruby, we found a SimpleDB DM adapter by Jeremy Boles. It worked well initially but as our needs grew (and to make it compatible with the current version of DM) it became necessary to add and update the features of the adapter. These changes lived hidden in our project’s code for awhile, for no other reason than we were too lazy to really commit it all back on GitHub. Recently though there has been a renewed interest about working with on SimpleDB with Ruby. I started pushing the code updates on GitHub, then I got a couple requests and suggestions here and there to improve the adapter. One of these suggestions cam from Ara Howard, who is doing impressive work of his own on Ruby and AWS, specifically SimpleDB. His suggestion on moving from the aws_sdb gem to right_aws, which along with other changes improved performance significantly (1.6x on write, up to 36x on reading large queries over the default limit of 100 objects). Besides performance improvements, we have recently added limit and sorting support to the adapter.

#new right_aws branch using AWS select
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
 1.020000   0.240000   1.260000 ( 35.715608)
Finding all users age 25 (all of them), 100 Times
 59.280000   8.640000  67.920000 ( 99.727380)

#old aws_sdb using query with attributes
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
  1.290000   0.530000   1.820000 ( 52.916103)
Finding all users age 25 (all of them), 100 Times
  356.640000  53.090000 409.730000 (3574.260988)

view this gist

As I added features, testing the adapter also became slow, (over a minute a run) because the functional tests actually connect to and use SimpleDB. Since Devver is all about speeding up Ruby tests, I decided to get the tests running on Devver. It was actually very easy and sped up the test suite from 1 minute and 8 seconds down to 28 seconds. You can check out how much Devver speeds up the results yourself.

We are currently using the SimpleDB adapter to power our Devver.net website as well as the Devver backend service. It has been working well for us, but we know that it doesn’t cover everyone’s needs. Next time you are creating a simple project, give SimpleDB a look, we would love feedback about the DM adapter, and it would be great to get some other people contributing to the project. If anyone does fork my SDB Adapter Github repo, feel free to send me pull requests. Also, let me know if you want to try using Devver as you hack on the adapter, it can really speed up testing, and I would be happy to give out a free account.

Lastly, at a recent Boulder Ruby users group meet up, the group did a code review for the adapter. It went well and I should finish cleaning up the code and get the improvements suggested by the group committed to GitHub soon.

Update: The refactorings suggested at the code review are now live on GitHub.

Advertisements

Written by DanM

June 22, 2009 at 11:27 am

Managing Amazon EC2 with your iPhone

I wanted a quick way when out and about to easily manage our AWS EC2 instances while out and about. It hasn’t happened often, but occasionally I am away from the computer and I need to reboot the instances. Perhaps I remember our developer cluster isn’t being used and want to shut it down to save some money.

I didn’t find anything simple and free with a quick Google search, so in a about an hour I wrote a nice little Sinatra app that will let me view our instances, shutdown, or reboot any specific instance or all of them. The tiny framework actually turned out to be even more useful as I now have options that let us tail error logs, reboot Apache, reboot mongrel clusters, or execute any common system administration task.

I won’t be going into detail on how to build a iPhone webapp using Sinatra and iUI, because Ben already created an excellent post detailing all of those steps. In fact I used his old project as the template when I created this project. I can’t begin to explain how amazingly simple it is to build an iPhone webapp using Sinatra, so if you have been thinking of a quick project I highly recommend it.

Here are some screen shots showing the final app. (screenshot courtesy of iPhoney):

ec2 manager home view

ec2 manager home view.

ec2 manager describe view

ec2 manager describe instances view.

ec2 manager instance view.

ec2 manager instance view.

This app uses the Amazon EC2 API Tools to do all the heavy lifting. So this app assumes that you already have the tools installed and working on the machine you want this app to run on. This normally involves installing the tools and setting up some environment variables like EC2_HOME, so make sure you can run ec2-describe-instances from the machine. After that you should just have to change EC2_HOME in the Sinatra app to match the path where you installed the EC2 tools.

Let me know if you have any issues, it is quick and dirty, but I have already found it useful.

To run the app:
cmd> ruby -rubygems ./ec2_manager.rb

require 'sinatra'

EC2_HOME = '~/.ec2'

use Rack::Auth::Basic do |username, password|
  [username, password] == ['some_user', 'some_pass']
end

get "/" do
  @links = %w{describe_ec2s restart_all_ec2s shutdown_all_ec2s}.map { |cmd|
    cmd_link(cmd)
  }.join
  erb :index
end

get "/describe_ec2s" do
  results = `cd #{EC2_HOME}; ec2-describe-instances`
  instances = results.scan(/INSTANCE\ti-\w*/).each{|i| i.sub!("INSTANCE\t",'')}
  @links = instances.map { |i|
    instance_link(i)
  }.join
  erb :index
end

get "/restart_all_ec2s" do
  @results = `cd #{EC2_HOME}; ec2-describe-instances`
  instances = @results.scan(/INSTANCE\ti-\w*/).each{|i| i.sub!("INSTANCE\t",'')}
  cmd="cd #{EC2_HOME}; ec2-reboot-instances #{instances.join(' ')}"
  @results = `cmd`
  erb :index
end

get "/shutdown_all_ec2s" do
  @results = `cd #{EC2_HOME}; ec2-describe-instances`
  instances = @results.scan(/INSTANCE\ti-\w*/).each{|i| i.sub!("INSTANCE\t",'')}
  cmd="cd #{EC2_HOME}; ec2-terminate-instances #{instances.join(' ')}"
  @results = `cmd`
  erb :index
end

get "/instance/:id" do
  id = params[:id] if params[:id]
  verify_id(id)
  @results = `cd #{EC2_HOME}; ec2-describe-instances #{id}`
  @links = "<li><a href='/shutdown/#{id}' target='_self'>shutdown #{id}</a></li>"
  @links += " <li><a href='/reboot/#{id}' target='_self'>reboot #{id}</a></li>"
  erb :index
end

get "/reboot/:id" do
  id = params[:id] if params[:id]
  verify_id(id)
  @results = `cd #{EC2_HOME}; ec2-reboot-instances #{id}`
  erb :index
end

get "/shutdown/:id" do
  id = params[:id] if params[:id]
  verify_id(id)
  @results = `cd #{EC2_HOME}; ec2-terminate-instances #{id}`
  erb :index
end

helpers do

  def cmd_link(cmd)
    "<li><a href='#{cmd}' target='_self'>#{cmd}</a></li>"
  end

  def instance_link(instance)
    "<li><a href='/instance/#{instance}' target='_self'>#{instance}</a></li>"
  end

  def verify_id(id)
    raise Sinatra::ServerError, 'bad-id, What you doin?' unless id.match(/i-\w*/)
  end

end

use_in_file_templates!

__END__

@@ index



@import "/stylesheets/iui.css";




<div class="toolbar">
<h1 id="pageTitle"></h1>
</div>


<ul id="home">
<li><a href='/' target='_self'>home</a></li>


</ul>





<li><strong>results</strong></li>

<ul id="home">
<li><a href='/' target='_self'>home</a></li>

&lt;%= @results.gsub(&quot;\n&quot;,&quot;<br />") %&gt;
</ul>




view this gist

Written by DanM

March 5, 2009 at 10:03 am

Boulder.Me: An interesting way to get employees

Boulder has a lot going on in terms of tech startups. A bunch of Boulder startups got together for a pretty interesting plan. A group of startups will be flying 100 interviewees out to Boulder to interview potential employees and let them get a feel for the city. There has already been some pretty good coverage of the plan on TechCrunch and elsewhere.

So if you are interested in getting a great job with some of the awesome startups in Boulder, I highly recommend checking out and applying to Boulder.Me.

Written by DanM

September 27, 2008 at 2:23 pm

Ruby Messaging Shootout

We have spent a bit of time looking into various Ruby messaging systems. We briefly posted about the speed of Ruby messaging in the past and promised some more detailed numbers. We will share a bit of code to run some basic tests on various Ruby messaging systems, and benchmark the performance. We are sure you can do more to get even more accurate results, but these were good enough for our purposes and we thought we should share them.

We decided that we would place and take X messages through each messaging system. Our baseline for the best performance was a standard Ruby in-memory queue. In an effort to reduce the effects of initial loading, we ran a small staging with each of the queues, and threw away the first set of results. We ended up comparing SQS, Starling, and Beanstalk. In the comparisons we ran tests using local queue servers running on the same machine as the tests (LocalStarlingQueue, BeanstalkClient). We also ran the queue servers remotely on an EC2 server (BeanstalkClientRemote, RemoteStarlingQueue). We are only showing results for 10 SQS messages, because it was so slow with any sizable amount of messages. We quickly found that both Beanstalk and Starling were faster than SQS by over 10x, on the remote servers and insanely faster on local servers. Surprisingly running Starling and Beanstalk between multiple EC2 instances is almost as fast as having local queue servers, and completely puts SQS in the dust (1000x faster).

When we started running larger tests we found some interesting results. Compare the last two sets of test in which we run remote queue servers. The first set shows that Starling is faster by a decent amount mostly because taking messages off the queue is significantly faster. This is actually because on Starling removing an item from a queue is a operation, while on Beanstalk it is two operations get and remove. This made Starling seem better for our needs since in our app any job taken should be considered completed. Once moving to messaging between EC2 instances though we can see that the overhead of the multiple messages disappears because the internal network is very fast between EC2 machines.

The speed of Beanstalk between EC2 was one of the reasons we eventually went with it as our messaging choice. The other isn’t really shown in our results here, but running tests on Starling over time show that the system begins to slow down with use. I am assuming that this is related to Starling persisting queues to disk, and the overhead related to persistence. In fact I found that if I killed the Starling server and cleared its persistent storage, Starling would return to its original performance. We currently have no need for persisting our queues, so taking a performance hit to support that feature was the final reason we went with Beanstalk.

We also looked at ActiveMessaging, but ultimately decided not to take the time to implement and test it. I would love to see others take our messaging test harness and put other messaging systems through the paces. We have decided to share the code we used to generate the results you see below, feel free to contact us if you have any questions, or if you add any other queues to the tests.

Download Ruby Messaging Tests

UPDATE: I updated the results to include runs with 10,000 and finally 100,000 messages, because there was some interest in seeing those numbers. There is some interesting discussion about this going on at the beanstalk Google group, about the results.

Running 10 messages on all systems
Queue type               user     system      total        real
MemoryQueue:             0.000000   0.000000   0.000000 (  0.000067)
LocalStarlingQueue:      0.000000   0.010000   0.010000 (  0.015040)
BeanstalkClient:         0.010000   0.000000   0.010000 (  0.005700)
SQS1:                    0.120000   0.050000   0.170000 ( 10.608450)
BeanstalkClientRemote:   0.020000   0.030000   0.050000 (  4.263844)
RemoteStarlingQueue:     0.010000   0.020000   0.030000 (  3.366750)

MemoryQueue::::: mean time for put : 0.000020 std dev for put: 0.000014 mean time for take: 0.000013 std dev for take: 0.000001 put mean is 1.0 slower than MemoryQueue take mean is 1.0 slower than MemoryQueue LocalStarlingQueue::::: mean time for put : 0.000867 std dev for put: 0.0002 mean time for take: 0.001340 std dev for take: 0.000639 put mean is 43.3659117997616 slower than MemoryQueue take mean is 101.259459459459 slower than MemoryQueue BeanstalkClient::::: mean time for put : 0.000166 std dev for put: 0.000026 mean time for take: 0.000288 std dev for take: 0.000020 put mean is 8.2777115613826 slower than MemoryQueue take mean is 21.7963963963964 slower than MemoryQueue SQS1::::: mean time for put : 1.081836 std dev for put: 2.264929 mean time for take: 0.670880 std dev for take: 0.045992 put mean is 54082.8140643623 slower than MemoryQueue take mean is 50700.4522522523 slower than MemoryQueue BeanstalkClientRemote::::: mean time for put : 0.101100 std dev for put: 0.007356 mean time for take: 0.202382 std dev for take: 0.012278 put mean is 5054.16805721097 slower than MemoryQueue take mean is 15294.6 slower than MemoryQueue RemoteStarlingQueue::::: mean time for put : 0.111742 std dev for put: 0.008853 mean time for take: 0.155392 std dev for take: 0.110507 put mean is 5586.18355184744 slower than MemoryQueue take mean is 11743.4378378378 slower than MemoryQueue 100 messages, Remote Queue servers on EC2 (client and tests running locally) Queue type user system total real MemoryQueue: 0.000000 0.000000 0.000000 ( 0.000165) BeanstalkClientRemote: 0.130000 0.250000 0.380000 ( 33.909095) RemoteStarlingQueue: 0.080000 0.170000 0.250000 ( 22.677569)
MemoryQueue::::: mean time for put : 0.000015 std dev for put: 0.000006 mean time for take: 0.000013 std dev for take: 0.000002 put mean is 1.0 slower than MemoryQueue take mean is 1.0 slower than MemoryQueue BeanstalkClientRemote::::: mean time for put : 0.113252 std dev for put: 0.004145 mean time for take: 0.227982 std dev for take: 0.031950 put mean is 7479.36954810266 slower than MemoryQueue take mean is 17757.1864438254 slower than MemoryQueue RemoteStarlingQueue::::: mean time for put : 0.132354 std dev for put: 0.188740 mean time for take: 0.112539 std dev for take: 0.003803 put mean is 8740.89418989135 slower than MemoryQueue take mean is 8765.5156917363 slower than MemoryQueue 100 messages, Remote Queue servers on EC2 (client and tests running on a separate EC2 instance) Queue type user system total real MemoryQueue: 0.020000 0.000000 0.020000 ( 0.006392) BeanstalkClientRemote: 0.010000 0.000000 0.010000 ( 0.793841) RemoteStarlingQueue: 0.010000 0.000000 0.010000 ( 1.067932)
MemoryQueue::::: mean time for put : 0.000009 std dev for put: 0.000102 mean time for take: 0.000030 std dev for take: 0.000769 put mean is 1.0 slower than MemoryQueue take mean is 1.0 slower than MemoryQueue BeanstalkClientRemote::::: mean time for put : 0.000353 std dev for put: 0.002849 mean time for take: 0.000527 std dev for take: 0.002545 put mean is 38.4510216814849 slower than MemoryQueue take mean is 17.5400444232905 slower than MemoryQueue RemoteStarlingQueue::::: mean time for put : 0.000773 std dev for put: 0.004554 mean time for take: 0.000805 std dev for take: 0.006630 put mean is 84.2330369677118 slower than MemoryQueue take mean is 26.7816119308266 slower than MemoryQueue 10,000 messages, Remote Queue servers on EC2 (client and tests running on a separate EC2 instance) Queue type user system total real MemoryQueue: 0.040000 0.000000 0.040000 ( 0.127432) BeanstalkClientRemote: 0.390000 0.090000 0.480000 ( 7.646054) RemoteStarlingQueue: 0.070000 0.020000 0.090000 ( 10.685410)
MemoryQueue::::: mean time for put : 0.000024 std dev for put: 0.001314 mean time for take: 0.000015 std dev for take: 0.000685 put mean is 1.0 slower than MemoryQueue take mean is 1.0 slower than MemoryQueue BeanstalkClientRemote::::: mean time for put : 0.000283 std dev for put: 0.002114 mean time for take: 0.000526 std dev for take: 0.002925 put mean is 11.7913700313271 slower than MemoryQueue take mean is 35.8050617861459 slower than MemoryQueue RemoteStarlingQueue::::: mean time for put : 0.000602 std dev for put: 0.004539 mean time for take: 0.000546 std dev for take: 0.004831 put mean is 25.0511140001964 slower than MemoryQueue take mean is 37.2107648228277 slower than MemoryQueue 100,000 messages, Remote Queue servers on EC2 (client and tests running on a separate EC2 instance) Queue type user system total real MemoryQueue: 0.260000 0.040000 0.300000 ( 0.677368) BeanstalkClientRemote: 3.200000 0.940000 4.140000 ( 76.989950) RemoteStarlingQueue: 0.820000 0.240000 1.060000 (110.507879)
MemoryQueue::::: mean time for put : 0.000019 std dev for put: 0.000915 mean time for take: 0.000018 std dev for take: 0.001125 put mean is 1.0 slower than MemoryQueue take mean is 1.0 slower than MemoryQueue BeanstalkClientRemote::::: mean time for put : 0.000274 std dev for put: 0.002125 mean time for take: 0.000531 std dev for take: 0.003302 put mean is 14.6932435272862 slower than MemoryQueue take mean is 30.0050748059956 slower than MemoryQueue RemoteStarlingQueue::::: mean time for put : 0.000592 std dev for put: 0.006346 mean time for take: 0.000577 std dev for take: 0.004349 put mean is 31.7037894886494 slower than MemoryQueue take mean is 32.5781596463523 slower than MemoryQueue

Written by DanM

June 30, 2008 at 8:44 am

Speed with Messaging Matters

Devver started with a good idea (web-based tools for Ruby
hackers) and a working prototype, which couldn’t scale. We started working on making Devver scalable and decided to go with EC2, from Amazon. Unfortunately we quickly learned that a library, Rinda, which we built our messaging system on, couldn’t connect between multiple EC2 instances.

No worries, we thought, Amazon has its own queuing system, which can be used for messaging, SQS. We thought since Devver was on Amazon’s EC2, it maked sense to switch to Amazon’s Queues. We first refactored our code and isolated our old messaging away from the rest of the system. Then we we switched over to SQS which we noticed was slower. Since, Devver takes developer tools into the cloud, we have to run our tools against various projects to verify correct functionality and performance. Running Devver against on our test and toy projects, SQS was slower, but it was working out OK.

After building in a stats collection system allowing us to start charting our progress against real metrics, we started running larger projects on Devver. We quickly learned that Devver suddenly couldn’t handle or deal with any real projects. Using Devver’s metrics to guide us, we isolated the problem down to messaging.

We found out that Devver was spending approximately 60 seconds messaging our distributed machines, for every 6 seconds of work they would do. The round trip of messaging through Devver made the entire project useless. Knowing this wasn’t a problem in our Rinda based prototype, we decided Devver just needed a better messaging system. Since we had just refactored the code, the messaging system part was very isolated from the rest of the system and it was easy to drop in a new solution.

While Amazon SQS is robust and reliable, fast it is not (.4 seconds put/take a message remotely, .04 within AWS (on EC2)). We did some research and found many different messaging systems for Ruby, and held a Ruby messaging shootout. This post isn’t about what system we eventually went with, because I am sure many of the options would have been fine. I mostly want to share a story of using metrics to pinpoint the worst offender, and isolating parts of your application, so that in a single afternoon you can replace a highly critical part of a project. Changing out one component that is heavily relied on , especially if there is a flaw or a bottleneck, can be the difference between success and failure.

A quick example of the performance changes we saw from swapping out our messaging:

Devver processing Mocha with SQS, 63.8-80.8 seconds
Devver processing Mocha with new messaging, 3.9 seconds

That is it for this post, but look forward to some stats and our detailed review of Ruby messaging systems in the future.

Written by DanM

June 18, 2008 at 9:25 am

Using Ruby to configure EC2 instances: a lesson learned

On Tuesday, we had Mike Culver from Amazon give a great talk about Amazon’s web services. We had pretty much decided on using AWS for Devver, but Mike’s talk convinced us even more.

So for the past few days we’ve been porting our code to work with EC2 . We’ve also been building scripts that will configure the machine instances on boot.

When you’re using EC2, you can store custom machine images, but you really don’t really want to be saving a new image for every tweak you make to the machine. It’s a lot more flexible to just save a basic image and give the instance a script to run when it boots, like this:

ec2-run-instances [some-image-name]  -f config.rb

In this example, I’m passing a Ruby script, but you can pass any type of file at all.

I found a couple of great tutorials, but unfortunately they didn’t do quite what I wanted, so I hacked up my own solution. It’s just a few lines in the /etc/rc.local file (which runs after the machine boots) that grabs the config.rb file and runs it.

Unfortunately, for some reason, my Ruby script was not getting run. After a painfully long debugging session, I figured out that the environment had not been configured for RubyGems when my script was run, so my ‘require’ statements were breaking the code. The key lesson is that no matter what language you’re using, /etc/rc.local is running in a different environment than a logged-in user, so environment variables like PATH (and in this case, RUBY_OPT) will not be set up the same way.

Anyway, here’s the code I added to my /etc/rc.local file:

# AMI configuration code, based on code from
#http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1085&amp;categoryID=100

# grab the file from Amazon's web service. Yes, this IP address is always the same
/usr/bin/wget http://169.254.169.254/latest/user-data -O /tmp/payload.rb

# if wget error code is 0, there was no error
if [ $? -eq 0 ]; then
  if [ -e /tmp/payload.rb ]; then
    # DON'T forget the -rubygems option! You need it if you call 'require' in your config code
    /usr/bin/ruby -rubygems /tmp/payload.rb
  else
    echo "rc.local : No payload.rb script to run"
  fi
else
  echo "rc.local : error retrieving user data"
fi

I hope that helps anyone out there starting to play with EC2…

Written by Ben

June 6, 2008 at 10:31 pm