The Devver Blog

A Boulder startup improving the way developers work.

SimpleDB DataMapper Adapter: Progress Report

From the beginning of Devver, we decided we wanted to work with some new technologies and we wanted to be able to scale easily. After looking at options AWS seemed to have many technologies that could help us build and scale a system like Devver. One of these technologies was SimpleDB. One of the other new things we decided to try was DataMapper (DM) rather than the more familiar ActiveRecord. This eventually let me to work on my own SimpleDB DataMapper adapter.

Searching for ways to work with SDB using Ruby, we found a SimpleDB DM adapter by Jeremy Boles. It worked well initially but as our needs grew (and to make it compatible with the current version of DM) it became necessary to add and update the features of the adapter. These changes lived hidden in our project’s code for awhile, for no other reason than we were too lazy to really commit it all back on GitHub. Recently though there has been a renewed interest about working with on SimpleDB with Ruby. I started pushing the code updates on GitHub, then I got a couple requests and suggestions here and there to improve the adapter. One of these suggestions cam from Ara Howard, who is doing impressive work of his own on Ruby and AWS, specifically SimpleDB. His suggestion on moving from the aws_sdb gem to right_aws, which along with other changes improved performance significantly (1.6x on write, up to 36x on reading large queries over the default limit of 100 objects). Besides performance improvements, we have recently added limit and sorting support to the adapter.

#new right_aws branch using AWS select
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
 1.020000   0.240000   1.260000 ( 35.715608)
Finding all users age 25 (all of them), 100 Times
 59.280000   8.640000  67.920000 ( 99.727380)

#old aws_sdb using query with attributes
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
  1.290000   0.530000   1.820000 ( 52.916103)
Finding all users age 25 (all of them), 100 Times
  356.640000  53.090000 409.730000 (3574.260988)

view this gist

As I added features, testing the adapter also became slow, (over a minute a run) because the functional tests actually connect to and use SimpleDB. Since Devver is all about speeding up Ruby tests, I decided to get the tests running on Devver. It was actually very easy and sped up the test suite from 1 minute and 8 seconds down to 28 seconds. You can check out how much Devver speeds up the results yourself.

We are currently using the SimpleDB adapter to power our Devver.net website as well as the Devver backend service. It has been working well for us, but we know that it doesn’t cover everyone’s needs. Next time you are creating a simple project, give SimpleDB a look, we would love feedback about the DM adapter, and it would be great to get some other people contributing to the project. If anyone does fork my SDB Adapter Github repo, feel free to send me pull requests. Also, let me know if you want to try using Devver as you hack on the adapter, it can really speed up testing, and I would be happy to give out a free account.

Lastly, at a recent Boulder Ruby users group meet up, the group did a code review for the adapter. It went well and I should finish cleaning up the code and get the improvements suggested by the group committed to GitHub soon.

Update: The refactorings suggested at the code review are now live on GitHub.

Written by DanM

June 22, 2009 at 11:27 am

12 Responses

Subscribe to comments with RSS.

  1. I'm curious, if you write an application using this adapter, would that make it fairly easy to swap SimpleDB for another data store? Like if you moved off of EC2 and therefore also SimpleDB?

    charlieok39431

    June 22, 2009 at 12:19 pm

  2. Yeah that is one reason we are trying to limit our usage of SimpleDB to only what we support through the DM adapter. We test this frequently as well, because all our local tests actually run through DM to SQLite, while we use SimpleDB for our Dev and Production environments.

    danmayer

    June 22, 2009 at 2:48 pm

  3. Have you looked at adding M/DB support to your adapter? M/DB is an Open Source clone of SimpleDB. It should simply be a matter of allowing a different endpoint URL to be set, and thereafter M/DB should appear to behave identically to SimpleDB. See http://www.mgateway.com/mdb.html

    Rob

    June 23, 2009 at 10:51 pm

  4. Good call, I think the old version allowed you to pass in base_url, but currently we don't allow users to override any of the settings. I should make it easy to override all the settings that are used for logging and such. Thanks I will try to get that in on the next release.

    I think there is also another similar Open Source project called SimplerDB as well, so there might be some interest from those users as well.

    danmayer

    June 23, 2009 at 11:10 pm

  5. OK I added support for passing in variables like enpoints and logs. So it should work with M/DB. I also merged all that work back into Master if you try it out let me know how it goes.

    danmayer

    July 10, 2009 at 2:18 pm

  6. Are there any changes required to run your adapter with DM 0.10?

    Andres Orta

    October 31, 2009 at 12:10 pm

  7. There probably are changes required. I haven't tried it yet. If you try it out let me know if it works. If not I can try to get to updating it and making it work sometime soon.

    danmayer

    November 2, 2009 at 6:12 pm

  8. I actually can't get it to work. Running the benchmark gives me

    from /usr/local/lib/ruby/gems/1.8/gems/dm-core-0.10.1/lib/dm-core/model/property.rb:242:in `method_missing': undefined method `auto_migrate!' for Person:Class (NoMethodError)

    from simple_benchmark.rb:51

    auto_migrate! doesn't work when called on DataMapper either.

    Finally, when I remove the call to auto_migrate!, just to see what fails next, I get an error that the sdb variable inside the adapter is nil (hasn't been initialized).

    Thanks!

    A

    Andres Orta

    November 3, 2009 at 3:53 pm

  9. Andres, are you using DM 0.10? The SimpleDB adapter only supports DM 0.9. I'm working on an update for 0.10 (the adapter API changed significantly), but it's not quite ready yet.

    Avdi

    November 12, 2009 at 7:33 am

  10. Yes, I am using with DM 0.10… also working on an update to the adapter, but if you have something ready earlier, please let me know😉

    Andres

    November 12, 2009 at 7:48 am

  11. Just finished the port, here's the commit: http://github.com/avdi/dm-adapter-simpledb/commit

    Avdi

    November 15, 2009 at 3:55 pm

  12. I just saw this –thanks much, Avdi! I'll try it right away!

    Andres Orta

    November 23, 2009 at 7:19 am


Comments are closed.

%d bloggers like this: