The Devver Blog

A Boulder startup improving the way developers work.

Posts Tagged ‘SimpleDB

Announcing DataMapper/SimpleDB Adapter 1.0.0

Today we are pleased to announce the release of dm-adapter-simpledb 1.0.0.

The big new feature is compatibility with DataMapper 0.10.*. The DataMapper adapter API changed significantly in 0.10, necessitating a broad rewrite. This release breaks backwards compatibility with DataMapper 0.9.*.

In addition, the way nils are stored has changed, so datasets created with the 0.9.* adapter may need to be updated. Previously nils were stored in the form of a special nil value (by default, the string “nil”). Starting with version 1.0.0, any attributes set to nil will be removed from the SimpleDB record. This is more in line with the SimpleDB concept of NULL, and enables

IS NOT NULL

predicates to work as expected.

New and existing SimpleDB adapter features include:

  • NEW: Supports all DataMapper query predicates.
  • NEW: DataMapper identity map support for record caching
  • NEW: Lazy-loaded attributes
  • NEW: DataMapper Serial property support via UUIDs.
  • Uses the RightAWS gem for efficient SimpleDB operations.
  • Full set of CRUD operations
  • Can translate many queries into efficient native SELECT operations.
  • Migrations
  • Array properties
  • Basic aggregation support (Model.count(“…”))
  • String “chunking” permits attributes to exceed the 1024-byte limit

If you have Gemcutter in your Gem sources, you can install the gem immediately:

gem install dm-adapter-simpledb

Check out the code at http://github.com/devver/dm-adapter-simpledb. You can see metrics for the code here. Give it a try, and feel free to send us any feedback, patches, and suggestions. The project’s issue tracker is on GitHub, or you can always email us.

Advertisements

Written by avdi

November 16, 2009 at 12:45 pm

Posted in Uncategorized

Tagged with ,

SimpleDB DataMapper Adapter: Progress Report

From the beginning of Devver, we decided we wanted to work with some new technologies and we wanted to be able to scale easily. After looking at options AWS seemed to have many technologies that could help us build and scale a system like Devver. One of these technologies was SimpleDB. One of the other new things we decided to try was DataMapper (DM) rather than the more familiar ActiveRecord. This eventually let me to work on my own SimpleDB DataMapper adapter.

Searching for ways to work with SDB using Ruby, we found a SimpleDB DM adapter by Jeremy Boles. It worked well initially but as our needs grew (and to make it compatible with the current version of DM) it became necessary to add and update the features of the adapter. These changes lived hidden in our project’s code for awhile, for no other reason than we were too lazy to really commit it all back on GitHub. Recently though there has been a renewed interest about working with on SimpleDB with Ruby. I started pushing the code updates on GitHub, then I got a couple requests and suggestions here and there to improve the adapter. One of these suggestions cam from Ara Howard, who is doing impressive work of his own on Ruby and AWS, specifically SimpleDB. His suggestion on moving from the aws_sdb gem to right_aws, which along with other changes improved performance significantly (1.6x on write, up to 36x on reading large queries over the default limit of 100 objects). Besides performance improvements, we have recently added limit and sorting support to the adapter.

#new right_aws branch using AWS select
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
 1.020000   0.240000   1.260000 ( 35.715608)
Finding all users age 25 (all of them), 100 Times
 59.280000   8.640000  67.920000 ( 99.727380)

#old aws_sdb using query with attributes
$ ruby scripts/simple_benchmark.rb
      user     system      total        real
creating 200 users
  1.290000   0.530000   1.820000 ( 52.916103)
Finding all users age 25 (all of them), 100 Times
  356.640000  53.090000 409.730000 (3574.260988)

view this gist

As I added features, testing the adapter also became slow, (over a minute a run) because the functional tests actually connect to and use SimpleDB. Since Devver is all about speeding up Ruby tests, I decided to get the tests running on Devver. It was actually very easy and sped up the test suite from 1 minute and 8 seconds down to 28 seconds. You can check out how much Devver speeds up the results yourself.

We are currently using the SimpleDB adapter to power our Devver.net website as well as the Devver backend service. It has been working well for us, but we know that it doesn’t cover everyone’s needs. Next time you are creating a simple project, give SimpleDB a look, we would love feedback about the DM adapter, and it would be great to get some other people contributing to the project. If anyone does fork my SDB Adapter Github repo, feel free to send me pull requests. Also, let me know if you want to try using Devver as you hack on the adapter, it can really speed up testing, and I would be happy to give out a free account.

Lastly, at a recent Boulder Ruby users group meet up, the group did a code review for the adapter. It went well and I should finish cleaning up the code and get the improvements suggested by the group committed to GitHub soon.

Update: The refactorings suggested at the code review are now live on GitHub.

Written by DanM

June 22, 2009 at 11:27 am