The Devver Blog

A Boulder startup improving the way developers work.

Archive for the ‘Hacking’ Category

Revisiting additional Ruby Tools

I have heard about new Ruby tools since I did my Ruby Tools Roundup. I am always interested in tools that can help improve our code, so I had to check some of them out. Similar to my last tools post, I will be trying out a tool and writing my general impressions along with the basic usage.

reek


I have to start with reek, since it has been the most requested and searched on our site since I originally wrote about tools. reek will help identify code smells, allowing you to fix up your code. Instead of looking at cyclomatic complexity or other metrics, reek looks at patterns to warn you about bad code. Reek currently detects a few code smells (Long Method, Large Class, Feature Envy, Uncommunicative Name, Long Parameter List, Utility Function, Nested Iterators, Control Couple, Duplication) but more are on the way.

I think this project is useful but would need to be more customized before a nightly run would yield very useful results. The biggest problem I have is the signal to noise ratio seemed pretty high. Reek was warning me about “long methods” that were only 7 statements long, which just isn’t something I am concerned about. The warnings on duplicate methods calls can be useful, after running reek on a few files I found a couple places where duplicate method calls were wasting time. Many of the other smells are interesting like ‘Feature Envy’, and ‘Utility Function’. I will need to use reek more before I know if these smells are good indicators or often false positives.

Below reek finds a utility function next_tick which is definitely a helper function that actually exists in two of our files, which probably should be moved into a helper mixin.

def next_tick
    if(EM.reactor_running?)
      EM.next_tick do
        yield
      end
    else
      yield
    end
end

I am really looking forward to see how the tool progresses. If the project allows for a simple config customization to change the thresholds as well as ignore some files/smells, this could become a very useful tool to help keep a team maintain a high expectation of code quality. It would be useful to get nightly reports about any code that might not meet expectations, so a quick group code review could decide if it is an exception (which can be quickly added to the config) or if the code should be refactored and cleaned up.

dmayer$ sudo gem install reek
dmayer$ reek ./lib/client/client.rb
[Utility Function] Client#next_tick doesn't depend on instance state
[Long Method] Client#process_done has approx 7 statements
[Duplication] Client#process_ready calls @buffer.create_reload_msg more than once
[Long Method] Client#process_ready has approx 10 statements
[Duplication] Client#report_system_message calls result.msg more than once
[Feature Envy] Client#report_system_message refers to result more than self
[Duplication] Client#send_tests calls Time.now more than once
[Long Method] Client#send_tests has approx 24 statements
[Feature Envy] Client#send_tests refers to tests more than self
#check a whole directory
dmayer$ reek ./lib/client/*

Towelie


Towelie helps discover duplication in Ruby code, it will help keep your code DRY. It doesn’t have a nice interface at the moment and it is pretty young code. That being said, it can still be a really useful tool to help guide refactoring and code cleanup.

~/projects dmayer$ git clone git://github.com/gilesbowkett/towelie.git
dmayer$ cd ~/projects/devver/
dmayer$ irb -r ~/projects/towelie/lib/towelie.rb
irb(main):001:0> @t = Towelie.new
=> #, @model=#>
irb(main):002:0> @t.parse "lib/client"
(string):24: warning: useless use of a variable in void context
=> nil
irb(main):003:0> puts @t.duplicates
found in:
lib/client/test_unit_reporter.rb
lib/client/rspec_reporter.rb

def nl
report_nl
end

... 2 more dupes in the reporters ...

found in:
lib/client/test_unit_reporter.rb
lib/client/rspec_reporter.rb

def report(str)
print(str.to_s)
end

found in:
lib/client/sync_client.rb
lib/client/rev_sync_client.rb
lib/client/rev_client.rb
lib/client/client.rb

def quit
send(@buffer.create_quit_msg)
end

found in:
lib/client/sync_client.rb
lib/client/rev_sync_client.rb
lib/client/rev_client.rb
lib/client/client.rb

def send_quit
send(@buffer.create_quit_msg)
end

=> nil
irb(main):004:0>

There are currently many duplications because we are maintaining two clients while deciding what route to eventually take. We have also moved a lot of our shared client code into a mixin, and Towelie finds some methods that really should be moved there as well such as the methods “quit” and “send_quit”, which is currently duped in 4 files. Towelie also points to the fact that we should refactor our reporters because they both duplicate code.

I have always been annoyed with copied and pasted functions accidentally working its way in code, this could be a useful nightly run to keep a team DRY. Sometimes two team members implement the same functionality without even knowing a solution already exists in the code base. If you want to go a bit more in depth, check out Giles Bowkett’s (creator of Towelie) How to use Towelie

Flay


Flay is another great tool by Ryan Davis who also works on Heckle and Flog which I covered in the past. Flay, like Towelie, helps keep your code DRY, it detects exact and similar code throughout a project. It seems to be more powerful than Towelie, as seen in this Towelie and Flay comparison. My biggest complaint is the current release has some pretty basic output that you see below. The output I got from Towelie was immediately more recognizable and useful, while Flay currently requires you to dig in a bit deeper on your own into its suggestions. An improvement is already being worked on and a verbose output mode should be in the release soon. Once better output is included I think Flay will be immediately useful out of the box even with small amounts of developer effort.

I like that Flay has weight system, which should make it easy to set some threshold to ignore, high level weights are more likely to be worth your time and attention. One piece of code Flay tagged with a low weight was code that rescued and logged different errors thrown, which while similar actually served a purpose.

rescue Errno::EISDIR => ed
      @stderr.puts "Error: #{ed.message}" if @stderr
      @stderr.puts "You can't pass a directory to devver only test files. Quitting." if @stderr
      send_quit
    rescue LoadError => le
      @stderr.puts "Error: #{le.message}" if @stderr
      @stderr.puts "Not all of the files can be found. Quitting." if @stderr
      send_quit
    rescue SyntaxError, NameError => se
      @stderr.puts "Error: #{se.message}" if @stderr
      @stderr.puts "This file doesn't appear to be a valid Ruby file. Quitting." if @stderr
      send_quit
end

Digging into the Flay results turned up some duplicate code that Towelie had missed. Since Towelie also caught a method that was duped in 4 client files that Flay missed (I was expecting Towelie’s results to be a subset of what Flay found), perhaps there is room for both of the tools and learning to work with both a little bit is worth the time. After a little bit of work perhaps one of the projects will become a clearly better option. Until then I will be following both of these projects.

sudo gem install flay
dmayer$ flay lib/client/*.rb
Processing lib/client/client.rb...
Processing lib/client/mod_client.rb...
...
Processing lib/client/syncer.rb...

Matches found in :defn (mass = 84)
lib/client/mod_client.rb:86
lib/client/mod_rev_client.rb:124

Matches found in :block (mass = 57)
lib/client/client.rb:201
lib/client/client.rb:205
lib/client/client.rb:209

... 6 more results ...

Matches found in :if (mass = 34)
lib/client/mod_client.rb:63
lib/client/mod_rev_client.rb:111

Matches found in :defn (mass = 32)
lib/client/mod_rev_client.rb:36
lib/client/mod_rev_client.rb:50

Conclusions


That should cover it for this Ruby tools post, but I am really enjoying checking out the tools showing up in the Ruby scene. So as always let me know if I missed something, or if there is a tool you would like to see a full write up on. After some of the tools mature a little bit I will have to revisit a few of the tools which are currently in the early stages. I hope the Ruby tools scene keeps as active as it has been lately because there are some interesting projects being worked on.

honorable mentions (things I didn’t think really needed a full write up)


  • metric-fu a great gem to give quick access to a bunch of tools and metrics about your code (RCov, Saikuro, Flog, SCM Churn, and Rails Stats)
  • CruiseControl.rb when you start using all of these tools, continuous integration starts to become more important (or doing nightly runs). CruiseControl.rb is dead simple continuous integration.
  • Simian another code duplication tool, which is mentioned in 3 tools for drying your Ruby code (free for OSS, $99 for a license)
  • Ruby Tidy a tool for cleaning up HTML (I haven’t used this in Ruby, but loved the Java version in my Java days)
  • Watir is an open-source library for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.
  • Autotest, if you haven’t heard of autotest, check it out, continuously run your tests every time you save a file in your project.
  • Rufus a tool that checks if code you are about to load is safe. Allows you to look for custom patterns that you don’t want to run.
  • I wrote about a couple benchmarking tools last time and here is a great article / tutorial on Ruby benchmarking

Written by DanM

December 3, 2008 at 10:01 am

Building a iPhone web app in under 50 lines with Sinatra and iUI

One awesome thing about the iPhone is that it can display documents very nicely, including Word, Excel, and PDF files. However, the other day I was complaining that it’s not very easy to view the documents you store on your computer on your iPhone. Sure, you could email them to yourself, but then you have to search through your mail on your iPhone to find your documents. And I’m sure there is a snazzy iPhone app from the App Store to do this as well. But instead, let’s build a quick web app using Sinatra and iUI.

Here’s what we’ll be building (screenshot courtesy of iPhoney, which rules, by the way):

A screenshot of the butler iPhone web app

It doesn't look like much, but hey, it's less than 50 lines of code

When you click on git-tutorial.pdf, you'll see the full document

When you click on git-tutorial.pdf, you'll see this

Sinatra is a really awesome minimalist web framework. It lets you build web applications with just a few lines of code. iUI is a collection of JavaScript, CSS, and images that lets you easily make your web sites look great on the iPhone. Using these two tools, it’s really easy to build simple iPhone apps.

To begin, install the Sinatra gem:

$ gem install sinatra

Now, let’s start with a simplest version of our app, which we’ll call ‘butler’. Let’s make a directory for butler.

$ mkdir butler
$ cd butler
$ touch butler.rb

Open up butler.rb in your favorite editor and type:

require 'sinatra'
get "/" do
  "<h1>Your files, sir.</h1>"
end

view this gist

Now start butler on your command line:

 $ ruby -rubygems ./butler.rb 

and point your browser to http://localhost:4567 (you can use your computer’s browser or the one on your iPhone – it doesn’t matter. I find it’s better to use the one on my computer while building the app, since it’s easier to read Sinatra’s debugging messages if something goes wrong). You should see a page that just says “Your files, sir.” Congrats! You’ve made your first Sinatra app. Wasn’t that easy?

OK, let’s make butler a little more useful. Sinatra will serve up any files in a subdirectory named public. Since we’ll eventually be using this public directory for holding other JavaScript and CSS files as well, we’ll actually put our files in ./public/files. We’ll also make a link for convenience. Finally, while we’re at it, let’s put a few test files in there.

$ mkdir -p public/files
$ ln -s public/files files
$ echo "foo" > public/files/foo.txt
$ echo "bar" > public/files/bar.txt 

We want butler to link to each file, so let’s build a little helper for that. In Sinatra, you can include helpers within a helper block. We’ll also try out our helper for one file.

require 'sinatra'
require 'pathname'

get "/" do
  html = "<h1>Your files, sir.</h1>"
  dir = "./files/"
  html += file_link("./files/foo.txt")
  html
end

helpers do
  def file_link(file)
    filename = Pathname.new(file).basename
    "<a href='#{file}'>#{filename}</a><br />"
  end
end

view this gist

Go refresh your browser to see the changes. There’s no need to restart your application, because Sinatra automatically reloads changes (very cool!). You should see a link to foo.txt. Click on it, and you’ll see the contents.

Clearly, we don’t want to hardcode this for just one file. Let’s alter butler to look for every file within the ./files directory.

require 'sinatra'
require 'pathname'

get "/" do
  html = "<h1>Your files, sir.</h1>"
  dir = "./files/"
  Dir[dir+"*"].each do |file|
    html+=file_link(file)
  end
  html
end

helpers do

  def file_link(file)
    filename = Pathname.new(file).basename
    "<a href='#{file}'>#{filename}</a><br />"
  end

end

view this gist

OK, refresh your browser and you should see both foo.txt and bar.txt. This is looking pretty good, but we’re not really creating valid HTML right now. We’re missing html, head, and, body tags at the very least. We could add this all within our “get” handler, but that would clutter up the code.

Instead, let’s put this code into a view. Sinatra actually lets you put the view right after your other code, so you can build an entire application in one file. For simplicity, I’m going to do that for this tutorial. However, if this approach bothers you (or just messes up syntax highlighting in your editor), rest assured you can place the view code in a views directory and it would work the same way.

Let’s add the view to the end of our file, and use it in our handler. Notice that I name the view ‘index’ by beginning my declaration with @@ index – if I wanted a separate file, I would just put it in ./views/index.erb (you can also use Haml, if that’s your cup of tea). Note I assign @links in the handler and it automatically is available in the view.

require 'sinatra'
require 'pathname'

get "/" do
  dir = "./files/"
  @links = Dir[dir+"*"].map { |file|
    file_link(file)
  }.join
  erb :index
end

helpers do

  def file_link(file)
    filename = Pathname.new(file).basename
    "<a href='#{file}'>#{filename}</a><br />"
  end

end

use_in_file_templates!

__END__

@@ index
<html>
  <head>
  </head>

  <body>
    <h1>Your files, sir.</h1>
    <%= @links %>
  </body>
</html>

view this gist

Refreshing the browser now isn’t really that exciting, since things look the same, but if you wanted, you could easily play around with the view to make things look different.

One glaring problem is that this page isn’t very usable on the iPhone itself. That’s where iUI comes in. Start by downloading it (URL is in instructions below) to your butler directory, unzipping it, and copying the necessary files into your public directory.

$ mkdir iui
$ cd iui
$ wget http://iui.googlecode.com/files/iui-0.13.tar.gz
$ tar -xzvf iui-0.13.tar.gz
$ cd ..
$ mkdir public/images
$ cp iui/iui/*.png public/images
$ cp iui/iui/*.gif public/images
$ mkdir public/javascripts
$ cp iui/iui/*.js public/javascripts
$ mkdir public/stylesheets
$ cp iui/iui/*.css public/stylesheets

To use iUI, you’ll need to include the JavaScript and CSS in your view. You’ll also need to add some elements to the body of your view. When you’re done, the view will look like this:

<html>
  <head>
    <meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
    <style type="text/css" media="screen">@import "/stylesheets/iui.css";</style>
    <script type="application/x-javascript" src="/javascripts/iui.js"></script>
  </head>

  <body>
    <div class="toolbar">
    <h1 id="pageTitle"></h1>
    </div>
    <ul id="home" title="Your files, sir." selected="true">
       <%= @links %>
    </ul>
  </body>

</html>

view this gist

This html is probably a bit confusing, but don’t worry. There are a few examples in ./iui/samples/ to learn from (and good iUI tutorials on the web). Finally, you’ll want to alter the file_link helper to print out iUI code, like so:

helpers do

  def file_link(file)
    filename = Pathname.new(file).basename
    "<li><a href='#{file}' target='_self'>#{filename}</a></li>"
  end

end

view this gist

Note that target='_self' code. You need that to get iUI to open a link in a normal way. If you leave it off, it will use an AJAX call to load the file within the current page, which looks really funny when you try to open a binary file like a PDF.

The final code looks like this:

require 'sinatra'
require 'pathname'

get "/" do
  dir = "./files/"
  @links = Dir[dir+"*"].map { |file|
    file_link(file)
  }.join
  erb :index
end

helpers do

  def file_link(file)
    filename = Pathname.new(file).basename
    "<li><a href='#{file}' target='_self'>#{filename}</a></li>"
  end

end

use_in_file_templates!

__END__

@@ index
<html>
  <head>
    <meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"/>
    <style type="text/css" media="screen">@import "/stylesheets/iui.css";</style>
    <script type="application/x-javascript" src="/javascripts/iui.js"></script>
  </head>

  <body>
    <div class="toolbar">
    <h1 id="pageTitle"></h1>
    </div>
    <ul id="home" title="Your files, sir." selected="true">
       <%= @links %>
    </ul>
  </body>

</html>

view this gist

And there you have it – an iPhone web app in less than 50 lines of code, thanks to Sinatra and iUI. Now, whenever you want to view some files on your iPhone, either copy the file:

$ cp path/to/my_file ./files

or if you prefer, link it:

$ ln -s path/to/my_file ./files

… and then run butler

$ ruby -rubygems ./butler.rb

Figure out the IP address of your computer and simply point your iPhone browser to http://<ip&gt;:4567

I use butler primarily within my home network, but if you want to be able to view your files on the go, you’ll need to poke a hole in your firewall. That’s a bit outside the scope of this tutorial, but a quick Google search should give you some good results.

Enjoy!

Update: Removed an unused parameter from the code after pmccann called it to my attention.
Update: Added -z option to tar after Peter pointed out the omission. The tar command without -z worked for me on OS X 10.5, but this is definitely more correct.
Update: Added -rubygems option to ruby command. If you’d prefer to not use this option, check the comments below for ways to use RubyGems in a Ruby script.

Written by Ben

November 25, 2008 at 9:14 am

Posted in Hacking, Tools

Ruby Beanstalkd distributed worker intermediate lessons

This post is a follow up to Ruby beanstalkd basics, I will try to make the example code little more interesting and useful. I am calling this is a Ruby beanstalkd intermediate write up, it sets up a few workers and distributes and receives results simultaneously. In this example the code resembles real code a bit more (using a queue cache and block passing). If there is enough interest in the Ruby/beanstalkd community, I will follow up with beanstalkd advanced lessons, and go into how we deal with failure cases such as worker dying during jobs, random jobs failing, processing multiple ‘projects’ at one time, using job priority settings, and using TTR/timeouts.

So in this example we are making an estimate of PI. Yes I know that there are far better approximations out there than my simple results, but this was what I came up with for an incredibly simple distributed computing problem. I based my example on the PI Calculation problem from an Introduction to Parallel Computing. The basic idea is that you can calculate pi by guessing random points in a square and then seeing how many points are inside a circle that fits inside the square (PI= 4 * points_in_circle/total_points).

I made a bunch of comments in the code that should help you follow but there are a few key sections worth pointing out.

In the Ruby beanstalkd Basics, both the Server and the Clients only used one queue at a time. Now since we are sending on one queue while also listening on another we need access to both queues at once. We simply have a helper function with a queue_cache to make getting and reusing multiple queues incredibly easy.

def get_queue(queue_name)
    @queue_cache ||= {}
    if @queue_cache.has_key?(queue_name)
      return @queue_cache[queue_name]
    else
      queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
      queue.watch(queue_name)
      queue.use(queue_name)
      queue.ignore('default')
      @queue_cache[queue_name] = queue
      return queue
    end
  end

In the basic example each class had a function that got a job and did some work and deleted the job. It is easy to imagine workers that might have many different kinds of work to do on jobs. In every case they are going to grab a job, work on the job, and delete the job. We decided to break that up and make it easy to just pass a work block when workers get a job.

def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    body = msg.ybody
    if block_given?
      yield(body)
    end
    msg.delete
  end

#call take_msg like so
take_msg(queue) do |body|
  #work on body
end

One other thing you should keep a look out for in the code below is checking if a queue has any jobs. Many times workers will check if jobs exist and take them, and if there aren’t any jobs the process is free to do something else. I do this in this example, the server continually checks incoming results to immediately display. If no results have arrived yet, the server continues sending out job requests as fast as it can. This is useful since taking jobs from beanstalkd is a blocking call. They did add support for non-blocking calls in beanstalkd 1.1, but I haven’t started using the newest version yet. I think everything else should be pretty self explanatory, feel free to ask me any questions. To run the code it is the same as before: download beanstalk_intermediate.rb, start beanstalkd, and run the example with ruby.

$ beanstalkd &amp;
$ ruby beanstalk_intermediate.rb
starting distributor
starting client(s)
distributor sending out  jobs
.......................................................
.............................................
received all the results our estimate for pi is: 3.142776
# of workers time to complete
1 real 0m7.282s
user 0m4.114s
sys 0m0.978s
2 real 0m5.667s
user 0m2.736s
sys 0m0.670s
3 real 0m4.999s
user 0m2.014s
sys 0m0.515s
4 real 0m4.612s
user 0m1.608s
sys 0m0.442s
5 real 0m4.517s
user 0m1.474s
sys 0m0.416s
require 'beanstalk-client.rb'

DEFAULT_PORT = 11300
SERVER_IP = '127.0.0.1'
#beanstalk will order the queues based on priority, with the same priority
#it acts FIFO, in a later example we will use the priority
#(higher numbers are higher priority)
DEFAULT_PRIORITY = 65536
#TTR is time for the job to reappear on the queue.
#Assuming a worker died before completing work and never called job.delete
#the same job would return back on the queue (in TTR seconds)
TTR = 3

class BeanBase

  #To work with multiple queues you must tell beanstalk which queues
  #you plan on writing to (use), and which queues you will reserve jobs from
  #(watch). In this case we also want to ignore the default queue
  #you need a different queue object for each tube you plan on using or
  #you can switch what the tub is watching and using a bunch, we just keep a few
  #queues open on the tubes we want.
  def get_queue(queue_name)
    @queue_cache ||= {}
    if @queue_cache.has_key?(queue_name)
      return @queue_cache[queue_name]
    else
      queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
      queue.watch(queue_name)
      queue.use(queue_name)
      queue.ignore('default')
      @queue_cache[queue_name] = queue
      return queue
    end
  end

  #this will take a message off the queue, and process it with the block
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    body = msg.ybody
    if block_given?
      yield(body)
    end
    msg.delete
  end

  def results_ready?(queue)
    queue.peek_ready!=nil
  end

end

class BeanDistributor < BeanBase

  def initialize(chunks,points_per_chunk)
    @chunks = chunks
    @points_per_chunk = points_per_chunk
    @messages_out = 0
    @circle_count = 0
  end

  def get_incoming_results(queue)
    if(results_ready?(queue))
      result = nil
      take_msg(queue) do |body|
        result = body.count
      end
      @messages_out -= 1
      print "." #display that we received another result
      @circle_count += result
    else
      #do nothing
    end
  end

  def start_distributor
    request_queue = get_queue('requests')
    results_queue = get_queue('results')
    #put all the work on the request queue
    puts "distributor sending out #{@messages} jobs"
    @chunks.times do |num|
      msg = BeanRequest.new(1,@points_per_chunk)
      #Take our ruby object and convert it to yml and put it on the queue
      request_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
      @messages_out += 1
      #if there are results get them if not continue sending out work
      get_incoming_results(results_queue)
    end

    while @messages_out > 0
      get_incoming_results(results_queue)
    end
    npoints = @chunks * @points_per_chunk
    pi = 4.0*@circle_count/(npoints)
    puts "\nreceived all the results our estimate for pi is: #{pi}"
  end

end

class BeanWorker < BeanBase

  def initialize()
  end

  def write_result(queue, result)
    msg = BeanResult.new(1,result)
    queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
  end

  def in_circle
    #generate 2 random numbers see if they are in the circle
    range = 1000000.0
    radius = range / 2
    xcord = rand(range) - radius
    ycord = rand(range) - radius
    if( (xcord**2) + (ycord**2) <= (radius**2) )
      return 1
    else
      return 0
    end
  end

  def start_worker
    request_queue = get_queue('requests')
    results_queue = get_queue('results')
    #get requests and do the work until the worker is killed
    while(true)
      result = 0
      take_msg(request_queue) do |body|
        chunks = body.count
        chunks.times { result += in_circle}
      end
      write_result(results_queue,result)
    end

  end

end

############
# These are just simple message classes that we pass using beanstalks
# to yml and from yml functions.
############
class BeanRequest
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

class BeanResult
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

#how many different jobs we should do
chunks = 100
#how many points to calculate per chunk
points_per_chunk = 10000
#how many workers should we have
#(normally different machines, in our example fork them off)
workers = 5

# Most of the time you will have two entirely separate classes
# but to make it easy to run this example we will just fork and start our server
# and client separately. We will wait for them to complete and check
# if we received all the messages we expected.
puts "starting distributor"
server_pid = fork {
  BeanDistributor.new(chunks,points_per_chunk).start_distributor
}

puts "starting client(s)"
client_pids = []
workers.times do |num|
  client_pid = fork {
    BeanWorker.new.start_worker
  }
  client_pids << client_pid
end

Process.wait(server_pid)
#take down the clients
client_pids.each do |pid|
  Process.kill("HUP",pid)
end

Written by DanM

November 19, 2008 at 3:19 pm

Posted in Development, Hacking, Ruby

Someone please build an awesome embeddable code widget

One awesome thing about working at a startup is that you get to focus very deeply on the problem you’re trying to solve. On the other hand, if you’ve taken the leap and founded a startup, it’s probably because you tend to see solutions and opportunities everywhere. It can be really hard to focus on one thing when you often have ideas for services that you’d like to use, or better yet, build.

Dan and I regulary talk about services that we wish existed but we simply can’t work on due to our commitment to Devver. The other day we were discussing one problem we wish someone would solve: why can’t we easily post nicely formatted code in our blog posts?

All I want is this: I copy/paste some code into a web site, choose the programming language, copy some widget code and paste that code into my blog. The code is indented and formatted, has syntax coloration, wraps correctly (for any iPhone readers) and can be easy copied/pasted. Including line numbers (that don’t mess with copy/paste) is a bonus.

In other words, I just want Pastie in my blog posts.

Yes, I know there are a few really nice projects that you can install on your server that will do all this. But we could all host our own video as well, but it’s just easier to upload and embed a video on YouTube or Vimeo.

This wouldn’t just have to be for the good of humanity either. Such a service could make money off ads (each widget could have a link to the full-screen code on the main site, which could have ads for programming jobs, books, and conferences) or even sell off the data about which programming languages were most popular (in blogs and on the main site).

Maybe there is a solution for this (if there is, please let me know in the comments. I’m more than willing to publicly display my ignorance in order to learn about it), but if there is, I don’t see it widely being used and it’s not easy to find on Google. If there isn’t (yet), please go forth and build. I’ll be anxiously waiting.

Written by Ben

October 30, 2008 at 7:53 am

Posted in Development, Hacking

Ruby Beanstalkd distributed worker basics

At Devver we have a lot of jobs to do quickly, so we distribute our work out to a group of EC2 workers. We have tried and used a number of queuing solutions with Ruby, but in the end beanstalkd seemed to be the best solution for us at the time.

I have only seen a few posts about the basics of using beanstalkd with Ruby. I decided to make two posts evolving a simple Ruby beanstalkd example into a more complicated example. This way people new to beanstalkd could see how easy it can be to get up and running with distributed processing using Ruby and beanstalkd. Then people that are doing more advanced work with beanstalkd could see some examples of how we are working with it here at Devver. It would also be great for more experienced beanstalkd warriors to share their thoughts as there aren’t many examples out in the wild. The lack of examples makes it harder to learn and difficult to decide what the best practices are when working with beanstalkd queues.

I have also shared two scripts we have found useful while working with beanstalkd. beanstalk_monitor.rb, which lets you see all the queue statistics about current usage, or to monitor the information of a single queue you are interested in. Finally, beanstalk_killer.rb, which is useful if you want to work on how your code will react to beanstalkd getting backed up or stalling (in beanstalkd speak, “Putting on the brakes”). It was a little harder to pull everything out and make a simple example from our code than I thought, and obviously the example is a bit useless. It should still give a solid example of how to do the basics of distributing jobs with beanstalkd.

For those new to beanstalk, there are a few things you will need to know like how to get a queue object, how to put objects on the queue, how to take objects off the queue, and how to control which queue you are working with. For a higher level overview or more detailed information, I recommend checking out the beanstalkd FAQ. The full example code is below, but first taking a look at the basic snippets might help.

#to work with beanstalk you need to get a client connection
queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
#by default you will be working on the 'default' tube or queue
#if we wanted to work on a different queue we could change tubes, like so
queue.watch('test_queue')
queue.use('test_queue')
queue.ignore('default')
#to put a simple string on a queue
queue.put('hello queue world')
#to receive a simple string
job = queue.reserve
puts job.body #prints 'hello queue world'
#if you don't delete the job when you're done, the queue assumes there is an error
#and the job will show back up on the queue again
job.delete

How to run this example (on OS X, with macports installed)

> sudo port install beanstalkd
> sudo gem install beanstalk-client
> beanstalkd
> ruby beanstalk_tester.rb

Download: beanstalk_tester.rb

require 'beanstalk-client.rb'

DEFAULT_PORT = 11300
SERVER_IP = '127.0.0.1'
#beanstalk will order the queues based on priority, with the same priority
#it acts FIFO, in a later example we will use the priority
#(higher numbers are higher priority)
DEFAULT_PRIORITY = 65536
#TTR is time for the job to reappear on the queue.
#Assuming a worker died before completing work and never called job.delete
#the same job would return back on the queue (in seconds)
TTR = 3

class BeanBase


  #To work with multiple queues you must tell beanstalk which queues
  #you plan on writing to (use), and which queues you will reserve jobs from
  #(watch). In this case we also want to ignore the default queue
  def get_queue(queue_name)
    queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
    queue.watch(queue_name)
    queue.use(queue_name)
    queue.ignore('default')
    queue
  end

end

class BeanDistributor < BeanBase

  def initialize(amount)
    @messages = amount
  end

  def start_distributor
    #put all the work on the request queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      msg = BeanRequest.new(1,num)
      #Take our ruby object and convert it to yml and put it on the queue
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    puts "distributor now getting results"
    #get all the results from the results queue
    bean_queue = get_queue('results')
    @messages.times do |num|
      result = take_msg(bean_queue)
      puts "result: #{result}"
    end

  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    msg.delete
    return count
  end

end

class BeanWorker < BeanBase

  def initialize(amount)
    @messages = amount
    @received_msgs = 0
  end

  def start_worker
    results = []
    #get and process all the requests, on the requests queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      result = take_msg(bean_queue)
      results << result
      @received_msgs += 1
    end

    #return all of the results, by placing them on the separate results queue
    bean_queue = get_queue('results')
    results.each do |result|
      msg = BeanResult.new(1,result)
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    #this is just to pass information out of the forked process
    #we return the number of messages we received as our exit status
    exit @received_msgs
  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    result = count*count
    msg.delete
    return result
  end

end

############
# These are just simple message classes that we pass using beanstalks
# to yml and from yml functions.
############
class BeanRequest
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

class BeanResult
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

#write X messages on the queue
numb = 10

recv_count = 0

# Most of the time you will have two entirely seperate classes
# but to make it easy to run this example we will just fork and start our server
# and client seperately. We will wait for them to complete and check
# if we received all the messages we expected.
puts "starting distributor"
server_pid = fork {
  BeanDistributor.new(numb).start_distributor
}

puts "starting client"
client_pid = fork {
  BeanWorker.new(numb).start_worker
}

Process.wait(client_pid)
recv_count = $?.exitstatus
puts "client finished received #{recv_count} msgs"
if(numb==recv_count)
  puts "received the expected number of messages"
else
  puts "error didn't receive the correct number of messages"
end

Process.wait(server_pid)

Written by DanM

October 28, 2008 at 2:35 pm

Tracking down open files with lsof

The other day I was running in a weird error on Devver. After running around twenty test runs on the system, the component that actually runs individual unit tests was crashing due to “Too many open files – (Errno::EMFILE)”

Unfortunately, I didn’t know much more than that. Which files were being kept open? I knew that this component loaded quite a few files, and that by default, OS X only allows 256 open file descriptors (

ulimit -n

will tell you the default on your system). If this was a valid case of needing to load more files, I could just up the limit using

ulimit -n <bigger_number>

.

Fortunately, a quick Google or two pointed the way to

lsof

. Unfortunately, my Unix-fu is never nearly as good as I wish and I didn’t know much about this handy utility. But I quickly discovered that it’s very useful for tracking down problems like this. I quickly used

ps

to find the PID of the Devver process and then a quick

lsof -p <PID>

displayed all the files that the process had open. So easy!

Sure enough, there were a ton of redundant file handles to the file that we use to store information about the Devver run. Armed with this information, it was easy to find the buggy code where we called File.open but failed to ever close the file.

Unfortunately, I still don’t know how to write a good unit test for this case. I guess I could do something ugly like call sytem(“lsof -p pid | wc -l”) before and after calling the code and make sure the number of descriptors stays constant, but that’s really ugly. Is there a way to test this within Ruby? I’m open to ideas.

Still, it’s always good to learn more about a powerful Unix tool. I’m constanly amazed by the power and depth of the Unit tool set.

Written by Ben

October 9, 2008 at 12:23 pm

Ruby Tools Roundup

Update: Devver now offers a hosted metrics service for Ruby developers which can give you useful feedback about your code. Check out Caliper, to get started with metrics for your project.

I collected all of the Ruby tools posts I made this week into a single roundup. You can quickly jump to any tool that interests you or read my reviews start to finish. If you just want to read a individual section here are the previous posts Ruby Code Quality Tools, Ruby Test Quality Tools, and Ruby Performance Tools.

There have been a bunch of interesting tools released for Ruby lately. I decided to write about a few of my favorite Ruby tools and give some of the new tools a shot as well. Simply put, better tools can help you be a better developer. I am ignoring the entire topic of IDEs as tools, as I have written about Ruby IDEs before, and it is basically a religious war. If you use any Ruby tools I don’t mention be sure to let me know as I am always interested in trying something new out.

Tool Name Description
Code Quality Tools
Roodi Roodi gives developers information about common mistakes in their Ruby code. It makes it easy to clean up your code before things start to get ugly.
Dust Dust is a new tool that will analyze your code, detect unsafe blocks and unused code. Dust is being created by the same mind behind Heckle
Flog Flog essentially scores an ABC metric, giving you a good understanding of the overall code complexity of any give file or method.
Saikuro When given Ruby source code, Saikuro will generate a report listing the cyclomatic complexity of each method found.
Test Quality Tools
Heckle Heckle helps test your Ruby tests (how cool is that?). Heckle is a mutation tester. It alters/breaks code and verifies that tests fail.
rcov rcov is the easiest way to get information about your current code coverage.
Ruby/Rails Performance Tools
ruby-prof ruby-prof is a fast and easy-to-use Ruby profiler. The first of four tools that can help you solve performance issues.
New Relic New Relic is one of the three Rails plugin performance debugging and monitoring tools recently released.
TuneUp TuneUp a Rails performance tool from FiveRuns. This tool has an interesting community built around it as well.
RubyRun Ruby Run is a Rails performance tool similar to New Relic and TuneUp

Lets get into it…

Roodi


Roodi gives you a bunch of interesting warnings about your Ruby code. We are about to release some code, so I took the opportunity to fix up anything Roodi complained about. It helped identify refactoring opportunities, both with long methods, and overly complex methods. The code and tests became cleaner and more granular after breaking some of the methods down. I even found and fixed one silly performance issue that was easy to see after refactoring, which improved the speed of our code. Spending some time with Roodi looks like it could easily improve the quality and readability of most Ruby projects with very little effort. I didn’t solve every problem because in one case I just didn’t think the method could be simplified anymore, but the majority of the suggestions were right on. Below is an example session with Roodi

dmayer$ sudo gem install roodi
dmayer$ roodi lib/client/syncer.rb
lib/client/syncer.rb:136 - Block cyclomatic complexity is 5.  It should be 4 or less.
lib/client/syncer.rb:61 - Method name "excluded" has a cyclomatic complexity is 10.  It should be 8 or less.
lib/client/syncer.rb:101 - Method name "should_be_excluded?" has a cyclomatic complexity is 9.  It should be 8 or less.
lib/client/syncer.rb:132 - Method name "find_changed_files" has a cyclomatic complexity is 10.  It should be 8 or less.
lib/client/syncer.rb:68 - Rescue block should not be empty.
lib/client/syncer.rb:61 - Method name "excluded" has 25 lines.  It should have 20 or less.
lib/client/syncer.rb:132 - Method name "find_changed_files" has 27 lines.  It should have 20 or less.
Found 7 errors.

After Refactoring:

~/projects/gridtest/trunk dmayer$ roodi lib/client/syncer.rb
lib/client/syncer.rb:148 - Block cyclomatic complexity is 5.  It should be 4 or less.
lib/client/syncer.rb:82 - Rescue block should not be empty.
Found 2 errors.

I did have one problem with Roodi – the errors about rescue blocks just seemed to be incorrect. For code like the little example below it kept throwing the error even though I obviously am doing some work in the rescue code.

Roodi output: lib/client/syncer.rb:68 - Rescue block should not be empty.
begin
  socket = TCPSocket.new(server_ip,server_port)
  socket.close
  return true
rescue Errno::ECONNREFUSED
  return false
end

Dust


Dust detects unused code like unused variables,branches, and blocks. I look forward to see how the project progresses. Right now there doesn’t seem to be much out there on the web, and the README is pretty bare bones. Once you can pass it some files to scan, I think this will be something really useful. For now I didn’t think there wasn’t much I could actually do besides check it out. Kevin, who also helped create the very cool Heckle, does claim that code scanning is coming soon, so I look forward to doing a more detailed write up eventually.

Flog


Flog gives feedback about the quality of your code by scoring code using the ABC metric. Using Flog to help guide refactoring, code cleanup, and testing efforts can be highly effective. It is a little easier to understand the reports after reading how Flog scores your code, and what is a good Flog score. Once you get used to working with Flog you will likely want to run it often against your whole project after making any significant changes. There are two easy ways to do this a handy Flog Rake task or MetricFu which works with both Flog and Saikuro.

Running Flog against any subset of a project is easy, here I am running it against our client libraries

find ./lib/client/ -name \*.rb | xargs flog -n -m &gt; flog.log

Here some example Flog output when run against our client code.

Total score = 1364.52395469781

Client#send_tests: (64.3)
    14.3: assignment
    13.9: puts
    10.7: branch
    10.5: send
     4.7: send_quit
     3.4: message
     3.4: now
     2.0: create_queue_test_msg
     1.9: create_run_msg
     1.9: test_files
     1.8: dump
     1.7: each
     1.7: report_start
     1.7: length
     1.7: get_tests
     1.7: -
     1.7: open
     1.7: load_file
     1.6: empty?
     1.6: nil?
     1.6: use_cache
     1.6: exists?
ModClient#send_file: (32.0)
    12.4: branch
     5.4: +
     4.3: assignment
     3.9: send
     3.1: puts
     2.9: ==
     2.9: exists?
     2.9: directory?
     1.9: strftime
     1.8: to_s
     1.5: read
     1.5: create_file_msg
     1.4: info
Syncer#sync: (30.8)
    13.2: assignment
     8.6: branch
     3.6: inspect
     3.2: info
     3.0: puts
     2.8: +
     2.6: empty?
     1.7: map
     1.5: now
     1.5: length
     1.4: send_files
     1.3: max
     1.3: >
     1.3: find_changed_files
     1.3: write_sync_time
Syncer#find_changed_files: (26.2)
    15.6: assignment
     8.7: branch
     3.5: <<
     1.8: to_s
     1.7: get_relative_path
     1.7: >
     1.7: mtime
     1.6: exists?
     1.6: ==
     1.5: prune
     1.4: should_be_excluded?
     1.3: get_removed_files
     1.3: find
... and so on ...

Saikuro


Saikuro is another code complexity tool. It seems to give a little less information than some of the others. It does generate nice HTML reports. Like other code complexity tools it can be helpful to discover the most complex parts of your projects for refactoring and to help focus your testing. I liked the way Flog broke things down for me into a bit more detail, but either is a useful tool and I am sure it is a matter of preference depending on what you are looking for.

saikuro screenshot
Saikuro Screenshot

Heckle


Heckle is an interesting tool to do mutation testing of your tests. Heckle currently supports Test:Unit and RSpec, but does have a number of issues. I had to run it on a few different files and methods before I got some useful output that helped me improve my testing. The first problem was it crashing when I passed it entire files (crashing the majority of the time). I then began passing it single methods I was curious about, which still occasionally caused Heckle to get into an infinite loop case. This is a noted problem in Heckle, but -T and providing a timeout should solve that issue. In my case it was actually not an infinite loop timing error, but an error when attempting to rewrite the code, which lead to a continual failure loop that wouldn’t time out. When I found a class and method that Heckle could test I got some good results. I found one badly written test case, and one case that was never tested. Lets run through a simple Heckle example.

#install heckle
dmayer$ sudo gem install heckle

#example of the infinite loop Error Heckle run
heckle Syncer should_be_excluded? --tests test/unit/client/syncer_test.rb -v

Setting timeout at 5 seconds.
Initial tests pass. Let's rumble.

**********************************************************************
*** Syncer#should_be_excluded? loaded with 13 possible mutations
**********************************************************************
...
2 mutations remaining...
Replacing Syncer#should_be_excluded? with:

2 mutations remaining...
Replacing Syncer#should_be_excluded? with:
... loops forever ...

#Heckle run against our Client class and the process method

dmayer$ heckle Client process --tests test/unit/client/client_test.rb

Initial tests pass. Let's rumble.

**********************************************************************
*** Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

--- original
+++ mutation

def process(command)

case command
when @buffer.Ready then
process_ready
- when @buffer.SetID then
+ when nil then
process_set_id(command)
when @buffer.InitProject then
process_init_project
when @buffer.Result then
process_result(command)
when @buffer.Goodbye then
kill_event_loop
when @buffer.Done then
process_done
when @buffer.Error then
process_error(command)
else
@log.error("client ignoring invalid command #{command}") if @log
end
end

--- original
+++ mutation
def process(command)
case command
when @buffer.Ready then
process_ready
when @buffer.SetID then
process_set_id(command)
when @buffer.InitProject then
process_init_project
when @buffer.Result then
process_result(command)
when @buffer.Goodbye then
kill_event_loop
when @buffer.Done then
process_done
when @buffer.Error then
process_error(command)
else
- @log.error("client ignoring invalid command #{command}") if @log
+ nil if @log
end
end

Heckle Results:

Passed : 0
Failed : 1
Thick Skin: 0

Improve the tests and try again.

#Tests added / changed to improve Heckle results

def test_process_process_loop__random_result
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    client.log.expects(:error).with("client ignoring invalid command this is random")
    client.process("this is random")
  end

  def test_process_process_loop__set_id
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    cmd = DataBuffer.new.create_set_ids_msg("4")
    client.expects(:process_set_id).with(cmd)
    client.process(cmd)
  end

#A final Heckle run, showing successful results

dmayer$ heckle Client process --tests test/unit/client/client_test.rb

Initial tests pass. Let's rumble.

**********************************************************************
*** Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...
No mutants survived. Cool!

Heckle Results:

Passed : 1
Failed : 0
Thick Skin: 0

All heckling was thwarted! YAY!!!

rcov


rcov is a code coverage tool for Ruby. If you are doing testing you should probably be monitoring your coverage with a code coverage tool. I don't know of a better tool for code coverage than rcov. It is simple to use and generates beautiful, easy-to-read HTML charts showing the current coverage broken down by file. An easy way to make you project more stable is to occasionally spend some time increasing the coverage you have on your project. I have always found it a great way to get back into a project if you have been off of it for awhile. You just need to find some weak coverage points and get to work.
Rcov Screenshot
rcov screenshot

ruby-prof


ruby-prof does what every other profiler does, but it is much faster than the one built in to Ruby. It also makes it easy to output the information you are seeking to HTML pages, such as call graphs. If you are just looking for a simple write up to get started with ruby-prof I recommend the previous link. I will talk a little more about the kinds of problems I find and how I have solved them with ruby-prof.

I have used ruby-prof a number of times to isolate the ways to speed up my code. I haven't used it to identify why an entire Rails application is slow (there are better tools I discuss later for that), but if you have a small but highly important piece of code ruby-prof is often the best way to isolate the problem. I used ruby-prof to identified the two slowest lines of code of a spellchecker, which was rewritten to become twice as fast.

Most recently I used it to identify where the code was spending all of its time in a loop for a file syncer. It turns out that for thousands of files each time through the loop we were continually calling Pathname.new(path).relative_path_from(@dir_path) over and over. Putting a small cache around that call essentially eliminated all delays in our file synchronization. Below is a simple example of how a few lines of code can make all the difference in performance and how easily ruby-prof can help you isolate the problem areas and where to spend your time. I think seeing the code that ruby-prof helped isolate, and the changes made to the code might be useful if you are new to profiling and performance work.

changes in our spellchecker / recommender

#OLD Way
 alteration = []
    n.times {|i| LETTERS.each_byte {
        |l| alteration << word[0...i].strip+l.chr+word[i+1..-1].strip } }
 insertion = []
     (n+1).times {|i| LETTERS.each_byte {
        |l| insertion << word[0...i].strip+l.chr+word[i..-1].strip } }
 #NEW Way
    #pre-calculate the word breakups
    word_starts = []
    word_missing_ends = []
    word_ends = []
    (n+1).times do |i|
      word_starts << word[0...i]
      word_missing_ends << word[i+1..-1]
      word_ends << word[i..-1]
    end

 alteration = []
    n.times {|i|
      alteration = alteration.concat LETTERS.collect { |l|
        word_starts[i]+l+word_missing_ends[i] } }
 insertion = []
    (n+1).times {|i|
      insertion = insertion.concat LETTERS.collect { |l|

        word_starts[i]+l+word_ends[i] } }

Changes in our file syncer

#OLD
 path_name = Pathname.new(path).relative_path_from(@dir_path).to_s
 #NEW
 path_name = get_relative_path(path)

  def get_relative_path(path)
    return @path_cache[path] if @path_cache.member?(path)
    retval = Pathname.new(path).relative_path_from(@dir_path).to_s
    @path_cache[path] = retval
    return retval
  end

New Relic


New Relic is a performance monitoring tool for Rails apps. It has a great development mode that will help you track down performance issues before they even become a problem, and live monitoring so that you can find any hiccups that are slowing down the production application. The entire performance monitoring space for Ruby/Rails seems to be heating up. I guess it is easy to see why, when scaling has been such an issue for some Rails apps. Just playing around with New Relic was exciting and fun. I could quickly track down the slowest pages, and our most problematic SQL calls, in this case I was testing New Relic on Seekler (an old project of ours) since I didn't think I would find much interesting on our current Devver site. Seekler had some glaring performance issues and I think if we had New Relic from the beginning we could have avoided many of them. Sounds like I might have a day project involving New Relic and giving Seekler as much of a performance boost as possible. New Relic turned out to be my favorite of the performance monitoring tools. For a much more detailed writeup check out RailsTips New Relic Review.

newrelic screenshot
New Relic screenshot

TuneUp


TuneUp another easy-to-install and use Rails performance monitoring solution. The problem I had with TuneUp was I couldn't get it working on test app for these sorts of things. I tried running Seekler with TuneUp, but had no luck. I found that many people on the message boards seemed to be having various compatibility issues. I looked at the TuneUp screencast and the kind of information that they give you and I feel like this would be equal to New Relic if it works for you. I am emailing back and forth with FiveRuns support who have been very attentive and helpful, so if I get it working I will update this section.

Update: FiveRuns is pretty amazing with their support. I haven't got TuneUp fully working yet, but have made some progress. Some good things to know are that some plugins like safe_erb and output_compression can cause problems with TuneUp. They are aware of the issues, and actively looking into it.

Ruby Run


RubyRun provides live performance monitoring and debugging tools. I hadn't ever heard of this product before I started doing some research while writing this blog article. I am sorry to say but this was the hardest to set up, and gave back less valuable information. I think they need a simple screencast on how to get set up and get useful information back. After getting setup and running I could only get ugly CSV reports that didn't tell me much more than the regular Rails log files. I started reading the RubyRun Manual but it was about as long as Moby Dick and all I wanted was how to view simple easy-to-read reports which is a snap in New Relic and TuneUp. Since the site didn't mention RubyRun providing better data than New Relic or TuneUp which were much more user friendly, I don't think I would recommend RubyRun.

UPDATE: After reading about my difficulties with RubyRun the great folks from Rubysophic got in touch with me. They offered to help me get the tool working and posted a RubyRun quick start guide to their site. I got it working in a snap thanks to an email from their dev and the amazingly simple quick start guide. I still didn't get the same depth of information that I got with New Relic, although RubyRun has a ton of settings so it is likely you can get more depth to the reports. Something worth pointing out is that RubyRun is working on Seekler, which I haven't been able to get TuneUp running on. So if you have been having problems with TuneUp or New Relic, definitely give RubyRun a look. In the end I think the other offerings are slightly more user friendly (less complex settings), and easier to explore the data (link in the feed to both reports, at least when in developer mode). That being said RubyRun offers some great information and options that the others don't and with a bit more UI tuning RubyRun would be at the top of the pack. Thanks to the helpful devs at Rubysophic for helping me to get the most out of RubyRun.

RubyRun screenshot
RubyRun screenshot
RubyRun second screen shot
screenshot of a different RubyRun report

That is it, hope you learned about a new Ruby tool. So get to work, try a new tool, and get to know your code a little better than you did before.

While I was writing this article, people pointed out to me two more tools worth pointing out. I didn't get a chance to try them out or review them, but thought I should point them out. Towlie, helps keep your code dry by finding redundant methods. and finally Source ANalysis (SAN), which is described as, "a Ruby gem for analyzing the contents of source code including comment to script ratios, todo items, declared functions, classes, and much more".

Written by DanM

October 3, 2008 at 10:25 am

Why Side Projects?

I just read a great post at Ekinoderm that nicely tied together two things Dan and I have been thinking and blogging about recently – the joys of side projects and why you should own your own IP. Check it out.

Written by Ben

September 5, 2008 at 3:46 pm

Posted in Development, Hacking

My Day Project: IMShoes

Inspired by two posts _Why’s ‘so create’ and GitHub’s Start a Side Project I decided that I needed to do a quick side project. After our demo day we were taking the weekend easy and I had some extra time on Sunday. I decided that I should find a really quick project something I could really just do in one day. Since I was partly inspired by _why, I decided to take some time to learn Shoes.

“Shoes is a very informal graphics and windowing toolkit. It’s for making regular old apps that run on Windows, Mac OS X and Linux. It’s a blend of my favorite things from the Web, some Ruby style, and a sprinkling of cross-platform widgets.”

I tried to think of a simple Shoes app that I could contribute to The Shoebox (a collection of Ruby Shoes apps). I saw a couple Shoe’s Twitter apps, but didn’t see any Shoes AIM apps. Since I knew a good AIM protocol library for Ruby existed (Net-Toc), I thought it would be a nice simple app.

I ran into a few problems with Shoes (getting it to work with Gems) and it seemed to have problems loading YAML files. Besides the mentioned problems it was a breeze to work with. I looked at a few different example apps from the Shoebox. I ended up using Gentle Reminder as a template of sorts for my app. So basically in couple hours on Sunday I read a bunch about Shoes, read the source code of a couple Shoes apps, and I created a very small but functional GUI for AIM. It was a great little side project that reminded me how much fun it can be to code little things once in awhile.

If you have a day off go try to write some code and contribute to one of your favorite projects. Or finally go learn a new library or framework that looks really cool and try to give something back to that community. Then write up a post and share your experience with others. The best way to learn is to go do something, quoting _why, “so create.”

So I proudly present my tiny day project IMShoes.

# A small gui for basic AIM messaging

# Author::    Dan Mayer (mailto:dan<@t>devver.net)
# Site:: http://devver.net/blog
# Copyright:: Copyright (c) 2008 Dan Mayer
# License::   revised BSD license
# (http://www.opensource.org/licenses/bsd-license.php)
# Version::   0.0.1 (AKA a "alpha it sucks" version)
# Thanks::
# Ian Henderson for creating TOC, _why for Shoes, and
# Oliver for Gentle Reminder which I used to learn (as my template).

#You need to install net-toc with gem install net-toc
#then copy toc.rb into shoes/contents/ruby/lib/net for this to work in shoes
require 'net/toc'

Shoes.app :title => "Shoes AIM",
  :width => 370, :height => 560, :resizable => false do

  #ADD your own user and password here, should allow you to do this via GUI
  @user = 'urUser'
  @password = 'urPass'

  background green, :height => 40

  caption "IM Shoes", :margin => 8, :stroke => white

  stack :margin => 10, :margin_top => 10 do
    para "Buddies", :stroke => red, :fill => yellow

    stack :margin_left => 5, :width => 1.0, :height => 200 do
      background white
      border white, :strokewidth => 3
      @gui_buddies = para
    end

     flow :margin_top => 10 do
       para "To"
       @send_to = edit_line(:margin_left => 10, :width => 180)
     end

     flow :margin_top => 10 do
       para "Message"
       @add = edit_line(:margin_left => 10, :width => 180)
       button("Send", :margin_left => 5)  do
         send_msg(@send_to.text, @add.text);
         @send_to.text = '';
         @add.text = '';
       end
     end
   end

  stack :margin_top => 10 do
    background darkgray
    para strong('Messages'), :stroke => white
  end

  @gui_messages = stack :width => 1.0, :height => 207

  def refresh_buddies
     @buddies = []
     @client.buddy_list.each_group { |g, b| @buddies = @buddies + b }

     @gui_buddies.replace *(
                         @buddies.map { |buddy|
                           [ buddy.screen_name, '  ' ] +
                           [ link('Message') { set_to buddy.screen_name } ] +
                           [ '  ' ] + [ "\n" ]
                         }.flatten
                         )
  end

  def refresh
    refresh_buddies
    refresh_msgs
  end

  def refresh_msgs
    @gui_messages.clear

    @gui_messages.append do
      background white

      @messages.keys.sort.reverse.each { |day|
        stack do
          background lightgrey
          para strong(day.strftime('%B %d, %Y')), :stroke => white
        end

        stack do
          inscription *(
                        ([@messages[day][0].to_s]+
                         [" "]+
                         [@messages[day][1].to_s]+
                         [" "]+
                         [link('Reply') { set_to(@messages[day][0].to_s) }]).flatten
                        )
        end

      }

    end
  end

  def set_to(screen_name)
    @send_to.text = screen_name;

    refresh
  end

  def send_msg(to_name, msg)
    msg = msg.strip
    to_name = to_name.strip

    return if msg == ''
    return if to_name == ''

    to_buddy = @client.buddy_list.buddy_named(to_name )

    if to_buddy.available?
      to_buddy.send_im(msg)
      @messages[Time.now]=[to_name,msg]
    end

    refresh_msgs
  end

  def load
    @client = Net::TOC.new(@user, @password)
    @client.connect
    sleep 3
    @messages = {}

    @buddies = []
    @client.buddy_list.each_group { |g, b| @buddies = @buddies + b }

    @client.on_im() do |message, buddy, auto_response|
      @messages[Time.now]=[buddy.screen_name, message.gsub(/<\/?[^>]*>/, "")]
      #buddy.send_im("#{message} responce")
      refresh_msgs
    end

    refresh
  end

  load

end

Screenshot:
IM Shoes Screenshot

Written by DanM

August 29, 2008 at 6:29 pm

Posted in Development, Hacking, Ruby

Leaving NetBeans for Emacs as my Ruby Editor

When I first started working with Ruby I used RadRails (which still has the best integrated test runner I have used). Various problems and crashes with RadRails along with exciting features being added to Ruby NetBeans enticed me to switch. I enjoyed NetBeans and some of its features, but the weight of my IDE began slowing me down. I didn’t have the control to tell it not to do some things that annoyed me.

My cofounder Ben had been using Emacs since learning Ruby. The quick speed that he could work with code, made me wonder what features I was actually using in my bloated IDE. It turned out that I had been getting annoyed with most of the advanced features because they stalled my system, crashed occasionally, or really weren’t that useful. An example of a cool feature that I hardly found useful would be Netbean’s Ruby code completion. In the past I had used Emacs to develop all my C and C++ code and I always liked it, so I decided to switch away from NetBeans and see how Emacs would hold up as my editor.

I have enjoyed using Emacs and think that I can more quickly work with code with far less hassle and frustration. I don’t expect to move back to a large IDE anytime soon. That being said there are definitely pros and cons to working with a full IDE vs an editor, and I find myself missing features sometimes. So I thought I would list out some of the pros and cons so others can weigh these issues themselves.

Reasons I miss NetBeans:

  • Graphical test integration linking to the lines of code in the stack trace
  • Navigate -> ‘Go to Declaration’, ‘Go to Test’, ‘Go to Action or View’
  • Easy code refactoring and renaming
  • Code completion, occasionally (e.g. was the Array function called include? or includes?)
  • File browsing with folder tree view
  • Graphical SVN browsing, easy branch, merge, get version, diff
  • Easy project file search (Yes, you can do, ‘grep -Ri ‘Logger’ lib/* bin/* app/* | grep -v .svn’, but that is a pain to type)
  • Easy code commenting (Apple-/ which toggles, is much nicer than Ctrl-x, Ctrl-m comment-region / uncomment-region which has to be added to your Emacs configuration)
  • Easy code formatting (Apple-shift-f, is easier than Ctrl-x, Ctrl-m indent-buffer / indent-region)

Reasons I hate NetBeans:

  • Frequently stalls (Especially if you have been out of the application for awhile)
  • Crashes (At least every couple days)
  • Not enough user control (stop indexing everything)
  • Magic behind the scenes (constantly connecting to SVN every time I saved a file)
  • Debugger didn’t work well (ended up running on the command line to get debugging)
  • Ruby Console didn’t work well (ended up using irb on the command line instead)
  • All tasks that NetBeans handles nicely for you like testing or migrations were slower (ended up running all rake task and tests on the command line)
  • Resource hog, making everything else I am running far slower
  • NetBeans ignores canceling tasks and continues trying to run them
  • Booting up takes a few minute before NetBeans completes its various start-up tasks (You can work during some of the start-up tasks, but it is slow and unresponsive until it has completed the tasks)

Things I like about Emacs:

  • Better/more key commands (I love kill/yank – the ability to kill until the end of line is incredibly useful)
  • Incredibly fast, if I am working live with someone I can jump in make an edit and be back live showing them the change on the project (Imagine having to boot NetBeans to do this)
  • Multiple copy and paste buffers. This is something you get with GNU Emacs on an X system. I can use both kill/yank from Emacs while also using my normal copy and paste to maintain two independent copy and paste buffers, very handy
  • Complete control over the editor, no longer do I have to deal with auto closing brackets or any other ‘helpful’ defaults the editor includes (You can turn them all off, but I get annoyed having to go change a bunch of settings)
  • An unexplainable feeling of being closer to the system and the code.

Written by DanM

August 8, 2008 at 3:27 pm

Posted in Development, Hacking, Misc, Ruby

Follow

Get every new post delivered to your Inbox.