The Devver Blog

A Boulder startup improving the way developers work.

Posts Tagged ‘Tips & Tricks

Speeding up multi-browser Selenium Testing using concurrency

I haven’t used Selenium for awhile, so I took some time to dig into the options to get some mainline tests running against Caliper in multiple browsers. I wanted to be able to test a variety of browsers against our staging server before pushing new releases. Eventually this could be integrated into Continuous Integration (CI) or Continuous Deployment (CD).

The state of Selenium testing for Rails is currently in flux:

So there are multiple gems / frameworks:

I decided to investigate several options to determine which is the best approach for our tests.

selenium-on-rails

I originally wrote a couple example tests using the selenium-on-rails plugin. This allows you to browse to your local development web server at ‘/selenium’ and run tests in the browser using the Selenium test runner. It is simple and the most basic Selenium mode, but it obviously has limitations. It wasn’t easy to run many different browsers using this plugin, or use with Selenium-RC, and the plugin was fairly dated. This lead me to try simplest next thing, selenium-client

open '/'
assert_title 'Hosted Ruby/Rails metrics - Caliper'
verify_text_present 'Recently Generated Metrics'

click_and_wait "css=#projects a:contains('Projects')"
verify_text_present 'Browse Projects'

click_and_wait "css=#add-project a:contains('Add Project')"
verify_text_present 'Add Project'

type 'repo','git://github.com/sinatra/sinatra.git'
click_and_wait "css=#submit-project"
verify_text_present 'sinatra/sinatra'
wait_for_element_present "css=#hotspots-summary"
verify_text_present 'View full Hot Spots report'

view this gist

selenium-client

I quickly converted my selenium-on-rails tests to selenium-client tests, with some small modifications. To run tests using selenium-client, you need to run a selenium-RC server. I setup Sauce RC on my machine and was ready to go. I configured the tests to run locally on a single browser (Firefox). Once that was working I wanted to run the same tests in multiple browsers. I found that it was easy to dynamically create a test for each browser type and run them using selenium-RC, but that it was increadly slow, since tests run one after another and not concurrently. Also, you need to install each browser (plus multiple versions) on your machine. This led me to use Sauce Labs’ OnDemand.

browser.open '/'
assert_equal 'Hosted Ruby/Rails metrics - Caliper', browser.title
assert browser.text?('Recently Generated Metrics')

browser.click "css=#projects a:contains('Projects')", :wait_for => :page
assert browser.text?('Browse Projects')

browser.click "css=#add-project a:contains('Add Project')", :wait_for => :page
assert browser.text?('Add Project')

browser.type 'repo','git://github.com/sinatra/sinatra.git'
browser.click "css=#submit-project", :wait_for => :page
assert browser.text?('sinatra/sinatra')
browser.wait_for_element "css=#hotspots-summary"
assert browser.text?('View full Hot Spots report')

view this gist

Using Selenium-RC and Sauce Labs Concurrently

Running on all the browsers Sauce Labs offers (12) took 910 seconds. Which is cool, but way too slow, and since I am just running the same tests over in different browsers, I decided that it should be done concurrently. If you are running your own Selenium-RC server this will slow down a lot as your machine has to start and run all of the various browsers, so this approach isn’t recommended on your own Selenium-RC setup, unless you configure Selenium-Grid. If you are using¬† Sauce Labs, the tests run concurrently with no slow down. After switching to concurrently running my Selenium tests, run time went down to 70 seconds.

My main goal was to make it easy to write pretty standard tests a single time, but be able to change the number of browsers I ran them on and the server I targeted. One approach that has been offered explains how to setup Cucumber to run Selenium tests against multiple browsers. This basically runs the rake task over and over for each browser environment.

Althought this works, I also wanted to run all my tests concurrently. One option would be to concurrently run all of the Rake tasks and join the results. Joining the results is difficult to do cleanly or you end up outputting the full rake test output once per browser (ugly when running 12 times). I took a slightly different approach which just wraps any Selenium-based test in a run_in_browsers block. Depending on the options set, the code can run a single browser against your locally hosted application, or many browsers against a staging or production server. Then simply create a separate Rake task for each of the configurations you expect to use (against local selenium-RC and Sauce Labs on demand).

I am pretty happy with the solution I have for now. It is simple and fast and gives another layer of assurances that Caliper is running as expected. Adding additional tests is simple, as is integrating the solution into our CI stack. There are likely many ways to solve the concurrent selenium testing problem, but I was able to go from no Selenium tests to a fast multi-browser solution in about a day, which works for me. There are downsides to the approach, the error output isn’t exactly the same when run concurrently, but it is pretty close.¬† As opposed to seeing multiple errors for each test, you get a single error per test which includes the details about what browsers the error occurred on.

In the future I would recommend closely watching Webrat and Capybara which I would likely use to drive the Selenium tests. I think the eventual merge will lead to the best solution in terms of flexibility. At the moment Capybara doesn’t support selenium-RC, and the tests I originally wrote didn’t convert to the Webrat API as easily as directly to selenium-client (although setting up Webrat to use Selenium looks pretty simple). The example code given could likely be adapted easily to work with existing Webrat tests.

namespace :test do
  namespace :selenium do

    desc "selenium against staging server"
    task :staging do
      exec "bash -c 'SELENIUM_BROWSERS=all SELENIUM_RC_URL=saucelabs.com SELENIUM_URL=http://caliper-staging.heroku.com/  ruby test/acceptance/walkthrough.rb'"
    end

    desc "selenium against local server"
    task :local do
      exec "bash -c 'SELENIUM_BROWSERS=one SELENIUM_RC_URL=localhost SELENIUM_URL=http://localhost:3000/ ruby test/acceptance/walkthrough.rb'"
    end
  end
end

view this gist

require "rubygems"
require "test/unit"
gem "selenium-client", ">=1.2.16"
require "selenium/client"
require 'threadify'

class ExampleTest  1
      errors = []
      browsers.threadify(browsers.length) do |browser_spec|
        begin
          run_browser(browser_spec, block)
        rescue => error
          type = browser_spec.match(/browser\": \"(.*)\", /)[1]
          version = browser_spec.match(/browser-version\": \"(.*)\",/)[1]
          errors < type, :version => version, :error => error}
        end
      end
      message = ""
      errors.each_with_index do |error, index|
        message +="\t[#{index+1}]: #{error[:error].message} occurred in #{error[:browser]}, version #{error[:version]}\n"
      end
      assert_equal 0, errors.length, "Expected zero failures or errors, but got #{errors.length}\n #{message}"
    else
      run_browser(browsers[0], block)
    end
  end

  def run_browser(browser_spec, block)
    browser = Selenium::Client::Driver.new(
                                           :host => selenium_rc_url,
                                           :port => 4444,
                                           :browser => browser_spec,
                                           :url => test_url,
                                           :timeout_in_second => 120)
    browser.start_new_browser_session
    begin
      block.call(browser)
    ensure
      browser.close_current_browser_session
    end
  end

  def test_basic_walkthrough
    run_in_all_browsers do |browser|
      browser.open '/'
      assert_equal 'Hosted Ruby/Rails metrics - Caliper', browser.title
      assert browser.text?('Recently Generated Metrics')

      browser.click "css=#projects a:contains('Projects')", :wait_for => :page
      assert browser.text?('Browse Projects')

      browser.click "css=#add-project a:contains('Add Project')", :wait_for => :page
      assert browser.text?('Add Project')

      browser.type 'repo','git://github.com/sinatra/sinatra.git'
      browser.click "css=#submit-project", :wait_for => :page
      assert browser.text?('sinatra/sinatra')
      browser.wait_for_element "css=#hotspots-summary"
      assert browser.text?('View full Hot Spots report')
    end
  end

  def test_generate_new_metrics
    run_in_all_browsers do |browser|
      browser.open '/'
      browser.click "css=#add-project a:contains('Add Project')", :wait_for => :page
      assert browser.text?('Add Project')

      browser.type 'repo','git://github.com/sinatra/sinatra.git'
      browser.click "css=#submit-project", :wait_for => :page
      assert browser.text?('sinatra/sinatra')

      browser.click "css=#fetch"
      browser.wait_for_page
      assert browser.text?('sinatra/sinatra')
    end
  end

end

view this gist

Written by DanM

April 8, 2010 at 10:07 am

A command-line prompt with timeout and countdown

Have you ever started a long operation and walked away from the computer, and come back half an hour later only to find that the process is hung up waiting for some user input? It’s a sub-optimal user experience, and in many cases it can be avoided by having the program choose a default if the user doesn’t respond within a certain amount of time. One example of this UI technique in the wild is powering off your computer – most modern operating systems will pop up a dialogue to confirm or cancel the shutdown, with a countdown until the shutdown proceeds automatically.

This article is about how to achieve the same effect in command-line programs using Ruby.

Let’s start with the end result. We want to be able to call our method like this:

puts ask_with_countdown_to_default("Do you like pie?", 30.0, false)

We pass in a question, a (possibly fractional) number of seconds to wait, and a default value. The method should prompt the user with the given question and a visual countdown. If the user types ‘y’ or ‘n’, it should immediately return true or false, respectively. Otherwise when the countdown expires it should return the default value.

Here’s a high-level implementation:

def ask_with_countdown_to_default(question, seconds, default)
  with_unbuffered_input($stdin) do
    countdown_from(seconds) do |seconds_left|
      write_then_erase_prompt(question, seconds_left) do
        wait_for_input($stdin, seconds_left % 1) do
          case char = $stdin.getc
          when ?y, ?Y then return true
          when ?n, ?N then return false
          else                  # NOOP
          end
        end
      end
    end
  end
  return default
ensure
  $stdout.puts
end                             # ask_with_countdown_to_default

Let’s take it step-by-step.

By default, *NIX terminals operate in “canonical mode”, where they buffer a line of input internally and don’t send it until the user hits RETURN. This is so that the user can do simple edits like backspacing and retyping a typo. This behavior is undesirable for our purposes, however, since we want the prompt to respond as soon as the user types a key. So we need to temporarily alter the terminal configuration.

  with_unbuffered_input($stdin) do

We use the POSIX Termios library, via the ruby-termios gem, to accomplish this feat.

def with_unbuffered_input(input = $stdin)
  old_attributes = Termios.tcgetattr(input)
  new_attributes = old_attributes.dup
  new_attributes.lflag &= ~Termios::ECHO
  new_attributes.lflag &= ~Termios::ICANON
  Termios::tcsetattr(input, Termios::TCSANOW, new_attributes)

  yield
ensure
  Termios::tcsetattr(input, Termios::TCSANOW, old_attributes)
end                             # with_unbuffered_input

POSIX Termios defines a set of library calls for interacting with terminals. In our case, we want to disable some of the terminal’s “local” features – functionality the terminal handles internally before sending input on to the controlling program.

We start by getting a snapshot of the terminal’s current configuration. Then we make a copy for our new configuration. We are interested in two flags: “ECHO” and “ICANON”. The first, ECHO, controls whether the terminal displays characters that the user has types. The second controls canonical mode, which we explained above. After turning both flags off, we set the new configuration and yield. After the block is finished, or if an exception is raised, we ensure that the original terminal configuration is reinstated.

Now we need to arrange for a countdown timer.

    countdown_from(seconds) do |seconds_left|

Here’s the implementation:

def countdown_from(seconds_left)
  start_time   = Time.now
  end_time     = start_time + seconds_left
  begin
    yield(seconds_left)
    seconds_left = end_time - Time.now
  end while seconds_left > 0.0
end                             # countdown_from

First we calculate the wallclock time at which we should stop waiting. Then we begin looping, yielding the number of seconds left, and then when the block returns recalculating the number. We keep this up until the time has expired.

Next up is writing, and re-writing, the prompt.

      write_then_erase_prompt(question, seconds_left) do

This method is implemented as follows:

def write_then_erase_prompt(question, seconds_left)
  prompt_format = "#{question} (y/n) (%2d)"
  prompt = prompt_format % seconds_left.to_i
  prompt_length = prompt.length
  $stdout.write(prompt)
  $stdout.flush

  yield

  $stdout.write("\b" * prompt_length)
  $stdout.flush
end                             # write_then_erase_prompt

We format and print a prompt, flushing the output to insure that it is displayed immediately. The prompt includes a count of the number of seconds remaining until the query times out. In order to make it a nice visually consistent length, we use a fixed-width field for the countdown (“%2d”). Note that we don’t use

puts

to print the prompt – we don’t want it to advance to the next line, because we want to be able to dynamically rewrite the prompt as the countdown proceeds.

After we are done yielding to the block, we erase the prompt in preparation for the next cycle. In order to erase it we create and output string of backspaces (“\b”) the same length as the prompt.

Now we need a way to wait until the user types something, while still periodically updating the prompt.

        wait_for_input($stdin, seconds_left % 1) do

We pass

wait_for_input

an input stream and a (potentially fractional) number of seconds to wait. In this case we only want to wait until the next second-long “tick” so that we can update the countdown. So we pass in the remainder of dividing seconds_left by 1. E.g. if seconds_left was 5.3, we would set a timeout of 0.3 seconds. After 3/10 of a second of waiting for input, the wait would time out, the prompt would be erased and rewritten to show 4 seconds remaining, and then we’d start waiting for input again.

Here’s the implementation of

wait_for_input

:

def wait_for_input(input, timeout)
  # Wait until input is available
  if select([input], [], [], timeout)
    yield
  end
end                             # wait_for_input

We’re using

Kernel#select

to do the waiting. The parameters to

#select

are a set of arrays – one each for input, output, and errors. We only care about input, so we pass the input stream in the first array and leave the others blank. We also pass how long to wait until timing out.

If new input is detected,

select

returns an array of arrays, corresponding to the three arrays we passed in. If it times out while waiting, it returns

nil

. We use the return value to determine whether to execute the given block or note. If there is input waiting we yield to the block; otherwise we just return.

While it takes some getting used to, handling IO timeouts with

select

is safer and more reliable than using the

Timeout

module. And it’s less messy than rescuing

Timeout::Error

every time a read times out.

Finally, we need to read and interpret the character the user types, if any.

          case char = $stdin.getc
          when ?y, ?Y then return true
          when ?n, ?N then return false
          else                  # NOOP
          end

If the user types ‘y’ or ‘n’ (or uppercase versions of the same), we return

true

or

false

, respectively. Otherwise, we simply ignore any characters the user types. Typing characters other than ‘y’ or ‘n’ will cause the loop to be restarted.

Note the use of character literals like

?y

to compare against the integer character code returned by

IO#getc

. We could alternately use

Integer#chr

to convert the character codes into single-character strings, if we wanted.

Wrapping up, we make sure to return the default value should the timeout expire without any user input; and we output a newline to move the cursor past our prompt.

  return default

And there you have it; a yes/no prompt with a timeout and a visual countdown. Static text doesn’t really capture the effect, so rather than include sample output I’ll just suggest that you try the code out for yourself (sorry, Windows users, it’s *NIX-only).

Full source for this article at: http://gist.github.com/148765

Written by avdi

July 16, 2009 at 10:45 pm

Spellcheck your files with Aspell and Rake

We recently redid our website. The new site included a new design and much more content explaining what we do. We wanted a quick way to check over everything and make sure we didn’t miss any spelling errors or typos. First I started looking for a web service that could scan the site for spelling errors. I found spellr.us, which is nice but would only catch errors once they were live. It also can’t scan all of the pages which require being logged in.

I was pairing with Avdi who thought we should just run Aspell, which worked out great. We were originally trying to just create a simple Emacs macro to go through all our HTML files and check them but in the end created simple Rake tasks, which makes it really easy to integrate spellcheck into CI. After Avdi figured out the commands we needed to use on each file to get the information we needed from Aspell, it was easy to just wrap the command using Rake’s FileList. To keep everyone on the same setup, we created a local dictionary of words to ignore or accept and keep that checked into source control as well.

The final solution grabs all the files you want to spell check, then runs them through Aspell with HTML filtering. We have two tasks: one that runs in interactive mode the the user can fix mistakes and one mode for CI that just fails if it finds any errors.

def run_spellcheck(file,interactive=false)
  if interactive
    cmd = "aspell -p ./config/devver_dictionary -H check #{file}"
    puts cmd
    system(cmd)
    [true,""]
  else
    cmd = "aspell -p ./config/devver_dictionary -H list  'spellcheck:interactive'

namespace :spellcheck do
  files = FileList['app/views/**/*.html.erb']

  desc "Spellcheck interactive"
  task :interactive do
    files.each do |file|
      run_spellcheck(file,true)
    end
    puts "spelling check complete"
  end

  desc "Spellcheck for ci"
  task :ci do
    files.each do |file|
      success, results = run_spellcheck(file)
      unless success
        puts results
        exit 1
      end
    end
    puts "no spelling errors"
    exit 0
  end
end

view this gist

Written by DanM

May 26, 2009 at 8:33 am