The Devver Blog

A Boulder startup improving the way developers work.

Unit Testing Filesystem Interaction

Like most Rubyists, I write unit tests to verify the non-trivial parts of my code. I also try to use mocks and stubs to stub out interactions with systems external to my code, like network services.

For the most part, this works fine. But I’ve always struggled to find a good way to test interaction with the filesystem (which can often be non-trivial and therefore should be tested). On the one hand, the filesystem could be considered “external” and mocked out. But on the other hand, the filesystem is accessible when the tests run. In this way, the filesystem is sort of like a local database – it could be mocked out, but it doesn’t have to be, and there are tradeoffs to both approaches.

Over the past year or so, I’ve tried out a few approaches for testing interactions with the filesystem, each of which I’ll explain below. Since none of the approaches met my needs, Avdi and I built a new testing library, which I’ll introduce below.

Mocking the file system.

Sometimes, it is simplest to just mock the interaction with the filesystem. This works well for single calls to methods like

File#read

or

File#exist?

(these examples use Mocha):

File.stubs(:read).returns("file contents")
File.stubs(:exist?).returns(true)

However, this approach breaks down when you want to test more complex code, which, of course, is the code you’re more likely to want to test thoroughly. For instance, imagine trying to set up mocks/stubs for the following method (which atomically rewrites the contents of a file):

require 'tempfile'

class Rewriter

  def rewrite_file!(target_path)
    backup_path = target_path + '.bak'
    FileUtils.mv(target_path, backup_path)
    Tempfile.open(File.basename(target_path)) do |outfile|
      File.open(backup_path) do |infile|
        infile.each_line do |line|
          outfile.write(yield(line))
        end
      end
      outfile.close
      FileUtils.cp(outfile.path, target_path)
    end
  rescue Exception
    if File.exist?(backup_path)
      FileUtils.mv(backup_path, target_path)
    end
    raise
  end

end

Now imagine setting up those same mocks/stubs for each of the five or so tests you’d want to test that method. It gets messy.

Even more importantly, mocking/stubbing out methods ties your tests to a specific implementation. For instance, if you use the above stub (

File.stubs(:read).returns("file contents")

) in your test and then refactor your implementation to use, say,

File.readlines

, you’ll have to update your tests. No good.

MockFS

MockFS is a library that mocks out the entire filesystem. It allows you write test code like this:

require 'test/unit'
require 'mockfs'

class TestMoveLog < Test::Unit::TestCase

  def test_move_log
    # Set MockFS to use the mock file system
    MockFS.mock = true

    # Fill certain directories
    MockFS.fill_path '/var/log/httpd/'
    MockFS.fill_path '/home/francis/logs/'

    # Create the access log
    MockFS.file.open( '/var/log/httpd/access_log', File::CREAT ) do |f|
      f.puts "line 1 of the access log"
    end

    # Run the method under test
    move_log

    # Test that it was moved, along with its contents
    assert( MockFS.file.exist?( '/home/francis/logs/access_log' ) )
    assert( !MockFS.file.exist?( '/var/log/httpd/access_log' ) )
    contents = MockFS.file.open( '/home/francis/logs/access_log' ) do |f|
      f.gets( nil )
    end
    assert_equal( "line 1 of the access log\n", contents )
  end
end

Although I suspect MockFS would be a great fit for some projects, I ended up running into issues.

First of all, it depends on a library (extensions) that can have strange monkey-patching conflicts with other libraries. For example, compare this:

require 'faker'
puts [].respond_to?(:shuffle) # true

to this:

require 'extensions/all'
require 'faker'
puts [].respond_to?(:shuffle) # false

Secondly, as you’ll notice in the above example, using MockFS requires you to use methods like

MockFS.file.exist?

instead of just

File.exist?

. This works fine if you’re only testing your own code. However, if your code calls any libraries that use filesystem methods, MockFS won’t work.

(Note: There is a way to mock out the default filesystem methods, but it’s experimental. From the MockFS documentation:

“Reading the testing example above, you may be struck by one thing: Using MockFS requires you to remember to reference it everywhere, making calls such as MockFS.file_utils.mv instead of just FileUtils.mv. As another option, you can use File, FileUtils, and Dir directly, and then in your tests, substitute them by including mockfs/override.rb. I’d recommend using these with caution; substituting these low-level classes can have unpredictable results. “)

All that said, MockFS is probably your best option if you’re only testing your code and you want to mock out files that you can’t actually interact with – for instance, if you need to test that a method reads/writes a file in

/etc

(although for the sake of testability, it’s generally good to avoid hardcoding fully-qualified paths in your code).

FakeFS is another library that uses this approach. I haven’t used it personally, but it looks quite nice.

Creating temp files and directories (with Construct)

Besides mocking the filesystem, another option is to have tests interact with actual files and directories on disk. The advantages are that the test code can be simpler to write and you don’t have to use any special filesystem methods.

Of course, as always, you want the test itself to contain all the relevant setup and teardown – you don’t want your tests to depend upon some set of files that have no explicit connection to the test itself (or create files that aren’t cleaned up).

To make this easy, we created a new library called Construct. Construct makes test setup simple by providing helpers to create temporary files and directories. It takes care of the cleanup by automatically deleting the directories and files that are created within the test. And because it creates regular files and directories, you can use plain old Ruby filesystem methods in your code and tests.

To install Construct, simply run:

# gem install devver-construct --source http://gems.github.com

Using Construct, you can write code like this:

require 'construct'

class ExampleTest < Test::Unit::TestCase
  include Construct::Helpers

  def test_example
    within_construct do |construct|
      construct.directory 'alice/rabbithole' do |dir|
        dir.file 'white_rabbit.txt', "I'm late!"
        assert_equal "I'm late!", File.read('white_rabbit.txt')
      end
    end
  end

end

Let’s look at each line in more detail.

    within_construct do |construct|

When you call

within_construct

, a temporary directory is created. All files and directories are, by default, created within that temporary directory and the temporary directory is always deleted before

within_construct

completes.

The block argument (

construct

) is a Pathname object with some additional methods (

#directory

and

#file

, which I’ll explain below). You can use this object to get the path to the temporary directory created by Construct and easily create files and directories.

Note that, by default, the working directory is changed to the temp dir within the block provided to

within_construct

.

      construct.directory 'alice/rabbithole' do |dir|

Here we are using the

construct

object to create a new directory within the temp directory. As you can see, you can create nested directories like

alice/rabbithole

in one step. The block argument (

dir

) is again a Pathname object with the same added functionality noted above.

Just like before, the working directory is changed to the newly created directory (in this case,

alice/rabbithole

) within the block.

        dir.file 'white_rabbit.txt', "I'm late!"

Here we use the

dir

object to create a file. In this case, the file will be empty. However, it’s easy to provide file contents using either an optional parameter or the return value of the supplied block:

within_construct do |construct|
  construct.file('foo.txt','Here is some content')
  construct.file('bar.txt') do
  <<-EOS
  The block will return this string, which will be used as the content.
  EOS
  end
end

As a more real-world example, here’s how you could use Construct to start testing the

#rewrite_file!

method we looked at before:

require 'test/unit'
require 'construct'
require 'shoulda'

class RewriterTest < Test::Unit::TestCase
  include Construct::Helpers

  context "#rewrite_file!" do

    should "alter each line in file" do
      within_construct do |c|
        c.file('bar/foo.txt',"a\nb\nc\n")
        Rewriter.new.rewrite_file!('bar/foo.txt') do |line|
          line.upcase
        end
        assert_equal "A\nB\nC\n", File.read('bar/foo.txt')
      end
    end

    should "not alter file if exception is raised" do
      within_construct do |c|
        c.file('foo.txt', "1\n2\nX\n")
        assert_raises ArgumentError do
          Rewriter.new.rewrite_file!('foo.txt') do |line|
            Integer(line)*2
          end
        end
        assert_equal "1\n2\nX\n", File.read('foo.txt')
      end
    end

  end

end

You can learn more at the project page (both the README and the tests have more examples).

(As an aside, since Construct changes the working directory, it doesn’t play nicely with

ruby-debug

. Specifically, if you place a breakpoint within a block, you’ll see the message “No sourcefile available for test/unit/foo_test.rb” and you won’t be able to view the source. If anyone knows an easy way to make

Dir.chdir

work with

ruby-debug

, I’d very much appreciate some help!)

Conclusion

We’ve been moving our filesystem tests over to using Construct and so far have found it to be very useful. How do you test interactions with the filesystem? Do you use one of the above approaches, or something else? Or do you skip testing the filesystem altogether?

Advertisements

Written by Ben

August 25, 2009 at 9:51 am

Posted in Hacking, Testing

Tagged with , ,

20 Responses

Subscribe to comments with RSS.

  1. Just earlier today I was thinking about how I was gonna test this small generator I'm writing and this seems to be the answer. Much thanks, and great work. 🙂

    Danny Tatom

    August 27, 2009 at 1:51 am

  2. Glad to hear it. After using it, please let me know if you have any problems or suggestions.

    bhb

    August 27, 2009 at 11:20 am

  3. I love this idea. Always feels like I have to implement a 1/2 assed version of this whenever I test file system stuff. I'd like to port this idea to .NET as I do that for my day job, so I'll hit you up if I do.

    Brian

    August 27, 2009 at 11:49 am

  4. Yep, we'd implemented two half-assed versions in our code, so we decided to put them together to make something whole :).

    Definitely feel free to contact me with any questions when you do the port to .NET.

    bhb

    August 27, 2009 at 8:10 pm

  5. What can I say? This blog rocks. I really appreciated the code you shared here using beanstalk and now just when I had a need to mock the filesystem to test another project I'm working on, you've come through again.

    Thanks for being there.

    Victor

    August 30, 2009 at 1:53 pm

  6. […] Unit Testing Filesystem Interaction – The Devver Blog […]

  7. Yep, we'd implemented two half-assed versions in our code, so we decided to put them together to make something whole :).

    bhb

    August 30, 2009 at 2:18 pm

  8. Awesome, glad to hear it's helpful!

    bhb

    August 30, 2009 at 2:18 pm

  9. Thank you for this! I've run into this issue several times and it's nice to have a gem which makes testing file systems easier. I'm a big believer in testing the full stack when possible, and mocking out file calls is a pain. However, I do have some questions about the implementation.

    1) Why change the current working directory? I think this may cause more problems than it solves because there are things which rely on the working directory staying the same (beyond ruby-debug I believe). There are also many times one is performing file operations on the full path (like when using RAILS_ROOT).

    I realize you need to be careful not to override existing, non-temp files, but if one tags a file with construct I think it's safe to assume it's temporary. You may be able to build some safe-checks to avoid accidentally deleting a non-tmp file. If there is a conflict I think this should be explicit in the test and it should be up to the developer to specify a separate temp path.

    2) Is there a reason you went with the within_construct block instead of hooking into setup/teardown methods? Maybe add this as an option.

    Good going though, I look forward to seeing how this progresses.

    Ryan Bates

    September 1, 2009 at 11:59 am

  10. Ryan,

    Good questions.

    1. We wanted to change the working directory (to the temp dir we've created) because then code like this works as expected:

    within_construct |c|
    c.file('foo.txt')
    assert File.exist?('foo.txt')
    end

    But if you don't want to change the working dir (and I have several tests that don't, because of the problems you mentioned), you can pass 'false' to 'within_construct', like this:

    within_construct(false) do |c|
    c.file('foo.txt')
    assert File.exist?(c+'foo.txt')
    end

    As for operations that test the full path, I'd recommend a solution like FakeFS, because Construct won't work for this scenario (it simply creates all files in a temp dir and then deletes that temp dir).

    It certainly would be possible to have a different mode that writes files anywhere, but as you noted, we'd have to be really careful about not overwriting existing files, deleting files correctly, etc. I think in general, it's better to write methods that don't depend on full paths (for testability).

    2. Frankly, this is only because I personally think setup/teardown methods are evil (Avdi is slowly starting to convince me it might not be so bad in a system like RSpec that has contexts, but I'm still skeptical). Of course, that's just me. I'll make a ticket to add methods like 'start' and 'stop' that would allow people to use construct in setup/teardown methods.

    bhb

    September 1, 2009 at 3:51 pm

  11. Interesting…I usually try to make my code that interacts the the filesystem configurable so I can set the root dir of my filesystem interactions to a temp dir when in a test environment. You compare the filesystem to a local database. I see it the same way. Why not configure your FS entry points (or root dirs) the same way rails configures different databases for each environment? I suppose you don't always have that luxury, and that's where your gem helps out.

    Ben Marini

    September 3, 2009 at 12:14 am

  12. Ben, thanks for the question.

    We do the same thing – making file system roots configurable – in our own code. While the current directory switching feature is sometimes useful, we don't see that as the primary value proposition of Construct.

    Construct makes three main aspects of file-based testing: 1. Creating a uniquely named temp directory (Ruby only provides facilities to create uniquely named *files*); 2. Ensuring that the temp directory tree will be cleaned up after the test; and 3) declaratively setting up a tree of directories and files for your code under test to interact with. All three of these features are applicable to testing code with a configurable filesystem root.

    Avdi

    September 3, 2009 at 7:16 am

  13. That should read "Construct makes three main aspects of file-based testing *easier*".

    Avdi

    September 3, 2009 at 7:17 am

  14. […] Unit Testing Filesystem Interaction – The Devver Blog […]

  15. Ryan, just wanted to let you know that #2 is now implemented in the latest version of Construct 🙂

    avdi

    September 9, 2009 at 9:53 pm

  16. Any hints on how I could use Construct with my Cucumber scenarios?

    schlick

    September 14, 2009 at 11:18 pm

  17. We recently added a non-block-form way to create and destroy a Construct container for just this reason. Here's an example of how to use it with Cucumber: http://gist.github.com/187353

    avdi

    September 15, 2009 at 8:01 am

  18. […] found a gem that does precisely what I needed named Construct.  Unfortunately there is a bug with Ruby 1.8.6 on Windows in regards to clean up of temp […]

  19. great work. though personally, i would prefer to see a unit test stub file system interaction, but let the real file system interaction get tested during integration testing.

    moonmaster9000

    January 29, 2010 at 3:27 pm

  20. Why use Dir.chdir at all though?

    EE

    August 24, 2012 at 11:01 pm


Comments are closed.

%d bloggers like this: