The Devver Blog

A Boulder startup improving the way developers work.

Tracking down open files with lsof

The other day I was running in a weird error on Devver. After running around twenty test runs on the system, the component that actually runs individual unit tests was crashing due to “Too many open files – (Errno::EMFILE)”

Unfortunately, I didn’t know much more than that. Which files were being kept open? I knew that this component loaded quite a few files, and that by default, OS X only allows 256 open file descriptors (

ulimit -n

will tell you the default on your system). If this was a valid case of needing to load more files, I could just up the limit using

ulimit -n <bigger_number>

.

Fortunately, a quick Google or two pointed the way to

lsof

. Unfortunately, my Unix-fu is never nearly as good as I wish and I didn’t know much about this handy utility. But I quickly discovered that it’s very useful for tracking down problems like this. I quickly used

ps

to find the PID of the Devver process and then a quick

lsof -p <PID>

displayed all the files that the process had open. So easy!

Sure enough, there were a ton of redundant file handles to the file that we use to store information about the Devver run. Armed with this information, it was easy to find the buggy code where we called File.open but failed to ever close the file.

Unfortunately, I still don’t know how to write a good unit test for this case. I guess I could do something ugly like call sytem(“lsof -p pid | wc -l”) before and after calling the code and make sure the number of descriptors stays constant, but that’s really ugly. Is there a way to test this within Ruby? I’m open to ideas.

Still, it’s always good to learn more about a powerful Unix tool. I’m constanly amazed by the power and depth of the Unit tool set.

Written by Ben

October 9, 2008 at 12:23 pm

%d bloggers like this: