The Devver Blog

A Boulder startup improving the way developers work.

Beware of pipe duplication in subprocesses

I’ve been writing a lot lately about managing subprocesses in Ruby, and in that vein I thought I’d write up an interesting subprocess gotcha we encountered recently.

I won’t go into the details of how we discovered the issue, except to say that it manifested when using the Main and Servolux gems (both of which I highly recommend) together to start daemon processes. The problem can be boiled down to this brief snippet, distilled by Tim Pease:

ruby -e 'IO.popen("ruby -e \"STDOUT.dup; STDOUT.close; sleep\"").read

If you try to execute that line, the #read will hang indefinitely, even though STDOUT was closed in the child process. Why? Tim explains:

“…the parent script will block waiting for pipe to close in the child process. However, because there is a duplicate of the STDOUT file descriptor in the child, both must be closed or the parent will never unblock from the read. […] ALL file descriptors have to be closed before an EOF is sent through the pipe to the parent. Hence, the read will never return.”

The moral of this story: don’t try to read from a subprocess which duplicates STDOUT or STDERR handles. Or if you do, take precautions by using calls which don’t wait for an EOF – such as #readpartial – instead of #read.

I can take no credit for solving this one. All the honor goes to Ara T. Howard, who expended much mental energy and beer to get to the bottom of the problem; and Tim Pease, who neatly summarised the issue and explained it in terms that even I could understand. I’m documenting it here in case anyone else runs into it.


Written by avdi

October 22, 2009 at 6:00 am

%d bloggers like this: