The Devver Blog

A Boulder startup improving the way developers work.

A Dozen (or so) Ways to Start Subprocesses in Ruby: Part 3

In part 1 and part 2 of this series, we took a look at some of Ruby’s built-in ways to start subprocesses. In this article we’ll branch out a bit, and examine some of the tools available to us in Ruby’s Standard Library. In the process, we’ll demonstrate some lesser-known libraries.

Helpers

First, though, let’s recap some of our boilerplate code. Here’s the preamble code which is common to all of the demonstrations in this article:

require 'rbconfig'

$stdout.sync = true

def hello(source, expect_input)
  puts "[child] Hello from #{source}"
  if expect_input
    puts "[child] Standard input contains: \"#{$stdin.readline.chomp}\""
  else
    puts "[child] No stdin, or stdin is same as parent's"
  end
  $stderr.puts "[child] Hello, standard error"
  puts "[child] DONE"
end

THIS_FILE = File.expand_path(__FILE__)

RUBY = File.join(Config::CONFIG['bindir'], Config::CONFIG['ruby_install_name'])

#hello is the method which we will be calling in a Ruby subprocess. It reads some text from STDIN and writes to both STDOUT and STDERR.

THIS_FILE and RUBY contain full paths for the demo source file and the the Ruby interpreter, respectively.

Method #6: Open3

The Open3 library defines a single method, Open3#popen3(). #popen3() behaves similarly to the Kernel#popen() method we encountered in part 2. If you remember from that article, one drawback to the #popen() method was that it did not give us a way to capture the child process’ STDERR stream. "]Open3#popen3() addresses this deficiency.

Open3#popen3() is used very similarly to Kernel#popen() (or Kernel#open() with a ‘|’ argument). The difference is that in addition to STDIN and STDOUT handles, popen3() yields a STDERR handle as well.

puts "6. Open3"
require 'open3'
include Open3
popen3(RUBY, '-r', THIS_FILE, '-e', 'hello("Open3", true)') do
  |stdin, stdout, stderr|
  stdin.write("hello from parent")
  stdin.close_write
  stdout.read.split("\n").each do |line|
    puts "[parent] stdout: #{line}"
  end
  stderr.read.split("\n").each do |line|
    puts "[parent] stderr: #{line}"
  end
end
puts "---"

When we execute this code, the result shows that we have captured the subprocess’ STDERR output:

6. Open3
[parent] stdout: [child] Hello from Open3
[parent] stdout: [child] Standard input contains: "hello from parent"
[parent] stdout: [child] DONE
[parent] stderr: [child] Hello, standard error
---

Method #7: PTY

All of the methods we have considered up to this point have shared a common limitation: they are not very well-suited to interfacing with highly interactive subprocesses. They work well for “filter”-style commands, which read some input, produce some output, and then exit. But when used with interactive subprocesses which wait for input, produce some output, and then wait for more input (etc.), their use can result in deadlocks. In a typical deadlock scenario, the expected output is never produced because input is still stuck in the input buffer, and the program hangs forever as a result. This is why, in previous examples, we have been careful to call #close_write on subprocess input handles before reading any output.

Ruby ships with a little-known and poorly-documented standard library called “pty”. The pty library is an interface to BSD pty devices. What is a pty device? In BSD-influenced UNIXen, such as Linux or OS X, a pty is a “pseudoterminal”. In other words, it’s a terminal device that isn’t attached to a physical terminal. If you’ve used a terminal program in Linux or OS X, you’ve probably used a pty without realizing it. GUI Terminal emulators, such as xterm, GNOME Terminal, and Terminal.app often use a pty device behind the scenes to communicate with the OS.

What does this mean for us? It means if we’re running Ruby on UNIX, we have the ability to start our subprocesses inside a virtual terminal. We can then read from and write to that terminal as if our program were a user sitting in front of a terminal, typing in commands and reading responses.

Here’s how it’s used:

puts "7. PTY"
require 'pty'
PTY.spawn(RUBY, '-r', THIS_FILE, '-e', 'hello("PTY", true)') do
  |output, input, pid|
  input.write("hello from parent\n")
  buffer = ""
  output.readpartial(1024, buffer) until buffer =~ /DONE/
  buffer.split("\n").each do |line|
    puts "[parent] output: #{line}"
  end
end
puts "---"

And here is the output:

7. PTY
[parent] output: [child] Hello from PTY
[parent] output: hello from parent
[parent] output: [child] Standard input contains: "hello from parent"
[parent] output: [child] Hello, standard error
[parent] output: [child] DONE
---

There are a few of points to note about this code. First, we don’t need to call #close_write or #flush on the process input handle. However, the newline at the end of “Hello from parent” is essential. By default, UNIX terminal devices buffer input until they see a newline. If we left off the newline, the subprocess would never finish waiting for input.

Second, because the subprocess is running asynchronously and independently from the parent process, we have no way of knowing exactly when it has finished reading input and producing output of its own. We deal with this by buffering output until we see a marker (“DONE”).

Third, you may notice that “hello from parent” appears twice in the output – once as part of the parent process output, and once as part of the child output. That’s because another default behaviour for UNIX terminals is to echo any input they receive back to the user. This is what enables you to see what you’ve just typed when working at the command line.

You can alter these default terminal device behaviours using the Ruby “termios” gem.

Note that both STDOUT and STDERR were captured in the subprocess output. From the perspective of the pty user, standard output and standard error streams are indistinguishable – it’s all just output. That means using pty is probably the only way to run a subprocess and capture standard error and standard output interleaved in the same way we would see if we ran the process manually from a terminal window. Depending on the application, this may be a feature or a drawback.

You can execute PTY.spawn() without a block, in which case it returns an array of output, input, and PID. If you choose to experiment with this style of calling PTY.spawn(), be aware that you may need to rescue the PTY::ChildExited exception, which is thrown whenever the child process finally exits.

If you’re interested in reading more code which uses the pty library, the Standard Library also includes a library called “expect.rb”. expect.rb is a basic Ruby reimplementation of the classic “expect” utility written using pty.

Method #8: Shell

More obscure even than the pty library is Ruby’s Shell library. Shell is, to my knowledge, totally undocumented and rarely used. Which is a shame, because it implements some interesting ideas.

Shell is an attempt to emulate a basic UNIX-style shell environment as an internal DSL within Ruby. Shell commands become Ruby methods, command-line flags become method parameters, and IO redirection is accomplished via Ruby operators.

Here’s an invocation of our standard example subprocess using Shell:

puts "8. Shell"
require 'shell'
Shell.def_system_command :ruby, RUBY
shell = Shell.new
input  = 'Hello from parent'
process = shell.transact do
  echo(input) | ruby('-r', THIS_FILE, '-e', 'hello("shell.rb", true)')
end
output = process.to_s
output.split("\n").each do |line|
  puts "[parent] output: #{line}"
end
puts "---"

And here is the output:

8. Shell
[child] Hello, standard error
[parent] output: [child] Hello from shell.rb
[parent] output: [child] Standard input contains: "Hello from parent"
[parent] output: [child] DONE
---

We start by defining the Ruby executable as a shell command by calling Shell.def_system_command. Then we instantiate a new Shell object. We construct the subprocess within a Shell#transact block. To have the process read a string from the parent process, we set up a pipeline from the echo built-in command to the Ruby invocation. Finally, we ensure the process is finished and collect its output by calling #to_s on the transaction.

Note that the child process’ STDERR stream is shared with the parent, not captured as part of the process output.

There is a lot going on here, and it’s only a very basic example of Shell’s capabilities. The Shell library contains many Ruby-friendly reimplementations of common UNIX userspace commands, and a lot of machinery for coordinating pipelines of concurrent processes. If your interest is piqued I recommend reading over the Shell source code and experimenting within IRB. A word of caution, however: the Shell library isn’t maintained as far as I know, and I ran into a couple of outright bugs in the process of constructing the above example. It may not be suitable for use in production code.

Conclusion

In this article we’ve looked at three Ruby standard libraries for executing subprocesses. In the next and final article we’ll examine some publicly available Rubygems that provide even more powerful tools for starting, stopping, and interacting with subprocesses within Ruby.

About these ads

Written by avdi

October 12, 2009 at 1:11 pm

10 Responses

Subscribe to comments with RSS.

  1. […] you might want to subscribe to the RSS feed for updates on this topic.Powered by WP Greet BoxThe third part of my Ruby subprocesses series just went up over at the Devver blog. In it I cover the Open3, PTY, and Shell standard […]

  2. Thanks for the discussion.

    It seems that each of these examples will open an new command window. Is there a way to run the subprocess in the same command window.

    E. g. I am running a ruby script from a ruby program and would prefer not to have a new command window open, but the script just run in the original shell window. Any ideas.

    Thanks

    mark

    October 26, 2009 at 10:06 am

  3. Mark, none of these techniques should have any interaction with your desktop windowing environment. They create new processes, not new windows. If you are seeing new windows opened when trying any of these methods I'd be curious to see the code you are using and to know the OS and version you are running it under.

    Avdi

    October 27, 2009 at 12:19 pm

  4. Avdi, could you point me to the gems you are going to be covering in the next installment of this series? I'm working on a gem implements some of this but I don't want to duplicate efforts unnecessarily. Thanks.

    Ben Mabey

    November 4, 2009 at 6:29 pm

  5. Ben, I will definitely be covering Ara T. Howard's Open4 gem, and Tim Pease' Servolux[1]. If I have time I may hit some others, but I see those two as the most important/interesting third-party Ruby subprocess libraries out there right now.

    [1] http://github.com/TwP/servolux

    Avdi

    November 6, 2009 at 3:05 pm

  6. […] A Dozen (or so) Ways to Start Subprocesses in Ruby: Part 3 – The Devver Blog […]

  7. I'm curious why none of these methods support manually defining the environment of the subprocess. Java's Runtime.exec() method, for example, takes an array of name=value strings as the variables to set in its subprocess' environment. Is there another way to do that in ruby?

    John Didion

    November 25, 2009 at 5:46 am

  8. Answered my own question. A few ways to do it here: http://www.ruby-forum.com/topic/149404. Ara's systemu seems like the best way to go.

    John Didion

    November 25, 2009 at 6:11 am

  9. Excellent question, and something I've been wondering about myself. Thanks for the link, it looks like systemu may deserve a place in my next article.

    Avdi

    November 25, 2009 at 7:33 am

  10. […] A dozen (or so) ways to start sub-processes in Ruby: Part 1 (Part 2, Part 3) […]


Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: