Capturing stdout & stderr from shell commands via Ruby

tl;dr If you want to run a shell command from Ruby and capture its stdout, stderr and return status, check out the Open3.capture3 method. If you'd like to process stdout and stderr data in a streaming fashion, check out Open3.popen3.

So many bad choices

There are literally 492 ways to execute shell commands from ruby and each of them works slightly differently. I bet you've used one of the approaches below. My go-to has always been the back-ticks (``).

exec("echo 'hello world'") # exits from ruby, then runs the command
system('echo', 'hello world') # returns the status code
sh('echo', 'hello world') # returns the status code
`echo "hello world"` # returns stdout
%x[echo 'hello world'] # returns stdout

But these approaches are pretty limited. Suppose that you need to capture not only your shell command's stdout, but also its stderr. You're just plain out of luck. Or suppose you'd like to process stdout data in a stream, and not all at once when the command finishes running? Out of luck.

There is another option. One that gives you the ability to run commands asynchronously, and which gives you stdout, stderr, exit codes, and PIDs. Let's check it out!

Open3

The oddly-named open3 module is part of Ruby's standard library. What does it do?

Open3 grants you access to stdout, stderr, exit codes and a thread to wait for the child process when running another program. You can specify various attributes, redirections, current directory, etc., of the program in the same way as for Process.spawn. (_Source: [Open3 Docs](http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html))_

Never used it? Never even heard of it? I'm guessing that's because it doesn't come off as the most friendly of libraries. The name itself sounds more like C than Ruby. And the documentation is pretty hard-core neck-beard. But once you give it a try, you'll find that it's not as intimidating as it sounds.

capture3

What if there were an easy way to capture stdout, stderr AND the status code? Well there is. If you don't have time to read the rest of this article, just know that you can use a method called capture3 and call it a day.

Let's take a look at an example. Suppose you want to get a list of files in your current directory. To do that you can run the ls command.

If you were to use the back-tick syntax it's look like this:

puts(`ls`)

With capture3 it looks like so:

require 'open3'
stdout, stderr, status = Open3.capture3("ls")

This will run your command and give you stdout and stderr as strings. No muss no fuss.

Security

You generally don't want to give your users the ability to run arbitrary commands on your web server. That's why code like identify #{ params[:filename] } is such a horrible idea.

Open3 lets you avoid problems like this by separating commands from data. It works just like the system method.

Open3.capture3("identify", params[:filename], other_unsafe_params)

popen3

Under the hood, capture3 uses a much more powerful method called popen3. This method works a little differently than more familiar methods like system().

Here's what it looks like:

require 'open3'
Open3.popen3("ls") do |stdout, stderr, status, thread|
  puts stdout.read
end

It's kind of like when you open and read from a file.  I'm sure you've seen code like this:

File.open("my/file/path", "r") do |f|
  puts f.read
end

Pipes

With Open3, stdout and stderr are all pipes, which behave a lot like file buffers. And like files, they need to be closed when you're done with them. That's the reason for the block syntax. (There's a non-block syntax, but you have to manually call close on stdout,and stderr.)

The read method waits until the pipes are closed before returning a value. But pipes also support reading lines as they become available. Imagine your shell command takes a few seconds to run. During that time, it's printing a status message to stderr. You'd like to capture that and display it to your users.

Here's how you'd capture stderr a line at a time.

require 'open3'
Open3.popen3("sleep 2; ls") do |stdout, stderr, status, thread|
  while line=stderr.gets do 
    puts(line) 
  end
end

Threads

There's one argument we haven't talked about yet. That's thread.

The thread argument gives you a reference to a ruby thread that's waiting on your command to finish. Now, the command isn't running in the thread. It's running in an entirely separate process. The thread just watches the process and waits until it's done.

You can get some useful data from that thread reference though.

• thread.pid - contains the process id of your shell command. You would need this if you wanted to do additional OS-level operations against that process.

• thread.status - contains the exit status of the process. 1 or 0 for success or failure.

Caveats

From the Open3 docs:

You should be careful to avoid deadlocks. Since pipes are fixed length buffers,[::popen3](http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-popen3)(“prog”) {|i, o, e, t| o.read } deadlocks if the program generates too much output on stderr. You should read stdout and stderr simultaneously (using threads or IO.select). However, if you don’t need stderr output, you can use [::popen2](http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-popen2). If merged stdout and stderr output is not a problem, you can use [::popen2e](http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-popen2e). If you really need stdout and stderr output as separate strings, you can consider [::capture3](http://ruby-doc.org/stdlib-2.1.0/libdoc/open3/rdoc/Open3.html#method-c-capture3).