How to Test Ruby Projects on Windows

Have you noticed that Ruby developers are overwhelmingly Linux and Mac users? Because of this collective blindspot, sometimes gems produced by the Ruby community fail to work properly on that other big platform: Windows. If you’re authoring a gem, setting up Windows-based continuous integration can help guard against this oversight.

AppVeyor, like Travis CI, is free for open source projects. Being Windows-based, AppVeyor’s audience is primarily .NET developers, and so Ruby documentation and examples seem to be hard to find. But running Ruby on AppVeyor is possible with these steps:

1. Set up a CI project

Visit appveyor.com and log in via GitHub. Once connected, create a project for the GitHub repository you wish to test. AppVeyor will configure webhooks to trigger a build on your next commit.

2. Create appveyor.yml

AppVeyor uses an appveyor.yml file at the root of a project to configure the build process (the concept is the same as .travis.yml, but note there isn’t a leading .).

Through some trial and error, I’ve figured out the basic appveyor.yml that works for building a Ruby gem project:

version: '{build}'

skip_tags: true

environment:
  matrix:
    - ruby_version: "21"
    - ruby_version: "21-x64"

install:
  - SET PATH=C:\Ruby%ruby_version%\bin;%PATH%
  - gem install bundler --no-document -v 1.10.5
  - bundle install --retry=3

test_script:
  - bundle exec rake

build: off

Most of this is self-explanatory, but there are a few options that are a bit tricky:

ruby_version

Note that Ruby 2.1.x is specified as "21" and "21-x64" for the 64-bit version. Ruby 2.2.x is also available on AppVeyor, but in my experience it does not work reliably.

test_script

These are the commands to actually execute the test suite. Typically for Ruby projects this is bundle exec rake.

build

Out of the box, AppVeyor seems to assume that you will be building a .NET project, and will fail if you are testing something different. For Ruby, disable this behavior with build: off.

3. Commit and troubleshoot

When you commit appveyor.yml and push it to GitHub, this will automatically trigger a build. Keep in mind that AppVeyor builds in their free open source tier are often queued for several minutes before they run. Be patient!

With luck, your first build will execute without errors. Mine, unfortunately, did not. But fixing things was pretty straightforward. Here are some of the issues I encountered:

• On AppVeyor, $stdout.tty? returns false, whereas on Travis it is true. This may cause trouble if you are testing code that behaves differently based on tty?.
• There is no such thing as > /dev/null on Windows, so running a command while muting its output using system("some command > /dev/null") won’t work. One alternative is `some command`, which captures the output; then testing the result with $?.success?.
• Etc.getpwuid returns nil on Windows. I’m guessing the entire Etc module probably doesn’t work.
• On Windows, the block form of Dir.mktmpdir (which is supposed to clean up after itself) failed for me with mysterious permission denied errors. I had better luck using the non-block form and cleaning up manually using FileUtils.rm_rf.
• For some reason the gem install bundler step tends to get stuck during AppVeyor builds, sometimes for several minutes. I couldn’t find a solution except to wait patiently for it to get un-stuck.

It works!

I’m happy to report that two of my open source projects, airbrussh and chandler, both have successful Windows builds. Even though I don’t use Windows myself, thanks to the continuous integration I now have some peace of mind that I am not putting Windows-incompatible code out into the world.

By the way, the Travis team also has Windows support on its radar, so eventually we may have a single tool that can support all the platforms we need. Until then AppVeyor is a good option and doesn’t take too much work to set up.

Have you used AppVeyor to build a Ruby project? How did it go?