Last entry in series about stable and faster test suite. Previous posts were focusing on parallel execution when running the test suite on CI nodes. This time we will see how we can run it locally as well. You can find previous articles here

• How to: Road to fast and stable test suite
• How to: Get most of the database cleaner
• How to: Execute RSpec in parallel locally (You are here)

ParallelTests gem

So far we used Knapsack gem for dividing our tests in order to evenly run them on CI, but we can’t really use it for running them all in parallel locally. This is where ParallelTests gem will come in handy.

ParallelTests splits tests into even groups (by the number of lines or runtime) and runs each group in a single process with its own database.

So in short, each process spawned by ParallelTests should have its resources isolated from one another, so they don’t interfere with each other.

Basic setup

Add parallel tests to development and test groups in Gemfile.

gem 'parallel_tests', group: %i[development test]

Next, let us decide on the number of processors You want to use. By default parallel_tests will set it to number of CPUs available (i.e. 4 cores with hyperthreading, will count as 8 cores). If You would like to override it, append [<no of processors to run>] to tasks below or better, export environment variable to have it the same for all the tasks, i.e.

export PARALLEL_TEST_PROCESSORS=8

Just remember to put it in something like .bashrc or .env file, in order to have it always loaded between sessions.

Resources configuration

1. Database

I assume You use a database, otherwise, you wouldn’t have issues with slow running tests in the first place. For this to work, we need to update our config/database.yml with an extended database name.

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

This will append the processor number to your database name, whenever you will run tests with more than 1 process, i.e.

1. First process yourproject_test
2. Second process yourproject_test1
3. etc

Now let’s create all databases.

rake parallel:create

And load the schema.rb or structure.sql to all DBs created above.

rake parallel:prepare

2. Capybara

Capybara servers should run on separate ports, i.e.

Capybara.configure do |config|
  config.server_port = 9887 + ENV['TEST_ENV_NUMBER'].to_i
end

3. Other resources

You should do the same for any other kind of resource You use, those should be completely separated from each other, i.e.:

• files, i.e. for rails/sprockets cache should lay in different directories
• redis, use different DBs per process
• sphinx, run a separate instance per each processor
• etc

Checkout extensive wiki for details.

Lets run this!

rake parallel:spec
# => 8 processes for 500 specs, ~ 62 specs per process

Awesome! Now we can run our test suite locally as fast as on CI.

Speedup process boot

Now when spawning all those processes, each of them will take some time, based on the size of your application. We can speed this up with spring, which comes by default with rails installations these days. To make it work we will have to do a small patch for it, otherwise it won’t work with parallel_tests.

This is due to the fact that with spring, when it boot up the server process, the configuration will be already set. This means DB name will always equal to yourproject_test in each spawned process based on the server one.

In order to mitigate this, we will pick up the correct DB configuration after forking the server process. You can find it in parallel_tests Wiki.

Create new file under config/spring.rb - it should get picked up by spring automatically.

require 'spring/application'

class Spring::Application
  alias connect_database_orig connect_database

  # Disconnect & reconfigure to pickup DB name with
  # TEST_ENV_NUMBER suffix
  def connect_database
    disconnect_database
    reconfigure_database
    connect_database_orig
  end

  # Here we simply replace existing AR from main spring process
  def reconfigure_database
    if active_record_configured?
      ActiveRecord::Base.configurations =
        Rails.application.config.database_configuration
    end
  end
end

Then you can either prepend DISABLE_SPRING=0 to commands, export it, or put in .bashrc/.env file.

With this, bin/rake parallel:spec will boot up way faster by making use of spring preloader. You should notice this by extra lines in output like

Running via Spring preloader in process 20005
Running via Spring preloader in process 20012
Running via Spring preloader in process 20015
... etc

Tests distribution

Another thing to consider is, how to evenly distribute tests across processes. parallel_tests has similar functionality as knapsack, it can log tests runtime in a JSON file and then use it to spread them evenly across all processes. To do so add .rspec_parallel file in project root directory, so on next run it will create the report, and use it in consecutive executions.

--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log

NOTE: Remember to put any significat config from .rspec file to .rspec_parallel, i.e. --require spec_helper - as parallel tests will use the later only, what can lead to issues with tests.

Now, this is good for having local runtime as low as possible, but what if we would like to use the knapsack report, which we already have? Sadly parallel_tests doesn’t have any integration for it, but we can play around and add it by ourselves - because we can :)

Let’s create a wrapper task we will use to run it

# lib/tasks/knapsack.rake
namespace :knapsack do
  task local: :environment do |_, _|
    ENV['CI_NODE_TOTAL'] = ENV['PARALLEL_TEST_PROCESSORS']
    ENV['RAILS_ENV'] = 'test'

    require 'knapsack'
    require_relative '../../config/boot'
    require_relative 'parallel_tests_patch'

    ParallelTests::CLI.new.run(['--type', 'rspec', 'spec'])
  end
end

Now a small monkey patch to use knapsack allocator in parallel tests

# lib/tasks/parallel_tests_patch.rb
require 'parallel_tests/rspec/runner'

class ParallelTests::RSpec::Runner
  def self.tests_in_groups(tests, num_groups, options = {})
    puts 'ParallelTests with Knapsack runtime report :woohoo:'

    (0...num_groups).map do |index|
      ENV['CI_NODE_INDEX'] = index.to_s
      Knapsack::AllocatorBuilder
        .new(Knapsack::Adapters::RSpecAdapter)
        .allocator
        .node_tests
    end
  end
end

Let’s test it!

rake knapsack:local
# ParallelTests with Knapsack runtime report :woohoo:
# 8 processes for 500 specs, ~ 62 specs per process

NOTE For local execution I would still use parallel_tests allocator instead, as it will be generated based on our machine performance, whereas knapsack is supposed to be based on the CI node.

Runtime comparison

As you can see above, our suite runtime without any parallelization takes quite a while, around ~42 minutes.

When we parallelize it with 8 processes result differs based on extra switches.

Without the runtime log to evenly distribute tests, take the longest (even with spring support).

We can also see that having more than 8 processes is also degrading runtime.

In our case the best results are achieved when:

• running against 8 processes on 8 available cores
• run together with runtime logs for tests distribution (-4'30")
• processes are preloaded by spring (-30")

This can be due to the fact that we have a lot of IO in tests, in tests and appications that do heavy computing, less processes can actually give better results. As usuall measure, compare and take the most performant option ;-)

Summary and what’s next

From now on it should be easy to run Your test suite in parallel, both locally and on CI with help of knapsack and parallel_tests gems.

The test suite itself should be also faster and more stable thanks to better usage of DatabaseCleaner.

Thankfully Rails 6 should bring us built-in support for

• running tests in parallel
• better transactions handling
• and multi-database connection support

So we won’t have to hack our way through in new apps, for old ones running on Rails <= 5 we are already covered.

Stay tuned and happy hacking!