What are the Code Coverage Metrics for Ruby on Rails?

At FastRuby.io we are constantly looking at code coverage metrics for Ruby on Rails applications. It’s a key indicator for us. We even use that information to decide whether we work on a Rails upgrade project or not opens a new window .

So, I was interested in seeing code coverage metrics for the Ruby on Rails framework opens a new window . I couldn’t find any information about this online, so I decided to generate a few reports for each component.

This is an article about my process and my findings.

Process

In order to calculate code coverage opens a new window , I used SimpleCov opens a new window and analyzed Rails at f22dd39 opens a new window .

I didn’t run the entire test suite from its root directory, I went into each component and ran the test suite for that component.

I found that this was a good idea because each component had its quirks. You can’t just run rake test on each component and expect it to work.

One thing that could be improved in Rails’s documentation: It would be nice to have clear documentation on running the test suite for each component. For example: When you run ActionCable you will need to increase your ulimit and you will need to have Redis running in your environment.

Before running each test suite, I added this snippet at the beginning of the helper file:

require "simplecov"

SimpleCov.command_name "Test: #{rand(1024)}"

SimpleCov.start do
  track_files '{lib}/**/*.rb'
  add_filter "/test/"
end

I had to add the SimpleCov.command_name to make sure that all the test rake tasks are considered and automatically merged by SimpleCov. Without that line I was getting unexpected results when running more than one test rake task (e.g. running rake test and rake test:system – the last process would override the coverage calculation from the first process)

I used Ruby v2.5.7, Node v12.18.3, Rails master, and SimpleCov v0.19.0 to run all my tests.

ActionCable

This one was a little tricky because I had to change my ulimit value. I ran into this issue opens a new window in two different MacBooks:

Error:
ClientTest#test_many_clients:
Errno::EMFILE: Too many open files - socket(2) for "127.0.0.1" port 3099
    /Users/etagwerker/.rvm/gems/ruby-2.5.7/bundler/gems/websocket-client-simple-e161305f1a46/lib/websocket-client-simple/client.rb:20:in `initialize'
    /Users/etagwerker/.rvm/gems/ruby-2.5.7/bundler/gems/websocket-client-simple-e161305f1a46/lib/websocket-client-simple/client.rb:20:in `new'
    /Users/etagwerker/.rvm/gems/ruby-2.5.7/bundler/gems/websocket-client-simple-e161305f1a46/lib/websocket-client-simple/client.rb:20:in `connect'
    /Users/etagwerker/.rvm/gems/ruby-2.5.7/bundler/gems/websocket-client-simple-e161305f1a46/lib/websocket-client-simple/client.rb:8:in `connect'
    /Users/etagwerker/Projects/fastruby/rails/actioncable/test/client_test.rb:113:in `initialize'
    /Users/etagwerker/Projects/fastruby/rails/actioncable/test/client_test.rb:200:in `new'
    /Users/etagwerker/Projects/fastruby/rails/actioncable/test/client_test.rb:200:in `websocket_client'
    /Users/etagwerker/Projects/fastruby/rails/actioncable/test/client_test.rb:244:in `block (2 levels) in test_many_clients'
    /Users/etagwerker/Projects/fastruby/rails/actioncable/test/client_test.rb:204:in `block (2 levels) in concurrently'

By following these steps I managed to solve that problem: https://medium.com/mindful-technology/too-many-open-files-limit-ulimit-on-mac-os-x-add0f1bfddde opens a new window

Also, I had to make sure that Redis was running because one of its tests depends on it.

The average code coverage percentage for ActionCable is 80.85%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionCable opens a new window

ActionMailbox

The average code coverage percentage for ActionMailbox is 91.94%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionMailbox opens a new window

ActionMailer

The average code coverage percentage for ActionMailer is 83.05%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionMailer opens a new window

ActionPack

The average code coverage percentage for ActionPack is 48.67%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionPack opens a new window

ActionText

This one was a little tricky. I noticed there were two test suites:

$ rake -T | grep test
rake test             # Run tests
rake test:system      # Run tests for test:system

I managed to run rake test successfully, but when trying to run rake test:system I encountered an issue: https://gist.github.com/etagwerker/370c4d4f48d777ce22cf704443bd7502 opens a new window

I tried to fix things inside the test/dummy directory by doing this:

cd test/dummy
rake yarn:install
rake assets:precompile

Then I ran into another issue opens a new window . Unfortunately I didn’t manage to run rake test:system so the coverage report was generated with rake test.

The average code coverage percentage for ActionText is 81.33%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionText opens a new window

ActionView

The average code coverage percentage for ActionView is 29.57%. I believe that average coverage for this component is higher than that, but I had a hard time running all the tests.

This one was a little bit tricky because rake test will only run 3 test suites for ActionView: ActionView’s rake test output opens a new window

So initially it was misreporting the code coverage percentage. It was telling me that the coverage percentage for ActionView was 29.57% which is not totally accurate.

I took a closer look at all the tests that are present in the component:

[etagwerker@luft actionview (master)]$ rake -T
rake assets:compile                  # Compile Action View assets
rake assets:verify                   # Verify compiled Action View assets
rake default                         # Default Task
rake test                            # Run all unit tests
rake test:integration:action_pack    # Run tests for action_pack
rake test:integration:active_record  # Run tests for active_record
rake test:template                   # Run tests for template
rake test:ujs                        # Run tests for rails-ujs
rake ujs:server                      # Starts the test server

In order to generate this coverage report, I decided to use rake test:template.

ActionView has tests for its JavaScript code. However, this code coverage report was generated considering only its Ruby code.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActionView opens a new window

ActiveJob

The average code coverage percentage for ActiveJob is 91.42%.

It’s very interesting that the test suite has to test different adapters, so you can see that there are many rake tasks available to test each adapter:

[etagwerker@luft activejob (master)] $ rake -T
rake test:async                      # Run adapter tests for async
rake test:backburner                 # Run adapter tests for backburner
rake test:default                    # Run all adapter tests
rake test:delayed_job                # Run adapter tests for delayed_job
rake test:inline                     # Run adapter tests for inline
rake test:integration                # Run integration tests for all adapters
rake test:integration:async          # Run integration tests for async
rake test:integration:backburner     # Run integration tests for backburner
rake test:integration:delayed_job    # Run integration tests for delayed_job
rake test:integration:inline         # Run integration tests for inline
rake test:integration:que            # Run integration tests for que
rake test:integration:queue_classic  # Run integration tests for queue_classic
rake test:integration:resque         # Run integration tests for resque
rake test:integration:sidekiq        # Run integration tests for sidekiq
rake test:integration:sneakers       # Run integration tests for sneakers
rake test:integration:sucker_punch   # Run integration tests for sucker_punch
rake test:integration:test           # Run integration tests for test
rake test:isolated                   # Run all adapter tests in isolation
rake test:que                        # Run adapter tests for que
rake test:queue_classic              # Run adapter tests for queue_classic
rake test:resque                     # Run adapter tests for resque
rake test:sidekiq                    # Run adapter tests for sidekiq
rake test:sneakers                   # Run adapter tests for sneakers
rake test:sucker_punch               # Run adapter tests for sucker_punch
rake test:test                       # Run adapter tests for test

You can see a test run over here: https://gist.github.com/etagwerker/888e9128b15da5d8d9574cd453a3418a opens a new window

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActiveJob opens a new window

ActiveModel

The average code coverage percentage for ActiveModel is 91.69%.

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActiveModel opens a new window

ActiveRecord

I approximate that the average code coverage percentage for ActiveRecord is higher than 87.23%.

I didn’t get to run all the test rake tasks because I ran into a couple of issues with MySQL and Oracle, so I ended up running only three rake tasks:

bundle exec rake test:postgresql test:sqlite3 test:sqlite3_mem

Here is the passing test suite: https://gist.github.com/etagwerker/f4233a95be711b5bfe37e22a081bfb62 opens a new window

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActiveRecord opens a new window

ActiveStorage

The average code coverage percentage for ActiveStorage is higher than 74.52%.

I had to install mupdf in order to run this test suite. Other than that it was quite straightforward.

While I tried to test everything, I didn’t get to run the tests associated with cloud providers: https://gist.github.com/etagwerker/2d855c108ed2acbcdb35dbd59593f296 opens a new window

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActiveStorage opens a new window

ActiveSupport

The average code coverage percentage for ActiveSupport is 38.02%.

One big caveat: I know for a fact that there is a bug that is misreporting code coverage data for ActiveSupport. I’m still looking into it!

You can find SimpleCov’s detailed report over here: Code Coverage Report for ActiveSupport opens a new window

Summary

Calculating code coverage for a project as big as Ruby on Rails is not trivial. Some components are quite tricky to test (e.g. ActiveRecord and ActiveStorage) because you need a bunch of external services (an Oracle database or a GCS opens a new window account).

When I started writing this article I set out to run the entire test suite in my local environment (Macbook Air), but after hours of trying, I decided to run only a few tests for some of Rails’s components. Hopefully you will find value in knowing how well covered our beloved framework really is.

As you can see, some components have a solid test suite that shows up with great code coverage percentages. Only one or two components could use more tests: Maybe this could be your next OSS contribution? <3

Resources

If you want to see the changes that would be necessary for Rails to generate code coverage reports in Buildkite, you can review this branch: https://github.com/rails/rails/compare/master…fastruby:simplecov opens a new window

There has been some discussion about adding code coverage to the test suite in the past: https://github.com/rails/rails/pull/24148 opens a new window . At the moment, Rails does not calculate code coverage on every pull request.