Making RSpec Tests More Robust

RSpec is a popular framework for testing Ruby code. With an expect assertion, a developer can make sure their code calls the proper method or an acceptable result is returned. The expect().to receive matcher in a test overrides the default implementation and can cause some unintended side effects.

To demonstrate this potential problem, assume a very simple API client exists that can update models.

Testing an API Client

An example api_client.rb class defines a single put method that calls the underlying API with a Faraday connection.

# api_client.rb

class APIClient
  def put(url, body)
    client.put(url, body)
  end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

Inheriting from api_client.rb is my_model.rb which defines an update method.

# my_model.rb

class MyModel < APIClient
  def update(payload)
    put('/my_models/1', payload)
  end
end

A typical test for the update method on MyModel in RSpec might look like:

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      expect(subject).to receive(:put)
      subject.update({ foo: :bar })
    end
  end
end

The subject in the above test is MyModel.new and is expected to receive the method put. Since MyModel#update calls the put method, this seems like a reasonable test.

$: rspec my_model_spec.rb
.

Finished in 0.0054 seconds (files took 0.07101 seconds to load)
1 example, 0 failures

Running the test produces a passing result, the MyModel class calls its parent method correctly and all is well. Until something changes that the test is unable to detect.

Breaking the Contract

If MyModel’s contract changes, the test should fail. If, for instance, the put method on the APIClient class is removed or commented out, the update method on MyModel would no longer work.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

However, the test still passes despite this method being removed.

$: rspec my_model_spec.rb
.

Finished in 0.00523 seconds (files took 0.0699 seconds to load)
1 example, 0 failures

So what is going on here?

The culprit is this line: expect(subject).to receive(:put). The test happily accepts the call to APIClient#put despite it being removed. This is obviously problematic and, in a worst case scenario, could lead to an outage if a test suite is the only gateway to production code.

Starting an irb repl and calling the problematic method reveals that the code is no longer working despite the test’s result.

2.5.5 :001 > model = MyModel.new
 => #<MyModel:0x00005585f950ca10>
2.5.5 :002 > model.update(foo: :bar)
Traceback (most recent call last):
        3: from /home/yez/.rvm/rubies/ruby-2.5.5/bin/irb:11:in `<main>'
        2: from (irb):2
        1: from /tmp/foo.rb:8:in `update'
NoMethodError (undefined method `put' for #<MyModel:0x00005585f950ca10>)

Since this code communicates to an API, validating that the underlying put request is done successfully can be tricky. Unlike tests in a Rails application, MyModel#update can not be validated in a local database.

With a passing test and broken code, it is clear that wrong thing has been tested. As is, the test only asserts that a method calls another method and happily ignores everything else. One possible solution could be to add and_call_original to the end of the expect(). to receive line. This would make sure the put method is really there but would also make a real HTTP request in a test (which is generally bad practice).

An easy way to make this test more robust without going overboard is to stub the response of the HTTP client. Instead of mocking the put method in api_client.rb, tools like webmock can be utilized to stub the Faraday response. This enables the code to fake an HTTP request instead of of calling out to https://some-cool-api.com.

Testing with Webmock

The webmock gem allows developers to define custom responses for specific HTTP requests.

After installing the gem with gem install webmock, a stub can be written that matches the URL and request body to return a specific response.

WebMock.stub_request(:put, 'https://some-cool-api.com/my_models/1').
  with(body: /foo.*bar$/).
  to_return(body: '{ "success": true }')

In this case the stubbed response is { "success": true } and will be returned for any put requests to https://some-cool-api.com/my_models/1 with a request body matching foo followed by bar. This stub can be added in a shared file or at the beginning of my_model_spec.rb.

With the stub added, my_model_spec.rb can be updated to remove the old expect().to receive line and instead validate the response.

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      response = subject.update({ foo: :bar })
      expect(JSON.parse(response.body)['success']).to eq(true)
    end
  end
end

The stub works and running my_model_spec.rb shows the test passes.

$: rspec my_model_spec.rb
.

Finished in 0.00531 seconds (files took 0.33028 seconds to load)
1 example, 0 failures

Now, if APIClient is changed in the same way as before, the test will appropriately fail and alert the developer that the put method does not exist.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

$: rspec my_model_spec.rb
F

Failures:

  1) MyModel update updates the model
     Failure/Error: put('/my_models/1', body)

     NoMethodError:
       undefined method `put' for #<MyModel:0x000055f148a80018>
       Did you mean?  puts
                      putc
     # ./my_model_spec.rb:17:in `update'
     # ./my_model_spec.rb:30:in `block (3 levels) in <top (required)>'

Finished in 0.00298 seconds (files took 0.4606 seconds to load)
1 example, 1 failure

Stricter Settings

RSpec allows developers to enable the flag verify_partial_doubles which will cause the original test to fail when the put method is removed. This setting is turned off by default.

RSpec.configure do |config|
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end
end

With this setting in place, running the original code and test produces a failing result.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      expect(subject).to receive(:put)
      subject.update({ foo: :bar })
    end
  end
end

$: rspec my_model_spec.rb
F

Failures:

  1) MyModel update updates the model
     Failure/Error: expect(subject).to receive(:put)
       #<MyModel:0x000055d4a4f740a8> does not implement: put
     # ./my_model_spec.rb:35:in `block (3 levels) in <top (required)>'

Finished in 0.00877 seconds (files took 0.46471 seconds to load)
1 example, 1 failure

This alternative does not require the use of a new gem and will fail correctly if the underlying contract is removed. However, if the put method or any layer between it and the API request changes, this test is not guaranteed to save a developer from that issue.

Determining what layer to mock and what to explicitly test is a very case by case basis.

Keep Tests Robust

Mocks and stubs in RSpec allow developers to make important assertions about their code. Unfortunately, mocking can also cause false positives when modifying real code.

Validating that the proper settings are in an application is a great first step towards more a more robust test suite.

Stubbing the response for the underlying API in this example enables a much more robust test that can alert developers when they make breaking changes. However, if the API response from https://some-cool-api.com changes, the webmock stub must be updated.

Stubbing and mocking tools should be used on a case by case basis. In the case of testing a third party API integration, providing canned responses in the test suite makes sense. In other “normal” Ruby or Rails tests, verifying doubles or other built in RSpec matchers could be all that’s needed.