The How and Why of Property-Based Testing in Ruby

Think about the last time you wrote unit tests (which hopefully is pretty recent). You had to come up with the happy-path, the tragic-path, and those hard-to-find edge cases. Since we are not infallible human beings, we tend to miss things more often than not. We miss an edge case here, forget about handling an error there, and discover our omissions during production.

Property-based testing totally flips the notion of unit testing on its head. Instead of writing specific examples of a test, you would write a property instead. A property, in this context, signifies a condition that would hold for all inputs that you define.

An example here is certainly helpful, if not required. Say you want to test a function that reverses an array. What could be some useful properties? Well, I can think of the following:

• The first element of the array becomes the last element
• The length of the array remains the same before and after reversing
• Reversing the array twice will give back the original array

You could probably come up with a few more. With a property-based testing tool, you could express the above as code and tell it to generate hundreds or even thousands of test cases with randomly generated arrays.

A cool feature of many property-based testing tools is shrinking. This means that the tool, to the best of its ability, will find the smallest input that causes the property to be unsatisfied.

This Sounds Familiar…

If the above description sounds familiar, then you might have heard of a testing tool called QuickCheck. QuickCheck was originally written in Haskell, but has found its way into languages like Erlang. There are other QuickCheck implementations, like ScalaCheck for Scala and FsCheck for F#.

While the above languages are mainly functional languages, this hasn’t stopped the creators of Rantly from coming up with a QuickCheck-like tool for Ruby along with a cute name.

In this article, we will expand our testing repertoire by trying out property-based testing in Ruby.

Setting Up Rantly with RSpec

We will first set up a demo project to use Rantly with RSpec. The instructions are similar for MiniTest and TestUnit. Let’s create a demo project:

% mkdir rantly-demo && cd rantly-demo

Then create a skeleton Gemfile:

% bundle init
Writing new rantly-demo/Gemfile

We only need rantly:

source "https://rubygems.org"

gem "rantly"

Time to install the dependencies:

% bundle
Fetching gem metadata from https://rubygems.org/..
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using rantly 0.3.2
Using bundler 1.10.6

Assuming you already have rspec installed, you can quickly set up your project to use RSpec:

rspec --init

Using Rantly

Here is how you would express a Rantly property:

it "define your property here" do property_of { Rantly { <GENERATOR GOES HERE> } }.check { |a_generated_value| <EXPECTATION GOES HERE> } end

Therefore, an example would be:

it "integer property only returns Integer type" do property_of { integer # the generator }.check { |i| # i is the generated value expect(i).to be_a(Integer) # the expectation } end

A Failing Test and Shrinking

Let’s see what Rantly tells us when a property fails. We will tell Rantly to create arrays of integers, and then check that every generated array has completely even elements. Obviously, that will fail. The interesting thing is how will it fail? Let’s write the shady property in spec/array_spec.rb :

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

RSpec.describe "Array" do
  it "even numbers" do
    property_of {
      Rantly { array { integer } }
    }.check { |i|
      expect(i).to all(be_even)
    }
  end
end

When you run the file, this is the output:

[0, 0, 0, 0, -1324248444, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -10102, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -77, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -1, -819907805037675589]
found a reduced failure case:
...
minimal failed data is:
[0, 0, 0, 0, 0, -819907805037675589]
F

Failures:

  1) Array even numbers
     Failure/Error: expect(i).to all(be_even)
       expected [-1384706466568309853, -2143298094606122148, 2181188094126790798, 1908884087348911076, -710950470620772656, -819907805037675589] to all be even

          object at index 0 failed to match:
             expected `-1384706466568309853.even?` to return true, got false

          object at index 5 failed to match:
             expected `-819907805037675589.even?` to return true, got false

Here, we can see that Rantly tries to find the smallest failure case by reducing the number of elements to as small as possible. This process is called shrinking. Rantly is able to perform shrinking on integers, strings, arrays, and hashes.

Unfortunately it didn’t decrease the last element. However, it is relatively obvious (after some squinting) that -819907805037675589 is a negative number.

Coming Up with Properties

By far the hardest task when doing property-based testing is coming up with the properties in the first place. In this section, we will cover some helpful techniques to figure out what kinds of properties to write. This list is not exhaustive, but serves as a good starting point.

1. Inverse Functions

These are usually pretty obvious to spot. Examples of inverse functions include:

• Encoding and Decoding (e.g. Base64)
• Serializing and Unserializing (e.g. JSON)
• Adding and Removing

While writing a property that exploits “inverse-ness” doesn’t really cover a lot, this property is extremely useful as a sanity check. Couple this with one hundred or more generated test cases and you should feel pretty confident.

Here’s an example of how to test Base 64 encoding and decoding. Create base64_spec.rb in spec:

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

require 'base64'

RSpec.describe "Base64" do
  it "encoding and decoding are inverses of each other" do
    property_of {
      Rantly { sized(30) { string } }
    }.check(1000) { |s|
      puts s
      expect(Base64.decode64(Base64.encode64(s))).to eq(s)
    }
  end
end

This basically creates random 30-character strings and generates 1000 tests, asserting that encoding and decoding are indeed inverses of each other. Here’s a sampling:

`` ... I.F@5!x}PI({m[8XPw=r1Vep(\*uIi Cz)ZkAcUUE],xoOI/@g*&;I):JVn$ J\Oo”PTR-8[A3);k*5Li0+v;[e8o= R{IL2Vz]$.KcOG<uy<gBpPc}T|+j7n .gsw?:?#"Iy%0O>-V0!]y#K}6>M!Ny xnkLFCLeim)VR9r|qaZuoYrNWd1GOU ?~m|6;;N~w.b)_+L-mIx-X?Lkb55Z7C”_Ps Kb[Or?V’ypYI!kwk>xB>:s-!D?GxS !lb%bw>u0V6xHwh9+D6c$/Q0!UkM\Y

success: 1000 tests . “`

2. Idempotence

Idempotent – A word to impress your friends and annoy your co-workers! This basically means doing it once is the same as doing it multiple times. For example, calling Array#uniq multiple times will result in the same value. Sorting functions also belong to the same category.

Let’s try out Array#uniq (you can put this in spec/uniq.rb):

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

RSpec.describe "Array" do
  it "uniq is idempotent" do
    property_of {
      Rantly { array { Rantly { i = integer; guard i >= 0; i } } }
    }.check { |a|
      expect(a.uniq.uniq).to eq(a.uniq)
    }
  end
end

Here, we are generating an array of non-negative integers. We are using generator guard to limit the generated integers to only non-negative ones:

Rantly { i = integer; guard i >= 0; i }

3. Using an existing implementation

Say you have discovered a new sorting algorithm called, QuickerSort, that you know is faster than the existing sorting algorithm implemented in Ruby. Now all you need to do is to make sure that your QuickerSort implementation produces the exact same results as the Ruby implementation.

With QuickCheck, we can easily express a property like so:

RSpec.describe "Array" do
  it "Array#quicker_sort works produces the same result as Array#sort" do
    property_of {
      Rantly { array(range(0, 100)) { integer }}
    }.check { |a|
      expect(a.quicker_sort).to eq(a.sort)
    }
  end
end

Here we are generating an array of random integers that can be an empty array all the way to an array of 100 elements.

Custom Generators: Generating a DNA Sequence

Let’s learn how to create a custom generator. Custom generators are useful when your input data has to fit certain requirements. For example, if a method only operates on binary digits, then using the default integer generator will not be very useful.

In this example, we will create a DNA sequence generator. For our purposes, a DNA sequence is basically a array that contains a combination of A, T, G and C.

An example would be ["C", "G", "A", "G", "A", "T", "G"]. Our first stop is Rantly#choose, which let’s the generator pick a value from the specified choices:

choose("A", "T", "G", "C")

Next, we know that we need an array. The array generator accepts a block, which is called to generate an element of the array. This is exactly what we need:

Rantly { array { choose("A", "T", "G", "C") } }

To add some variation, we can also have the generator produce various length arrays by specifying a range:

Rantly { array(range(0, 20)) { choose("A", "T", "G", "C") } }

Try this out on a console. You would need to do a require "rantly":

> 10.times { p Rantly { array(range(1,20)) { choose("A", "T", "G", "C") } } }
["T", "A", "A", "G", "A", "A", "T", "G", "G", "T", "A", "T", "T", "T"]
["T", "A", "T", "T", "G"]
["T", "T", "C", "G", "T", "T", "C", "A"]
["T", "C", "C"]
["G", "G", "T", "C"]
["A", "A", "A", "A", "C", "G", "T", "G", "G", "T"]
["C", "A", "G"]
["T", "G", "C", "C", "A", "C", "C", "T", "G", "C", "T", "C", "G", "C"]
["G", "C", "T", "T", "T", "A", "C", "A"]
["G", "G", "G", "A", "C", "T", "G", "C"]
 => 10

Pretty cool, eh?

Summary

Property-based testing proposes another way to think about tests. Instead of writing specific examples, why not write generic properties and let the tool generate test cases for you?

However, don’t go throwing away your unit-tests yet! Property-based testing would probably be most useful for testing things like data-structures, stateless functions (that is, functions in the functional programming sense of the word), and algorithms. It probably isn’t a great fit for testing business logic. In other words, Rantly is a new tool in your belt, not the entire belt.

Try out Rantly and let me know how it goes. Happy testing!

Frequently Asked Questions (FAQs) on Property-Based Testing in Ruby