The Optimist's Guide to Pessimistic Library Versioning

Upgrading software is much harder than it could be. Modern versioning schemes and package managers have the ability to help us upgrade much more than they do today.

Take for example my post on Upgrading to Rails 5 Beta -- The Hard Way. Most of the time was spent trying to find all the different libraries my app was using that weren't compatible yet with Rails 5 and upgrade them.

What if somehow bundle update rails could have known they didn't work and instead find a version that did work? It turns out that it can, but it requires some pain on the maintainer's part.

In this post, we'll look at different strategies for declaring dependencies in libraries, why one is dominant right now, and how we might be able to make major version bumps easier into the future.

In Ruby, when you create a library to be shared, you package it up using a gemspec. This is a file that describes your library enough to let other systems know how to use it. It includes info such as the files needed to run when you gem install so you don't have to download unnecessary files like tests and fixtures. It also allows a library to declare dependencies.

If your library explicitly require-s and uses another library, you must put it in your gemspec:

spec.add_runtime_dependency 'get_process_mem', ['~> 0.2']

Now we can use that dependency inside of the library:

require 'get_process_mem'
puts GetProcessMem.new.mb

I'll call this a direct dependency. In addition to dependencies that you must require, libraries can be an indirect dependency.

For example, Wicked is a gem that helps to make step-by-step wizard controllers in a Rails app. Even though Wicked does not require Rails, it depends on certain Rails API to be present, so it makes sense to declare Rails as a dependency.

When I wrote Wicked, Rails was at version 3.0.7, so I knew it worked with that version.

gem.add_runtime_dependency "railties", [">= 3.0.7"]

The rails gem will pull in Active Record, which some people don't use. Since Wicked does not rely on Active Record, we limit the dependencies to "railties."

The benefit here is that if you're using Rails 3.0.6 and you try to install Wicked, Bundler won't let you. It'll give you instant feedback that it is only designed for versions of Rails equal to or above 3.0.7.

You may have noticed earlier that we used different operators for our different cases. We used the pessimistic operator ("eating bacon") ~> and we used greater-than-or-equal-to >= in our other case. These are the two most common types of versioning strategies.

The pessimistic operator ~> 0.2 is telling us that we need version 0.2 or greater. However, we aren't compatible with a major version bump, so it wouldn't work with anything over version 0.x; it will fail 1.0, 2.0, 3.0, etc. You can read more about Ruby's Pessimistic Operator here.

This is a good practice. If the library is following anything like SemVer for Library Maintainers, then a major version bump would indicate that some backwards incompatibilities have been introduced and that the library we are writing might not be compatible. To know for sure, we need to test.

For Wicked, I used a >= dependency on "railties". This is really easy for me, the maintainer -- when a new version of Rails comes out, I don't have to release a new version of my gem. If it works, it works. If it doesn't work, someone reports a problem, and I release a new gem.

The >= versioning strategy requires the least amount of effort from a maintainer. Since maintaining a gem is already really hard, it's what most people use. The downside is that if you're using version 0.0.1 of Wicked with a Rails 5 app, it won't work. However, you'll still be allowed to install it and try it. This will waste the end user's time.

Unfortunately, you can't retroactively add dependency versions to gems. We can't go back in time and say, "Hey, Wicked 0.0.1 shouldn't be used with Rails 5." Released library versions are immutable; if you changed part of your library including the dependencies, it would no longer be the same version.

We could yank gems, but this is all around awful. It should pretty much never be done unless the gem poses an active threat, such as an accidental shell out to rm -rf for more info; see the There is no going back section on my "SemVer for Library Maintainers" post.

Using the pessimistic operator is good for end users. Let's pretend we have a fictional gem called schneems, like before this gem declares a dependency on get_process_mem.

spec.add_runtime_dependency 'get_process_mem', ['~> 0.2']

If an app owner has gem "schneems" locked in their Gemfile when they try to upgrade to version 1.0 of get_process_mem, then $ bundle update get_process_mem will tell them that there is a dependency mismatch and the schneems gem isn't valid.

$ bundle update get_process_mem
Bundler could not find compatible versions for gem "get_process_mem":
  In Gemfile:
    get_process_mem (= 1.0)
    schneems was resolved to 0.0.1, which depends on
      get_process_mem (~> 0.2)

Now we know we need to update that dependency as well:

$ bundle update get_process_mem schneems
Resolving dependencies...
Using get_process_mem 1.0 (was 0.2.0)
Using bundler 1.11.2
Using schneems 0.0.6 (was 0.0.1)
Bundle updated!

If all gem maintainers used the pessimistic operator on railties, then, when you tried to upgrade to Rails 5, the bundler would tell you what versions you are currently using that aren't valid out of the box. That alone would have saved me a few hours of randomly running commands, looking at backtraces, opening up gem sources, only to discover I was using an outdated and unsupported version.

In short, the pessimistic operator is preemptively trying to do version damage control. Since we can't go back in time and tell our gem what versions it works with, we are making an educated guess that it won't work if a major version is bumped.

Burnout is alive and well in software, especially in open-source gem maintainer-ship where people are working for free in their spare time. Pessimistic locking adds to the pain of maintaining a gem.

What kind of pain? If a gem is declared as a dependency and rev-s the major version frequently, then every time they release a gem, you have to release a version of your gem. It may only be a few commands to rev your version and release, but it takes time.

It's also a pain in that every time that new version comes, you have to figure out a way to test it. You can't simply rev a version every time they do; you must verify your gem works with the new version. This might be as simple as editing a travis.yml file, or it may be complicated and involved, generating new test fixtures and writing new test cases.

If you haven't already, you can add bundler gem release tasks to make releasing new gem versions even easier.

The ~> operator might be pessimistic, but the >= should be renamed the "screw it, you deal with versioning problems" operator. A little bit of effort on the maintainer can help save lots of user time when it comes to upgrades.

One other problem with using the pessimistic operator in a library is that it limits you to not using betas.

For example, if you're using ~> 5.0, then you cannot run version 5.0.0.beta1 because that's considered greater-less-than 5.0. Instead, you use "x" as a placeholder; so you would use ~> 5.x. Now you're allowed to try betas and release candidates with your library.

Right now, most libraries don't lock to an "x" version. It's hardly any wonder that so few libraries release beta versions, and so few people try them.

Similarly, it's not much of a surprise that lots of people get upset when a 1.0.0 or 1.0.1 version they just blindly deployed to production fails because they thought it was stable software. If more libraries switch to using the 5.x versioning instead of 5.0, it would make the process of trying out beta versions easier.

If a major dependency of yours releases a beta, you don't have to wait until they cut a full release to release your gem with support. You can cut a beta release too.

This is a big problem in the Rails ecosystem where lots of gems will bake support in for a major release of Rails but might hide it in a branch like rails5. So not only do they not release a version that's compatible on RubyGems, their controller version might not work either.

The sooner you release a version that works with a major dependency update, the sooner your users can try it out and report back any problems to either your gem or the dependency.

Betas are indicated by a non-numeric character in the last field, like 1.0.0.beta1. Typically you use a "beta" for unstable interface and an "rc" or release candidate for a stable interface. However the format is very open. You could cut a 1.0.0.beta-release-for-rails5-beta2 version if you want.

Users can use your beta version by installing local $ gem install <gemname> --pre or by specifying the version directly in the Gemfile.

Users aren't totally at the mercy of a maintainer's release schedule. These libraries are open source -- they can fork! This is pretty simple; you click the "fork" button on GitHub, edit the gemspec directly on the web using the built-in editor, and then in your Gemfile, point to your new fork:

 gem "gemname", github: "<username>/gemname", banch: "patch1"
 

Forking on GitHub takes only a minute or two and is very easy. When you try to use GitHub's built-in editor, it will automatically make a fork for you in a new branch if you:

Once you've confirmed that it works with your app, you already have a fork ready to submit a PR back to the maintainer:

"Hey, your gem's gemspec says it won't work with Rails 5, but it worked fine. Please accept this pull request and cut a new release."

Forking is easy, and it encourages us to do the right thing by communicating with the maintainer that the version change worked. What are the downsides then? Forking takes us off the managed path.

Now, you must somehow remember why you are using a fork. What if a co-worker adds a feature to your fork and updates the Gemfile.lock without telling anyone? If you try to go back to the Rubygems hosted library, you'll get failures. What if there is a catastrophic rm -rf problem with that gem version that you forked and it's yanked from Rubygems? You won't be helped.

The idea behind the pessimistic operator was to help end users upgrade, but now you've taken yourself out of the upgrade path. bundle update <your-forked-gem> won't save you now.

This isn't as bad as it sounds. If you always remember to contribute your gemspec patches back upstream, there won't be a problem when you try to go back to the official repo. If the maintainer doesn't want to accept your patch, maybe there is somewhere else you could change behavior to get what you want. This might take a little extra time, but it encourages you to do the right thing.

The other pain with forking is that it means that every user of your gem must fork; it pushes the pain to the end user. The way to mitigate this is, if you're forking a gem, submit a PR as soon as you can determine if a simple gemspec update works. That way, when people go to fork the gem, they'll see that controller is already updated and use that instead.

Forking does require a little bit of discipline, but the benefits outweigh the costs. I would trade a few minutes of forking gems for the hours I spent dealing with guessing what gem versions were incompatible with what dependencies.

If you're a gem maintainer, you should switch to the pessimistic version operator for your gemspec dependencies using the "x" syntax like ~> 1.x. Don't be afraid of releasing your own beta versions of your own gems. It may seem like more work for you at first, but you'll have lots more users giving you feedback and submitting dependency patches.

If you're using a gem and you hit a problem where you cannot install due to version constraints, check out their controller branch and see if the gemspec is updated there. If it isn't, you can make a fork by going to the gemspec file and hitting the edit button in the GitHub UI. Help maintainers help you by giving them feedback and patches.

The more gems that you have in your project that follow this versioning pattern, the easier it will be to do upgrades. If you are using a gem that is using "screw it" versioning, open a PR to change it to the pessimistic version operator ~> and point them at this post.

What do you think, Ruby community? Can we all be happier devs with a few more pessimistic operators in our lives?