Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A first naïve look at TruffleRuby #1054

Closed
thbar opened this issue Feb 1, 2018 · 20 comments
Closed

A first naïve look at TruffleRuby #1054

thbar opened this issue Feb 1, 2018 · 20 comments

Comments

@thbar
Copy link
Contributor

thbar commented Feb 1, 2018

I wanted to try out TruffleRuby, so I took raw notes as I went, and thought it would be maybe useful to report them (like a kind of onboarding experience).

Please feel free to close the issue or explode it into multiple parts, but at least this can be linked if other outsiders are willing to try it out.

My interest for TruffleRuby comes from the fact I'm doing data processing with Ruby (using Kiba ETL), so if ultimately I can get a much better throughput, that will be of high interest.

I managed to get the Kiba ETL test suite to pass (it's a small gem), and also to get some little benchmarks to run.

So far my test with TruffleRuby are much slower than MRI 2.5.0, but I think this is still interesting to report. I think I'm probably using the slower TruffleRuby without SVM (if I understand https://github.com/oracle/truffleruby/blob/master/doc/user/svm.md).

Please consider this as constructive feedback even if it's in raw form, I know raw input can be helpful to have an outsider look on things!

"Hot" points

  • It's not clear to a beginner looking quickly what is the fastest TruffleRuby available today, what is SubstrateVM etc.
  • Initially I wondered about RVM support, and I found that if I install TruffleRuby manually, and I patch RVM, I could at least use RVM to switch of version
  • I wondered if I could install gems, but ultimately discovered I could use gem to install bundler etc. Worked great after downgrading bundler to a compatible version.
  • A fully automated installation via RVM would be amazing & would really help to get more outside testers involved (this has been said before, but worth repeating).
  • The GraalVM download is .... huge (compared to a RVM install).
  • require_relative seems to work differently compared to MRI (via instance_eval at least).
  • My gem tests pass (yay!)
  • My little benchmark too, but I get 28.55 seconds vs 3.79 seconds with MRI 2.5.0 (and I'm probably doing something wrong, not using the right option to start TruffleRuby etc).

Raw notes

First I wondered what is the fastest TruffleRuby available today? I'm not sure to understand what exactly is SubstrateVM, how do you install it, how it is related to GraalVM etc or not.

Ultimately I just picked the first installable version I could find.

Then I can unpack:

cd graalvm
tar -zxvf graalvm-0.30.2-macosx-amd64-jdk8.tar.gz

This way I have:

graalvm/graalvm-0.30.2

and I can run Ruby with:

graalvm/graalvm-0.30.2/Contents/Home/bin/ruby -v
truffleruby 0.30.2, like ruby 2.3.5 <Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 with Graal> [darwin-x86_64]

Why is there a ruby in two places of the tgz, I don't know.

To go further, I wanted to run the Kiba test suite, here is what I had to do:

  • Figure out where gem command is located (I thought it wouldn't exist initially, but used find to see that it was available)
  • Tweak my path like this:
export PATH=~/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/bin:$PATH
  • Tell rvm not to try to do anything (or I'd get warnings)
rvm use system
  • Figure out weither or not bundler is supported (I didn't see a bundle command in the tgz)
  • Realising that gem install minitest-focus works
  • I was then able to run:
$ ruby -v
truffleruby 0.30.2, like ruby 2.3.5 <Java HotSpot(TM) 64-Bit Server VM 1.8.0_151-b12 with Graal> [darwin-x86_64]
$ gem install minitest-focus
# this worked
$ ruby test/test_parser.rb
# this did not (cannot load such file -- kiba)
$ ruby test/test*.rb

Then I realized that maybe installing bundler as a gem would work:

$ gem install bundler

It worked fast enough, but then it started installing the documentation, which took almost 2 minutes & made my CPU go to 400%:

Parsing documentation for bundler-1.16.1
Installing ri documentation for bundler-1.16.1
Done installing documentation for bundler after 115 seconds

(Yup, 115 seconds).

I added a ~/.gemrc file with this:

gem: --no-document

in order to try to avoid that.

Then I tried to install the Kiba development dependencies, but got this error:

$ bundle install
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/patches/bundler/bundler.rb:5:in `<top (required)>': unsupported bundler version please use 1.14.x (RuntimeError)

I checked the rubygems page to verify what's the latest 1.14.x here https://rubygems.org/gems/bundler/versions then ran:

gem uninstall bundler
gem install bundler -v 1.14.6

I finally ran:

bundle install

which took my CPU back to 360%, and showed a few warnings, but ultimately succeeded:


19:17 $ bundle install
warning: already initialized constant Bundler::Dependency::PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:12: warning: previous definition of PLATFORM_MAP was here
warning: already initialized constant Bundler::Dependency::REVERSE_PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:70: warning: previous definition of REVERSE_PLATFORM_MAP was here
Fetching gem metadata from https://rubygems.org/.........................
Fetching version metadata from https://rubygems.org/..
Resolving dependencies.......
Installing rake 12.3.0
Installing awesome_print 1.8.0
Using kiba 2.0.0 from source at `.`
Installing minitest 5.11.3
Using bundler 1.14.6
Using minitest-focus 1.1.2
Bundle complete! 5 Gemfile dependencies, 6 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

At this point I was able to run Kiba's test suite:

19:20 $ bundle exec rake
warning: already initialized constant Bundler::Dependency::PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:12: warning: previous definition of PLATFORM_MAP was here
warning: already initialized constant Bundler::Dependency::REVERSE_PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:70: warning: previous definition of REVERSE_PLATFORM_MAP was here
[ruby] WARN A nonstandard GEM_HOME is set /Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0
warning: already initialized constant Bundler::Dependency::PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:12: warning: previous definition of PLATFORM_MAP was here
warning: already initialized constant Bundler::Dependency::REVERSE_PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:70: warning: previous definition of REVERSE_PLATFORM_MAP was here
Run options: --seed 4569

# Running:

......................................

Fabulous run in 2.025488s, 18.7609 runs/s, 30.6099 assertions/s.

38 runs, 62 assertions, 0 failures, 0 errors, 0 skips

YAY! Success!

Since Kiba's tests were passing, I went further to the benchmarks:

cd ~/git/kiba-ruby-benchmarks
bundle install
brew install axel
bundle exec kiba setup.etl

Then I wanted to run the benchmark, but got:

19:25 $ bundle exec kiba csv_processing.etl
warning: already initialized constant Bundler::Dependency::PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:12: warning: previous definition of PLATFORM_MAP was here
warning: already initialized constant Bundler::Dependency::REVERSE_PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:70: warning: previous definition of REVERSE_PLATFORM_MAP was here
bundler: failed to load command: kiba (/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/bin/kiba)
LoadError: cannot load such file -- csv_processing.etl/etl/csv_source
  csv_processing.etl:2:in `require_relative'

I believe there is a subtle difference in the way require paths are handled, at least when using instance_eval like Kiba does.

I replaced this:

require_relative 'etl/csv_source'

by:

require './etl/csv_source'

Then I was able to start and get a time:

$ bundle exec kiba csv_processing.etl
warning: already initialized constant Bundler::Dependency::PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:12: warning: previous definition of PLATFORM_MAP was here
warning: already initialized constant Bundler::Dependency::REVERSE_PLATFORM_MAP
/Users/thbar/graalvm/graalvm-0.30.2/Contents/Home/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/bundler-1.14.6/lib/bundler/dependency.rb:70: warning: previous definition of REVERSE_PLATFORM_MAP was here
I, [2018-02-01T19:27:33.748000 #59828]  INFO -- : Running with ruby 2.3.5
I, [2018-02-01T19:28:02.335000 #59828]  INFO -- : Processing done (took 28.55 seconds)
1406423187 519371 data/output.csv

The output seems correct (the checksum for data/output.csv) but this is massively slower than Ruby 2.5.0:

19:29 $ bundle exec kiba csv_processing.etl 
I, [2018-02-01T19:30:03.872656 #60514]  INFO -- : Running with ruby 2.5.0
I, [2018-02-01T19:30:07.665543 #60514]  INFO -- : Processing done (took 3.79 seconds)
1406423187 519371 data/output.csv
@chrisseaton
Copy link
Collaborator

Thanks this is really helpful. It's great that you managed to install a gem, all its specs passed first time, and the application actually ran (did you check the output is correct?)

The speed isn't terrible for a first look. We can start to run it with some tooling enabled and we'll see where things are.

It's possible that:

@thbar
Copy link
Contributor Author

thbar commented Feb 1, 2018

@chrisseaton glad you find this useful!

I believe the output is correct but I will double check though. I'm generating a CSV file then running cksum on it, the value is the same compared to MRI 2.5.0.

Thanks for the hints! I'll definitely dig deeper.

@ylluminate
Copy link

This is wonderful to document your effort here @thbar. As a "user" and just an excited person I thank you!

And thank you also for beating the drum on rvm. It's truly the most important thing at this point to

  1. get community feedback like this and
  2. have a standardized installation that ameliorates the hiccups from glitches that come from manual install/setup like is being discussed above.
  3. quickly propagate changes that fix issues and such that may be necessary.

I'm sure that a lot of additional benefits could be enumerated, but yeah... bang, bang, bang... 😸

@NullVoxPopuli
Copy link

re: RVM, I use chruby, so I think whatever it takes to make TruffleRuby easy to install should be able to work for both. :-)
chruby doesn't replace / override existing shell commands (bad RVM!)

@ylluminate
Copy link

ylluminate commented Feb 2, 2018

While I agree that it should be agnostic, I also believe in going after the lowest hanging fruit that has the highest return on investment. If we have to choose, then choose the one that will yield initial highest return and them move on.

Personally I don't like chruby (or rbenv - I've had various problems with them) and prefer the way rvm handles things in the shell, but I'm not wanting to start an opinion-war.

I also much prefer emacs to vi[m], and Opal (isomorphism) is FAR better than using straight JS... ;)

@thbar
Copy link
Contributor Author

thbar commented Feb 2, 2018

chruby, rvm and rbenv are all widely used by various people (it's really a matter of personal preference), so if ultimately we can find a way to solve the root problems then work at proper integrations in each, it will be a win.

@ylluminate
Copy link

@thbar theoretically that would be nice. Maybe it's possible - I hope so. Realistically I think they'll have to focus on one first and then move to the others and if that proves to be true I think they'll choose the largest user bases first and then move on (rvm > rbenv > chruby).

@chrisseaton
Copy link
Collaborator

Let's not start discussing which Ruby manager is the best in this thread :)

TruffleRuby has a clear philosophy of not debating how people want to code or what tools they want to use, and we'll aim to support any Ruby manager we can https://github.com/oracle/truffleruby/blob/a72380e7d6743304909abd4f9e85919a4126dde0/doc/user/compatibility.md#compatibility.

I think the main blocker for this is not having an easily download-able tarball. I will check our support for managers and make sure tests and documentation is in place, and open issues for any problems.

@thbar
Copy link
Contributor Author

thbar commented Feb 2, 2018

@chrisseaton if you discuss this, I think ultimately it would be great to have a much smaller download (maybe a sub-build or something) including only the parts relevant to TruffleRuby. Well - I hope you'll be heard, thanks for tackling this!

@chrisseaton
Copy link
Collaborator

I will need to look into the performance later, but here are some answers.

I think I'm probably using the slower TruffleRuby without SVM

No the SVM is faster for start-up time, but slower for peak-performance in most cases.

It's not clear to a beginner looking quickly what is the fastest TruffleRuby available today, what is SubstrateVM etc.

Ok we'll add more documentation on this.

Initially I wondered about RVM support, and I found that if I install TruffleRuby manually, and I patch RVM, I could at least use RVM to switch of version

We'll renew our documentation and tests for the Ruby version managers.

I wondered if I could install gems, but ultimately discovered I could use gem to install bundler etc. Worked great after downgrading bundler to a compatible version.

That's normal isn't it? Use the included gem command to install bundler? What didn't work as expected?

A fully automated installation via RVM would be amazing & would really help to get more outside testers involved (this has been said before, but worth repeating).

Yes I'll renew looking into this.

The GraalVM download is .... huge (compared to a RVM install).

This isn't likely to change soon. It's a few hundred MB yes. I don't think that's totally unreasonable compared to a normal JVM.

require_relative seems to work differently compared to MRI (via instance_eval at least).

Did you open an issue for this?

Why is there a ruby in two places of the tgz, I don't know.

Each language provides its own directory of binaries, and then there are links to them from the main binary directory.

Figure out where gem command is located

Yes this has some internal technical problems behind it. We'll fix it.

@eregon
Copy link
Member

eregon commented Feb 2, 2018

A couple more answers:

warning: already initialized constant ...

The warnings are gone on master, it was a bug that got in GraalVM 0.30.

My little benchmark too, but I get 28.55 seconds vs 3.79 seconds with MRI 2.5.0

I think it's best to split that in another issue, specifically about performance of your workload.

@chrisseaton
Copy link
Collaborator

General documentation improvements in #1060.

@chrisseaton
Copy link
Collaborator

chrisseaton commented Feb 3, 2018

Apart from the macOS patch for RVM, which we need to upstream and will, all three major Ruby version managers work fine and are tested.

#1061

They just can't download TruffleRuby for you - that's a missing bit.

@chrisseaton
Copy link
Collaborator

Actually the macOS patch for RVM isn't needed any more, so they all work out-of-the-box now.

@chrisseaton
Copy link
Collaborator

I think I've opened new issues for all the specific remaining problems identified here.

@gettalong
Copy link

Will it be possible in the future to just download TruffleRuby without needing a one-time registration (and yet another one-time-use account)? If the setup would be fully automatic, you would probably get loads of more people testing with it.

@chrisseaton
Copy link
Collaborator

We hope soon to make it possible to just download TruffleRuby without any registration at all.

@ghost
Copy link

ghost commented Mar 30, 2018

what's the difference between Truffle ruby and Crystal lang? btw, thx for your huge report, it seems promising

e: thx for the responses btw.. very nice and good info :D

@eregon
Copy link
Member

eregon commented Mar 30, 2018

@girng I am no Crystal expert but here is my view of it, which might be slightly wrong.
Crystal is a statically-typed language, like say Kotlin and Scala. TruffleRuby is an implementation of Ruby, a highly-flexible dynamically-typed language.
So in practice, there is much less dynamism at runtime in Crystal, since almost everything (method lookups, etc) is resolved at compile time (and so, e.g., dynamic method definition is not possible).

@thbar
Copy link
Contributor Author

thbar commented Mar 30, 2018

@girng glad you found the report useful!

I believe TruffleRuby aims at being a drop-in replacement which you can use on regular ruby programs, while Crystal (which I used on occasions) is a ruby-spirited language (but you won't be able to run regular ruby programs unmodified, you will have to adapt, sometimes lightly, sometimes quite deeply, as I've experienced).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants