RailsConf 2018 | The Future of Rails 6: Scalable by Default

Transcript

1. The Future of Rails 6 Scalable by Default

2. Eileen M. Uchitelle eileencodes.com @eileencodes

3. a

4. Core Team

5. “Rails doesn’t scale!”

6. None

7. Rails doesn’t scale

8. Rails doesn’t scale easily

9. Rails doesn’t scale easily (yet)

10. What does “scalable” mean?

11. Your test suite shouldn’t block development

12. Your application can handle data & traffic

13. The Future of Rails 6 Scalable by Default

14. Parallel Testing Speeding up your test suite

15. a

16. a

17. Forking - vs - Threads

18. Forking Processes with dRB

19. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end

21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end

22. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size

    @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end

23. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size

    @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end

24. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end

25. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]

26. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]

27. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]

28. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]

29. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]

30. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]

31. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]

32. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]

33. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times

    { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end

34. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end

35. test-database-0 test-database-1

36. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]

37. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker|

    fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]

38. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) parallelize_setup do |worker| ActiveRecord::Tasks::DatabaseTasks.create(:animals)

    ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.load_schema end parallelize_teardown do |worker| ActiveRecord::Tasks::DatabaseTasks.drop(:animals) end end

39. Threaded testing with Minitest

40. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2, with: :threads) end

41. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
42. None

43. Connection ID: #03938752 Connection ID: #83321340

44. Connection ID: #83321340 Connection ID: #83321340

45. This is so weird. We seem to have an isolation

    problem…
46. None

47. Rails opens a connection

48. Puma opens a second connection Rails opens a connection

49. Puma opens a second connection Rails opens a connection

50. Test thread connection number 1 Test thread connection number 2

51. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end

52. $ rails test

53. $ PARALLEL_WORKERS=14 rails test

54. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]

    return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end

55. Parallel Testing PR github.com/rails/rails/pull/31900

56. Your test suite shouldn’t block development

57. Your application can handle data & traffic

58. None
59. None

60. DB 1 DB 2 DB 3

61. Multiple Databases Making multi-db behavior easy

62. Primary

63. DB 1 DB 2 DB 3

64. Table id: integer name: string Table id: integer name: string

    Table id: integer name: string Table id: integer name: string Primary Animals

65. X Best practices are undocumented The State of Multi-db in

    Rails 5

66. # database.yml production: <<: *default database: db_production production_animals: <<: *default

    database: db_animals_production

67. # database.yml production: <<: *default database: db_production production_animals: <<: *default

    database: db_animals_production

68. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production

69. db/animals_migrate ↳ 20180206210429_create_dogs.rb ↳ 20180206210437_create_cats.rb

70. ActiveRecord::Migrator.migrations_paths = "db/animals_migrate" ActiveRecord::Base.migrate

71. X Best practices are undocumented X Migrations don’t work The

    State of Multi-db in Rails 5

72. # app/models/animals_base.rb class AnimalsBase < ActiveRecord::Base self.abstract_class = true establish_connection

    :animals end

73. # app/models/dog.rb class Dog < AnimalsBase […] end # app/models/cat.rb

    class Cat < AnimalsBase […] end

74. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist The State of Multi-db in Rails 5

75. ActiveRecord::AdapterNotSpecified: 'production' database is not configured.

76. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning

77. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration

    begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning

78. X Best practices are undocumented X Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5

79. Migrations Connections don’t work Rake tasks

80. None

81. Fixing migrations for multiple databases

82. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db"

    task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end

83. # 3-tier database.yml production: primary: <<: *default database: db_production animals:

    <<: *default database: db_animals_production migrations_paths: db/animals_migrate

84. >> Dog.connection.migrations_paths => "db/animals_migrate"

85. Migrations Paths Refactoring PR github.com/rails/rails/pull/31727

86. X Best practices are undocumented ✔ Migrations don’t work X

    Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5

87. Migrations Connections don’t work Rake tasks

88. Improving the database tasks for multi-dbs

89. rails db:drop rails db:create rails db:migrate

90. >> ActiveRecord::Base.configurations["production"] >> { "adapter"=>"mysql2", "database"=>"single_database_production" }

91. >> ActiveRecord::Base.configurations["production"] >> { "primary"=>{"adapter"=>"mysql2", "database"=>"multiple_databases_production"}, "animals"=>{"adapter"=>"mysql2", “database"=>"multiple_databases_production"} }

92. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end

93. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end

94. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end

95. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config

    def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end

96. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon

    fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>

97. >> db_config.env_name "production"

98. >> db_config.env_name "production" >> db_config.spec_name "animals"

99. >> db_config.env_name "production" >> db_config.spec_name "animals" >> db_config.config {…"database"=>"multiple_databases_producti on_animals"}

100. def self.db_configs(configs = configurations) configs.each_pair.flat_map do |env_name, config| walk_configs(env_name, "primary",

     config) end end

101. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

     == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end

102. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

     == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end

103. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name

     == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end

104. >> DatabaseConfigurations.configs_for("production") >>[#<ActiveRecord::DatabaseConfigurations::Database Config:0x00007f994571edb0 @env_name="production", @spec_name="primary", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production"}>, #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0

     @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>]

105. task migrate: :load_config do ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["_dump"].invoke end

106. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

     ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end

107. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

     ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end

108. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

     ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end

109. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config|

     ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end

110. rails db:drop rails db:create rails db:migrate

111. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

     rails db:migrate:animals

112. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

     spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end

113. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

     spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end

114. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

     spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end

115. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

     spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end

116. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task

     spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end

117. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary

     rails db:migrate:animals

118. Database task improvements github.com/rails/rails/pull/32274 github.com/rails/rails/compare/ ab43b5405ea2...5ddcda6d5f56

119. X Best practices are undocumented ✔ Migrations don’t work ✔

     Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5

120. Connections don’t work Migrations Rake tasks

121. None

122. Thanks @tenderlove!

123. “Are you concerned that Rails will become bloated?”

124. What would Rails look like if we’d upstreamed these features

     5 years ago?

125. a +

126. ' ( ) * + ,

127. None

128. Maybe Rails doesn’t scale…?

129. Your application isn’t special.

130. Forces you to write generic code

131. Train your future workforce

132. Give back to the community

133. ' ( ) * + ,

134. a♥

135. Scaling should make us happy

136. Rails isn’t dying, it’s maturing

137. None
138. None
139. None

140. “Rails does scale!”

141. The Future of Rails 6 Scalable by Default

142. Eileen M. Uchitelle eileencodes.com @eileencodes