Add credentials using a generic EncryptedConfiguration class #30067

dhh · 2017-08-03T23:17:02Z

The combination of config/secrets.yml, config/secrets.yml.enc, and SECRET_BASE_KEY is confusing. It's not clear what you should be putting in these secrets and whether the SECRET_BASE_KEY is related to the setup in general.

This PR will deprecate secrets.yml* and instead adopt config/credentials.yml.enc to signify what these secrets are specifically for: Keeping API keys, database passwords, and any other integration credentials in one place.

This new file is a flat format, not divided by environments, like secrets.yml has been. Most of the time, these credentials are only relevant in production, and if someone does need to have some keys duplicated for different environments, it can be done by hand.

This leaves us with what to do with SECRET_BASE_KEY. For test and development, we'll instead simply derive values for both based off a known constant. These secrets are not supposed to withstand any sort of attack in test or development.

It's only in production (and derivative environments, like exposed betas) where the secret actually needs to be secret. So we will birth the new credentials.yml.enc file with secret_key_base set instead on new apps.

This change will setup the encrypted credentials along with the new application skeleton, because even if you make a mistake and forget your key, it's only production that's impacted. So it won't slow down anyone just getting started with development or test.

Finally, this would leave us with a single RAILS_MASTER_KEY for services like Heroku to set. Once this has become the standard, they could even ask for this key when it's missing on the rails buildkit deploy, such that the setup is all smooth.

Note: We will just keep Rails.secrets and friends around. The Rails.application.credentials setup will be a new, concurrent approach. All new apps would use it, but we wouldn't need to screw existing apps.

Work left:

Implement config.require_master_key instead of config.read_encrypted_secrets.
Alert at the end of "rails new app" that there is a master key, what it is, and that you should put it in your password manager.
Update all documentation to talk about credentials rather than secrets.
Update all railties tests to deal with missing secrets.yml and new master.key and credentials.yml.enc

This is sketch code so far.

morgoth · 2017-08-04T06:37:22Z

railties/lib/rails/application.rb

@@ -400,6 +401,13 @@ def secrets
      end
    end

+    def credentials
+      @credentials ||= ActiveSupport::EncryptedConfiguration.new \
+        config_path: Rails.root.join("config/credentials.yml.enc"), 


@dhh This is great that ActiveSupport::EncryptedConfiguration takes this file path as an argument.
Do you think we can expand it a little and make it configurable so one can easily work with few encrypted files, ie for "production (and derivative environments, like exposed betas)"?

And have some nice tooling like:
bin/rails credentials:edit config/credentials-with-my-custom-name.yml.enc that supports out of the box finding key with relative name?

I tried to bring it to secrets in #29909 but I can work on it here if you like the idea

I'd like to see bin/rails encrypted:edit FILE, which will use the new config/master.key as its key, and which we can make accessible through something like Rails.application.encrypted.config(path)[:key]. Let's do that as a separate PR.

dhh · 2017-08-04T18:44:56Z

Keen to come out with something generic that can be used to deal with many, separate encrypted files. But less keen to complicate the credentials feature itself with this. I see credentials as one specific use case for EncryptedConfiguration, but that there could be many other such uses. So let's see if we can't come up with some reusable primitives that people can use to build their own credentials-like feature without burdening the feature itself.

…

On Fri, Aug 4, 2017 at 1:38 AM, Wojciech Wnętrzak ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In railties/lib/rails/application.rb <#30067 (comment)>: > @@ -400,6 +401,13 @@ def secrets end end + def credentials + @credentials ||= ActiveSupport::EncryptedConfiguration.new \ + config_path: Rails.root.join("config/credentials.yml.enc"), @dhh <https://github.com/dhh> This is great that ActiveSupport:: EncryptedConfiguration takes this file path as an argument. Do you think we can expand it a little and make it configurable so one can easily work with few encrypted files, ie for "production (and derivative environments, like exposed betas)"? And have some nice tooling like: bin/rails credentials:edit config/credentials-with-my-custom-name.yml.enc that supports out of the box finding key with relative name? I tried to bring it to secrets in #29909 <#29909> but I can work on it here if you like the idea — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30067 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKtYvIueyQHZZ-XriqpLI7jmwL3Sabks5sUrxWgaJpZM4OtGcA> .

dhh · 2017-08-04T19:07:07Z

Thinking about this further, I wonder if there's even greater room for simplification. Maybe we can combine the secret_key_base with the RAILS_MASTER_KEY and then simply rely on a single master key that can live in that ENV or in config/master.key. Then we wouldn't have a separate secret_key_base, and all apps would be required to have either RAILS_MASTER_KEY env or config/master.key set. We could further more easily allow other encrypted configurations to piggy back off that env/file key, kinda like we do with the message_verifier generator.

Are there any security issues with using the RAILS_MASTER_KEY as the secret_key_base as well?

…er envs And document the usage.

Still need to make it happen, tho.

Usually we say we change defaults, not "spec" out a release. Can't use backticks in our sdoc generated documentation either.

kaspth · 2017-08-07T18:10:23Z

activesupport/lib/active_support/encrypted_configuration.rb

+      @config ||= deserialize(read).deep_symbolize_keys
+    end
+
+    # Saves the current configuration to file, but won't persist any comments that where there already!


"but won't persist any comments there already!" are you mentioning the updated_contents != contents line to only write contents if they're changed?

No, I mean that #save just serializes the in-memory config hash. It doesn't involve a text editor. Maybe we should drop that entirely.

kaspth · 2017-08-07T18:12:24Z

activesupport/lib/active_support/encrypted_configuration.rb

+    end
+
+    def config
+      @config ||= deserialize(read).deep_symbolize_keys


Should we use ActiveSupport::InheritedOptions here on top of this? It would allow things like Rails.credentials.aws_access_key?

kaspth · 2017-08-07T18:17:29Z

activesupport/lib/active_support/encrypted_file.rb

+    end
+
+
+    attr_reader :content_path, :key_path, :env_key


What do we gain by exposing these readers?

kaspth · 2017-08-07T18:26:38Z

activesupport/test/encrypted_file_test.rb

+  test "change content by key file" do
+    @encrypted_file.write(@content)
+    @encrypted_file.change do |file|
+      file.write(file.read + " and went by the lake")


Really dig the flow that Pathnames and the change method bring! 👏

Pathnames are mint!

kaspth · 2017-08-07T18:41:29Z

railties/lib/rails/application.rb

@@ -404,6 +404,32 @@ def secrets=(secrets) #:nodoc:
      @secrets = secrets
    end

+    # The secret_key_base is used as the input secret to the application's key generator, which in turn


Had this lined up to deprecate secrets but looks like it's a bit more involved than that. Let's defer the deprecation to after this is merged.

ActiveSupport::Deprecation.warn(<<-end_of_message.strip_heredoc) `Rails.application.secrets` has been deprecated in favor of `Rails.application.credentials` and will be removed in Rails 6.0. See `bin/rails credentials --help` for more. end_of_message

(Most trouble is just that our test suite relies on secrets, so will be seeing some deprecation warnings.)

kaspth · 2017-08-07T19:16:38Z

activestorage/test/dummy/config/environments/production.rb

-  config.read_encrypted_secrets = true
+  # Ensures that a master key has been made available in either ENV["RAILS_MASTER_KEY"]
+  # or in config/master.key. This key is used to decrypt credentials (and other encrypted files).
+  # config.require_master_key = true


wrong production.rb, now I get why I couldn't find it 😅

BerkhanBerkdemir · 2018-08-02T01:58:35Z

Today, I faced "the problem", env based encryption, when I trying to integrate Algolia. I think for now, the best practice is using secure_credentials gem. Is there any best practice?

kaspth · 2018-08-02T18:36:05Z

I'd like to make some things clear: Credentials is a conceptual rethink from Secrets and Encrypted Secrets. It charters a new path and thus solves fewer use cases than the old way did. The Core team is not in anyway opposed to supporting multiple environment credentials as long as it doesn't complicate the existing use case as @dhh made clear #30067 (comment).

However, the suggestions we've seen for multi environment credentials so far haven't been up to par. For instance, the secure_credentials gem flies right in the face of the motivation stated in the PR description:

The combination of config/secrets.yml, config/secrets.yml.enc, and SECRET_BASE_KEY is confusing. It's not clear what you should be putting in these secrets and whether the SECRET_BASE_KEY is related to the setup in general.

Do volunteer some PRs yourselves! I'm pretty confident to say that multi env credentials are a lot closer than you'd think. The biggest hurdle that I see is people continuing to conceptualize and compare with the secrets way. That's the hard way. 😄

@BerkhanBerkdemir my suggestion is here #30067 (comment). True multi env credentials could perhaps smooth this over.

Thanks for all the feedback and ideas! Those who feel the burn know best how to put out the fire, so let's level Rails up together ❤️

sidot3291 · 2018-08-02T20:22:00Z

Thanks for the rallying cry @kaspth

What do you think of initializing a key and credentials file for each environment within config/environments?

e.g:

> ls config/environments:
    development.rb
    development.key
    development.credentials.yml.enc
    production.rb
    production.key
    production.credentials.yml.enc
    test.rb
    test.key
    test.credentials.yml.enc

Then:

All keys are .gitignore'd
Update CLI so it's rails credentials:edit production to edit production credentials

This adds to the config directory a bit but still allows for atomic deploys of environment variables.

My hesitation with this is that my setup would require "staging" credentials, which intuitively would require a fourth "staging" environment. CLI could be built so generating this is easy, e.g:
rails environments:add staging

It's worth noting, though, Heroku explicitly pushes me away from this approach:

It may be tempting to create another custom environment such as “staging” and create a config/environments/staging.rb and deploy to a Heroku app with RAILS_ENV=staging. This is not a good practice. Instead we recommend always running in production mode and modifying any behavior by setting your config vars.
https://devcenter.heroku.com/articles/deploying-to-a-custom-rails-environment

That raises two options:

Don't care. The credentials concept is intentionally pushing atomic, checked-in environment variables instead of system environment variables.
Make environment settings for the credentials key and file, e.g. in production.rb:

config.credentials.key = ENV['CREDENTIALS_KEY'] || Rails.root.join('config/credentials/production.key').read
config.credentials.file = Rails.root.join(ENV['CREDENTIALS_FILE'] || 'config/credentials/production.credentials.yml.enc')

thomasnal · 2018-08-02T20:27:31Z

I would give it its own folder such as config/environments or config/credentials is a great way to go. My five cents, as explained earlier, seems to support all variety of requests we have seen.

sidot3291 · 2018-08-02T20:49:20Z

As is normally the case after clicking send, I find myself leaning toward the second option, since the entirety of my fourth "staging" environment would be copy of production, except for credentials.

So to round things out:

Add a config/credentials folder with:

ls config/credentials
    development.key
    development.yml.enc
    production.key
    production.yml.enc
    test.key
    test.yml.enc

Update the CLI
rails credentials:edit {token} edits the corresponding file in the credentials folder
rails credentials:add {token} generates a new key/file pair
Add two environment settings to the existing environments/{env}.rb files:

config.credentials.key = ENV['CREDENTIALS_KEY'] || Rails.root.join('config/credentials/production.key').read
config.credentials.file = Rails.root.join(ENV['CREDENTIALS_FILE'] || 'config/credentials/production.credentials.yml.enc')

Would this work?

Fedcomp · 2018-08-02T20:51:47Z

Seems to be over complicated, instead it''s easier to use plain old ENV variables.

morgoth · 2018-08-02T20:52:09Z

hey all - please take a look at #33521 if it would be enough for your usecases

christophermlne · 2018-08-02T23:22:36Z

I didn't find the old system confusing but this new system is very confusing and it confused everyone in our office. The devs are now passing the master key around through Slack.

printercu · 2018-08-03T07:53:03Z

@kaspth

For instance, the secure_credentials gem flies right in the face of the motivation stated in the PR description

There is no secrets.yml and secrets.yml.enc in secure_credentials gem. There are multienv secrets.yml and secrets.:env.yml.enc for any other envs. I've explained why having encrypted credentials for dev/test envs is a bad practice on this wiki page: because it gives illusion of security and encourages sharing secret key across all developers.

After trying many different ways of managing credentials, I've came to not checking secrets.yml to vcs but using secrets.sample.yml. This allows to share dev credentials selectively (yes, they are secret in some cases too) and prevent occasional check-ins of this files with sensitive data. And I like the idea of storing encrypted production credentials in vcs. So I've combined both approaches in secure_credentials.

prashantham · 2018-08-04T15:43:01Z

I had to implement a CI/CD environment for a rails 4.2 app on google cloud. The server image needed to have all the source including secrets but in encrypted .env files. The encrypted files would then be decrypted when instance was created using kms. Now this required different files for our stage/prod envs i.e .env.stage.enc and .env.prod.env. Google project would hold metadata related to cipher file name which are available to instances. systemd task would then decrypt them as .env.

So i think we need to have 2 env variables one for master key and another for cipher file.

kaspth · 2018-08-12T17:26:58Z

Thanks for the further feedback everyone! @morgoth even submitted a PR for us to hammer things out on! I've submitted my thoughts about a potential file structure here: #33521 (comment)

@christophermlne can you elaborate on what you and your team found confusing about this setup? Why wasn't it confusing with the old secrets.yml + secrets.yml.enc?

@printercu sorry I wasn't clear enough. It's not the files being present, it's having two similarly named files and only one being the encrypted one.

I'm less in favor of the ENV variables suggestions to do lookup. Credentials is an attempt to cut down on the number of ENV variables an app should manage. Though we could expose config.credentials.key_path or config.credentials.key and config.credentials.content_path, which you can override via ENV variables yourselves.

kaspth · 2018-09-20T17:30:57Z

Multiple environment credentials have just landed thanks to @morgoth! Feel free to throw as many GitHub reactions on his PR as you want 😄
#33521

I'd just like to say thanks to everyone who chimed in here with all their suggestions. It resulted in us ultimately moving forward and making Rails better. We couldn't have asked for a better outcome ❤️

Update the project from Rails 5 to Rails 6. This update introduces two major changes to the app: Rails credentials, and the Webpacker gem. Rails credentials ----------------- Rails 5.2 introduced a replacement for Secrets: [Credentials]. Credentials provides a method for saving environment variables in an encrypted format that is safe to keep in the repository. This feature depends on a master key, which must be shared out-of-band. Webpacker --------- The [Webpacker gem] allows for easy integration of [webpack] into Rails apps. This gem is designed to work with the asset pipeline, so existing JavaScript and SASS should continue to work. [Credentials]: rails/rails#30067 [Webpacker gem]: https://github.com/rails/webpacker [webpack]: https://webpack.js.org/

This file is deprecated and, for this particular app, doesn't contain any real secret. See rails/rails#30067

dhh added 4 commits August 3, 2017 18:16

WIP: Add credentials using a generic EncryptedConfiguration class

e2df7e4

This is sketch code so far.

Flesh out EncryptedConfiguration and test it

f9b6272

Better name

6e3948a

Add command and generator for credentials

820c655

dhh mentioned this pull request Aug 4, 2017

Turn secrets.yml* into credentials.yml.enc #30006

Closed

dhh changed the title ~~WIP: Add credentials using a generic EncryptedConfiguration class~~ Add credentials using a generic EncryptedConfiguration class Aug 4, 2017

Use the Pathnames

476809b

morgoth reviewed Aug 4, 2017

View reviewed changes

dhh added 11 commits August 4, 2017 18:10

Merge branch 'master' into credentials

96dd349

Extract EncryptedFile from EncryptedConfiguration and add serializers

1df091c

Test EncryptedFile

c9e2af5

Extract serializer validation

8d48b1b

Stress the point about losing comments

5be0993

Allow encrypted configuration to be read without parsing for display

1af0901

Use credentials by default and base them on the master key

264b261

Derive secret_key_base in test/dev, source it from credentials in oth…

329e764

…er envs And document the usage.

Document the new credentials setup

08cf8e1

Stop generating the secrets.yml file now that we have credentials

f8c249b

Document what we should have instead

5fc2d89

Still need to make it happen, tho.

dhh requested a review from kaspth August 7, 2017 17:02

kaspth added 5 commits August 7, 2017 20:18

[ci skip] Keep wording to key base; prefer defaults.

2a9f1e7

Usually we say we change defaults, not "spec" out a release. Can't use backticks in our sdoc generated documentation either.

Abstract away OpenSSL; prefer MessageEncryptor.

b994526

Spare needless new when raising.

378b5cc

Encrypted file test shouldn't depend on subclass.

1e90ea2

[ci skip] Some woordings.

71330c0

kaspth reviewed Aug 7, 2017

View reviewed changes

Ditch serializer future coding.

cc3d01a

kaspth reviewed Aug 7, 2017

View reviewed changes

morgoth mentioned this pull request Aug 2, 2018

Add support for multi environment credentials. #33521

Merged

soteras mentioned this pull request Aug 3, 2018

Kind of idea. Credentials phoenixframework/phoenix#2993

Closed

kirikiriyamama mentioned this pull request Oct 5, 2018

Active Storageの設定 nnctzoo/meeemories-server#3

Merged

jespr mentioned this pull request Oct 26, 2018

Use the new credentials API in Rails 5.2 excid3/jumpstart#63

Merged

storresy mentioned this pull request Dec 14, 2018

Send email when a rent is created wolox-training/st-rails#13

Merged

emmabeynon mentioned this pull request Nov 22, 2019

build(deps): bump rails from 5.2.3 to 6.0.1 alphagov/hmrc-manuals-api#430

Merged

evsasse mentioned this pull request Dec 12, 2019

Ignore Rails 6 master.key used on the new credentials mechanism toptal/gitignore#255

Closed

5 tasks

GremL1N mentioned this pull request Dec 24, 2019

credentials through rails credentials bigbluebutton/greenlight#905

Closed

bonniegee mentioned this pull request Mar 31, 2020

Look into Credentials for managing secrets RockefellerArchiveCenter/dm_log#67

Open

edwardkerry mentioned this pull request Aug 10, 2020

Update rails and other gems alphagov/datagovuk_find#790

Merged

pudiva added a commit to alphagov/datagovuk_find that referenced this pull request Aug 11, 2020

Remove stray /config/secrets.yml

09db4c9

This file is deprecated and, for this particular app, doesn't contain any real secret. See rails/rails#30067

nimmolo mentioned this pull request Jan 17, 2022

Created encrypted credentials and master key MushroomObserver/mushroom-observer#806

Merged

trevorturk mentioned this pull request Nov 15, 2022

Support ERB in Rails.credentials #46508

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add credentials using a generic EncryptedConfiguration class #30067

Add credentials using a generic EncryptedConfiguration class #30067

dhh commented Aug 3, 2017 •

edited by kaspth

morgoth Aug 4, 2017

dhh Aug 6, 2017

dhh commented Aug 4, 2017 via email

dhh commented Aug 4, 2017

kaspth Aug 7, 2017

dhh Aug 7, 2017

kaspth Aug 7, 2017

dhh Aug 17, 2017

kaspth Aug 7, 2017

kaspth Aug 7, 2017

dhh Aug 7, 2017

kaspth Aug 7, 2017

kaspth Aug 7, 2017

BerkhanBerkdemir commented Aug 2, 2018

kaspth commented Aug 2, 2018

sidot3291 commented Aug 2, 2018

thomasnal commented Aug 2, 2018

sidot3291 commented Aug 2, 2018

Fedcomp commented Aug 2, 2018

morgoth commented Aug 2, 2018

christophermlne commented Aug 2, 2018

printercu commented Aug 3, 2018 •

edited

prashantham commented Aug 4, 2018

kaspth commented Aug 12, 2018

kaspth commented Sep 20, 2018

Add credentials using a generic EncryptedConfiguration class #30067

Add credentials using a generic EncryptedConfiguration class #30067

Conversation

dhh commented Aug 3, 2017 • edited by kaspth

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhh commented Aug 4, 2017 via email

dhh commented Aug 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BerkhanBerkdemir commented Aug 2, 2018

kaspth commented Aug 2, 2018

sidot3291 commented Aug 2, 2018

thomasnal commented Aug 2, 2018

sidot3291 commented Aug 2, 2018

Fedcomp commented Aug 2, 2018

morgoth commented Aug 2, 2018

christophermlne commented Aug 2, 2018

printercu commented Aug 3, 2018 • edited

prashantham commented Aug 4, 2018

kaspth commented Aug 12, 2018

kaspth commented Sep 20, 2018

dhh commented Aug 3, 2017 •

edited by kaspth

printercu commented Aug 3, 2018 •

edited