New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add encrypted secrets #28038
Add encrypted secrets #28038
Conversation
require "active_support/core_ext/string/strip" | ||
require "rails/generators/rails/app/app_generator" | ||
|
||
require "rails/engine" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sekrets depends on Rails::Engine out of the box.
Later I'd like to inline just the parts of Sekrets we need, since it seems to a lot more than we require as well as pull in a number of dependencies (nothing wrong with that, but it's best Rails keeps slim). @ahoward how would you feel about that? We'd inline the parts we're using into it's own private namespace and give you credit with: Also: as a heads up, it seems Sekrets isn't distributed with a license :) |
Dang, I've got trouble getting https://github.com/kaspth/rails/blob/226db6a15bc5ce5aca1854d7d2a00ab221795f86/railties/lib/rails/application.rb#L388 to recognize |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
`RAILS_MASTER_KEY` environment variable, with the former taking precedence. | ||
|
||
By default Rails adds `config/secrets.yml.key` and `config/secrets.yml.enc` to | ||
`.gitignore`. If you don't use Git add these to the respective version control |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use Git, add
Rails looks for the decryption key in config/secrets.yml.key as well as the | ||
`RAILS_MASTER_KEY` environment variable, with the former taking precedence. | ||
|
||
By default Rails adds `config/secrets.yml.key` and `config/secrets.yml.enc` to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
default, Rails
to edit production secrets. | ||
|
||
When the temporary file is next saved the contents are encrypted and written to | ||
config/secrets.yml.enc while the file itself is destroyed to protect secrets |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
secrets from leaking.
I don't understand why I'm also hesitant to generating this key upfront. It's pretty easy for someone then starting a new project to use this feature but forget to do something about the key file, like commit it to 1Password. If they forget that, then change computers, the encrypted data is inaccessible. IMO, better that there's a kick-off for the first time you use encrypted secrets that explains that we're generating the key, putting it there, and you should store it securely somewhere outside of the repo. Otherwise I'm liking the simplicity of this a lot! |
@jeremy Could you review as well? |
So I'd add |
It shouldn't! That was me being a dummy, thanks 👍 |
Gemfile.lock
Outdated
fattr (>= 2.2.1) | ||
highline (>= 1.6.15) | ||
main (>= 5.1.1) | ||
map (>= 6.3.0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a lot of deps 😐
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking the same.
I'm not sure if this is really worth it to integrate since you can do almost all of this as described in https://github.com/ahoward/sekrets/blob/master/README when you need this feature. I think the only difference is key generation (where Sekrets will ask you for key). That can be solved in Sekrets and I think it will be enough to update guides to recommend this approach for storing secret data within project.
But maybe I'm missing something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed: #28038 (comment) 😊
@@ -80,7 +80,7 @@ def paths | |||
@paths ||= begin | |||
paths = super | |||
paths.add "config/database", with: "config/database.yml" | |||
paths.add "config/secrets", with: "config/secrets.yml" | |||
paths.add "config/secrets", with: "config", glob: "secrets.yml{,.enc}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maclover7 thanks for the tip!
We could potentially try to remove the need for the So when reading |
55dfeaa
to
21bc226
Compare
Adds support for atom and other editors with open commands that return immediately.
Couldn't be called without moving to the generator. Prefer duplication.
Adds a more pleasant first boot experience where the key and file plus ignoring them is set up if they're not present.
* Add support for immediately exiting editors such as atom (though done crudely with Time.now), while keeping vim etc. happy. * Prefill content if the file has none (in time, I'd like to inline sekrets with proper attribution and making it easier to do this prefill before the editing). * Add puts statement to guide clue the user in to what's happening.
Helps parsing yml files and wraps Sekrets to make it easier to remove later.
The whole point of this exercise is to add it to version control! 😇
Guess who's gonna die next!
Removes sekrets as a dependency but leaves an attribution in its place.
Will be nice to add this to CHANGELOG also. |
Is If we run |
Wrote a little getting-started/investigation blog post http://www.starkandwayne.com/blog/rails-5-1-applications-can-be-a-lot-more-secretive-on-cloud-foundry-and-heroku/ |
|
||
def generate_key | ||
cipher = new_cipher | ||
SecureRandom.hex(cipher.key_len)[0, cipher.key_len] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is limiting the key's entropy. The output of
SecureRandom.hex(n).size # => 2n
is twice as long because n
is telling SecureRandom
"give me an output that is equivalent to n
random bytes". Hex encoding encodes every byte as two characters, so the resulting hex string would be twice as long.
For the default AES-256 n = 32
, and by cutting the 64 hex bytes down to 32 again, the entropy in that value is now only equivalent to 16 random bytes anymore. This makes brute forcing the key much easier: Instead of 2^256 now there are only 2^128 possible keys.
I would suggest simply using the full output of SecureRandom.hex(cipher.key_len)
and decoding the resulting 64 hex bytes when the key is read from the file, yielding the 32-byte binary key.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this has been addressed: #28139
end | ||
|
||
def new_cipher | ||
OpenSSL::Cipher.new("aes-256-cbc") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be preferable to use an AEAD cipher mode like AES-GCM that also guarantees the integrity of the resulting ciphertext. As a fallback when the underlying OpenSSL version does not support GCM yet, "Encrypt-then-Mac" could be used. ActiveSupport::MessageEncryptor
already supports this with its #encrypt_and_sign
method - would it be possible to reuse it in this context?
Let me know what you think, I'd be happy to help.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@morgoth OK, it's already being discussed, that's great, thank you for the hint!
Sorry to sealion in here but someone referred me to this PR from Twitter and it seems like you've set things up to use only a single key. This will likely make key rotations in production environments either very tricky or impossible. I would ask if you have the time before the feature goes live to make it possible to keep multiple copies of encrypted settings such that new code and new keys don't have to be updated in perfect lockstep. Also I would recommend warning people about the dangers of storing the key in an environment variable. It does make life easier for PaaS users but caveat emptor. Tools like Airbrake and Sentry will need to be updated ASAP to not automatically log the environment variable (and configuration for each should be used in the interim). Also storing a key in an environment variable means it is passed to all subprocesses of Rails, which can be an issue for things like ImageMagick for thumbnail generation. |
Key rotation would be amazing. We don't have that for secret_key_base now do we? |
Also rollbar and sentry AFAIK use env var whitelisting so they won't need to be updated. If airbrake does something different then it has bigger problems than this PR. |
Probably the best that can be done for rotation for the moment is to rework the filename structure so rather than a single |
@coderanger - one of the main uses we've found for app specific secrets is keeping things like credit cards, private certs, etc, stored in a repo. aka, not just *.yml based config. this led to the development of an 'Sekrets.load(path)' api. from there it is trivial to support Sekrets.load("#{ Rails.env }.yml") etc. so totally agree. |
end | ||
|
||
def path | ||
@root.join("config", "secrets.yml.enc").to_s |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaspth Do you think this could be somehow configurable? I can imagine having few encrypted files based on environment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm open to hearing that argument. But let's revisit that with a written up use case once 5.1 has been out for some time proper 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 for making it configurable. Actually I would not use this feature at all until it's possible to set different encrypted secretes for production, development or staging (or any other) environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I cannot use this feature until there is a configureable option for the paths.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm open to hearing that argument. But let's revisit that with a written up use case
@ce07c3 ^ all yours. Volunteer the write up if you're in a hurry 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaspth I have messy but working code now. I named my files secrets.yml.enc.staging|production|...
, which clashed with if path.end_with?(".enc")
(took a while to figure that out), but now I am wondering: are there identical by value but different sets of paths used? I did an override on path
and key_path
inside Rails::Secrets class << self
, but those aren't called until later. Any idea how I am supposed to wire custom paths up? Also, how is @read_encrypted_secrets
set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaspth Also, how is @read_encrypted_secrets set?
figured that out :) I was a bit tired. The rest I am still wondering about, perhaps I just need to dig a bit deeper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ce07c3 Are you still working on making it configurable? Can I help you somehow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if anyone will hear this, but I would also like to get a different master key per environment.
# Attempt to read encrypted secrets from `config/secrets.yml.enc`. | ||
# Requires an encryption key in `ENV["RAILS_MASTER_KEY"]` or | ||
# `config/secrets.yml.key`. | ||
config.read_encrypted_secrets = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello @kaspth 😄
The USAGE file above says:
A
shared:
top level key is also supported such that any keys there is merged into the other environments.
This leads me to believe that any "shared" secret key will automatically be available in development.
However this is not true: encrypted secrets are currently only read in production.
How do you feel about changing this and reading encrypted secrets in every environment?
If you think about it, config/secrets.yml.enc
is already organized by environment (just like config/secrets.yml), so even if you read it in development, you will not be importing the keys under production:
, only those under shared:
and development:
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We skirted away from silently reading them in development just because users have a key and it's not our recommendation to use encrypted secrets as dev secrets storage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaspth Could you help me understand better what you recommend for development environment? I feel like my "development" keys are still just as worthy of keeping secret and using in the same way as production, no?
I tried finding something in edge guides or blog posts about upcoming rails 5.1 updates, and the PRs/issues around encrypted keys, but I don't find any advice on how to handle secret keys in my test / development environments, as it pertains to this new feature. In the past, I have been using Dotenv.
Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the moment we don't recommend using encrypted secrets in dev/test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, I got that. So, what is recommended? I need clarity on that.
Keep using Dotenv, or?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whatever works for you 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd say that the recommended way is to use the same method for storing/accessing secrets in either production or dev environment.
Once I have enabled encrypted secrets, I always need to open the editor to add a new key:
How do you feel about a new feature to programmatically add encrypted secrets without needing an editor?
The concept is very similar to the way in which Travis encrypts environment variables. |
Claudio, hard to do right now since we have the secrets file partitioned by
shared/envs. Also, it would leave the API_KEY in your history. Which might
not be a good idea.
…On Wed, Mar 22, 2017 at 2:39 PM, Claudio B. ***@***.***> wrote:
@kaspth <https://github.com/kaspth> @dhh <https://github.com/dhh>
Once I have enabled encrypted secrets, I always need to open the editor to
add a new key:
$ rails secrets:edit
[… add a new key in the editor, save and exit …]
# New secrets encrypted and saved.
How do you feel about a *new feature* to programmatically add encrypted
secrets without needing an editor?
$ rails secrets:add API_KEY=123
# New secret API_KEY encrypted and saved.
The concept is very similar to the way in which Travis encrypts
environment variables
<https://docs.travis-ci.com/user/environment-variables/#Encrypting-environment-variables>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#28038 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAKtSSH7TTfYwzkT_I59RLTPcIKOvrAks5roSSHgaJpZM4MDZRO>
.
|
I made it work. I'll see if I find the time to send a PR next week. :)
21 juni 2017 kl. 16:12 skrev Wojciech Wnętrzak <notifications@github.com>:
@morgoth commented on this pull request.
In railties/lib/rails/secrets.rb:
+ def handle_missing_key
+ raise MissingKeyError
+ end
+
+ def read_key_file
+ if File.exist?(key_path)
+ IO.binread(key_path).strip
+ end
+ end
+
+ def key_path
+ @root.join("config", "secrets.yml.key")
+ end
+
+ def path
+ @root.join("config", "secrets.yml.enc").to_s
@ce07c3 Are you still working on making it configurable? Can I help you somehow?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Just a small point, but could somebody add a few lines to the USAGE file (and, preferably, the Rails 5.1 release notes as that's what comes up when you search the web) detailing how you actually get the secrets OUT again? The release notes say
but they forget to mention where they're decrypted TO, and I suspect most people wanting to use this feature won't be too keen on having to spelunk through the 17 changed files on this PR… |
Also, how to fix files that are broken (YAML wise but not otherwise), since exceptions are raised...
18 juli 2017 kl. 14:57 skrev John Yeates <notifications@github.com>:
Just a small point, but could somebody add a few lines to the USAGE file (and, preferably, the Rails 5.1 release notes as that's what comes up when you search the web) detailing how you actually get the secrets OUT again?
The release notes say
Secrets will be decrypted in production, using a key stored either in the RAILS_MASTER_KEY environment variable, or in a key file.
but they forget to mention where they're decrypted TO, and I suspect most people wanting to use this feature won't be too keen on having to spelunk through the 17 changed files on this PR…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Please do investigate, then! |
I did - it loads as YAML for editing purposes which is not necessary. Perhaps that's just my setup.
18 juli 2017 kl. 16:08 skrev Kasper Timm Hansen <notifications@github.com>:
Please do investigate, then!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
It's an encrypted yaml file. Hence it's also yaml when it's decrypted, not just for "editing purposes". |
I've just updated to Rails 5.1, moved all my keys to |
So the encryption is specific to the data format? That makes no sense to me. I should be able to read it regardless of format. After all, Rails secrets allows me to save it in an invalid format.
… 18 juli 2017 kl. 18:26 skrev Kasper Timm Hansen ***@***.***>:
It's an encrypted yaml file. Hence it's also yaml when it's decrypted, not just for "editing purposes".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Fixes #25095
Adds
config/secrets.yml.enc
using Sekrets to allow storing production app secrets right in source control. When runningbin/rails secrets:edit
the file is decrypted and opened in a tmp file that then gets destroyed after the file is encrypted again and written back to its original location.Differs sligthly in the proposed first-call experience in that we generate a key instead of asking for it. Also supports storing the key in
config/secrets.yml.key
instead of always going theRAILS_MASTER_KEY
.Pending:
rails s
boot when no decryption key is present.