This adds encrypted attributes to Active Record models. This is an extraction from HEY.

You can check the guide included with this PR to learn more about functionality and features.

Basic usage

Encrypted attributes are declared at the model level. These are regular active record attributes backed by a column with the same name. The library will transparently encrypt these attributes before saving them into the database and will decrypt them when retrieving their values.

class Person < ApplicationRecord
  encrypts :name
end

By default, it will encrypt all the data using AES-GCM with a 256-bits key and a non-deterministic approach: encrypting the same text twice will result in different ciphertexts.

You can configure it to use a deterministic encryption approach instead. In this case, the initialization vector will be derived from the text to encrypt instead of being random, which enables queries against encrypted columns:

class Person < ApplicationRecord
  encrypts :name
  encrypts :email_address, deterministic: true
end

# Person.find_by(name: "jorge") # doesn't work
# Person.find_by(email_address: "jorge@basecamp.com") # works

The library includes many additional features.

• Encrypt action text attributes
• Support ignoring case
• Work with encrypted and unencrypted data simultaneously.
• Configurable encryption schemes
• Support old encryption schemes while migrating existing encrypted data to a new scheme
• Compression
• Fixtures support
• Filter parameters based on encrypted attributes (to remove sensitive params from logs).

Please check the guide in this PR to learn more about setup and usage.

Implementation overview

EncryptableRecord is the concern that makes ActiveRecord::Base records encryptable. It defines .encrypts and the record-level API.

When declaring an attribute as encrypted, internally, it uses a custom active record attribute type (EncryptedAttributeType). This performs the encryption and decryption when serializing and deserializing attribute values. It interacts with the encryption system to do this. The main components of the encryption system are:

• Encryptor: the internal API for encrypting and decrypting data. It interacts with a KeyProvider to build encrypted messages and deal with their serialization. The encryption/decryption itself is done by the Cipher and the serialization by MessageSerializer.
• Cipher the encryption algorithm (Aes 256 GCM)
• KeyProvider serves encryption and decryption keys.
• MessageSerializer: in charge of serializing and deserializing encrypted payloads (Message).

While most users won't need to do this, these components can be customized via config settings like:

config.active_record.encryption.encryptor = MyEncryptor.new

They can also be overridden just for a given block or on a per-attribute basis:

ActiveRecord::Encryption.with_encryption_context(encryptor: MyEncryptor.new) do
  ...
end

class Article
  encrypts :title, encryptor: MyEncryptor.new
end

Queries with deterministic attributes

When querying data encrypted deterministically, a tricky problem arises when you combine encrypted and unencrypted data. Or data encrypted with different schemes (e.g: because you decide to change encryption properties for a given attribute).

To solve this, we extend Active Record to modify queries automatically. So, for example, this query:

Contact.find_by(email_address: "jorge@hey.com")

Becomes this under the hood when you enable support for unencrypted data:

Contact.find_by(email_address: [ "jorge@hey.com", "<encrypted jorge@hey.com>" ])

The implementation of this system is in ActiveRecord::Encryption::ExtendedDeterministicQueries. This system has worked flawlessly for HEY since we launched, but the implementation feels a bit brittle and experimental. Because of that, I’ve put it behind a config flag encryption.extend_queries that is false by default.

I’d be happy to get advice (and help!) to improve this implementation.

Message structure

Each encrypted value is a Message that stores:

• The encrypted payload
• A public (unencrypted) list of headers (Properties)

Example of headers:

• Encryption-related information such as initialization vectors.
• Specific encryption system meta information, such as if the payload is compressed or not.
• Key-related information such as encrypted keys (for envelope encryption) or key references.

Serialization

Messages are serialized with a JSON-based serializer (MessageSerializer). The serializer is configurable via active_record.encryption.message_serializer.

It's important to use safe serializers that can't deserialize arbitrary objects. A common supported scenario is encrypting existing unencrypted data. An attacker can leverage this to enter a tampered payload before encryption takes place and perform RCE attacks. This means custom serializers should avoid Marshal, YAML.load (use YAML.safe_load instead) or JSON.load (use JSON.parse instead).

Performance

Time

OpenSSL AES encryption/decryption is fast, with submillisecond times for typical payloads (<700kb or less) or just a few milliseconds for multi-MB payloads.

As a result, and in our experience, the impact of adding this encryption system to an app is negligible in terms of performance. Even fast database queries are slower by one order of magnitude or more, so the overhead of encryption gets diluted when combined (not to mention if you add view-rendering to the mix).

This PR include some automated performance tests where a baseline without encryption gets compared with similar code using encryption for different key providers. You will see how the impact on tests is around 30% when encrypting data, but this is because there is no database latency in tests, so the relative impact is magnified. The same test shows identical performance when running in a more realistic environment. These tests are useful for optimization work and to prevent performance regressions.

Storage

Encryption adds storage overhead because:

• Values are encoded in Base64. This enables storing and transmitting binary data from ciphertexts and encryption metadata without encoding issues.
• It stores encryption metadata along with the encrypted payload. For example, initialization vectors, auth tags (AES integrity), other keys when using envelope encryption, etc.

The worst-case overload we have measured is around 250 bytes when the built-in envelope encryption key provider is used. For medium and large text columns this overload is negligible, but for string columns of 255 bytes, you should increase their limit accordingly (510 is recommended).

To help to mitigate the encryption overload, the library uses compression automatically when the payload exceeds a certain threshold (140 bytes). Compression is done before encrypting the content (compressing after wouldn't compress much due to randomness). We have observed an actual reduction in space for larger text columns.

Using compression opens a vulnerability where an attacker can enter highly compressible payloads that can cause trouble when uncompressed. To prevent this, the library adds a length validation to the encrypted column based on the database column limit. This can be disabled with the option active_record.encryption.config.validate_column_size.

There is a test StoragePerformanceTest measuring the storage overload in different scenarios. Notice that compression is not really showing much difference there due to using randomly-generated payloads. The test is meant to catch regressions and to help with storage optimization work.

It would be easy to make compression a config option, as a future improvement. Without compression you can expect a consistent 33% overhead due to the Base64 encoding.

Security audit

A security firm reviewed this library before launching HEY. Thanks to it, we could fix an important flaw regarding our deterministic encryption approach in the initial version.

Credits

• Many folks at Basecamp helped to shape this with their feedback. Special mention to @jeremy who contributed some key design decisions while we were trying to figure this out, and to @rosa and @georgeclaghorn for their help reviewing things and discussing approaches.
• I checked many similar libraries while working on this one, and I learned good things from all of them. I want to make a special mention to @reidmorrison and symmetric-encryption for showing how encryption was a great scenario for using a custom attribute. Other gems I checked and was inspired by include attr_encrypted, lockbox and vault-rails.

Pending

• Add configuration options to guide
• Make the build happy
• Move performance tests out of the main build (to a rake task)
• Address feedback by Rafael
• Test upstreamed version in HEY after all the changes