TL;DR
If you already know why putting credentials in a repo is a really bad idea and want a solid technical solution for keeping sensitive configuration data both manageable and secure, jump ahead to “Encrypted YAML.”
Still here? Let me tell you about a real security related problem that providers of web solutions will have to deal with and how we solved it. Be forewarned, this will get technical in the implementation, but first some background.
The problem: how to keep sensitive configs safe?
At Highland, we depend on GitHub to manage our software project codebases. We also take security seriously and do not store unencrypted sensitive data either in code or configurations managed by GitHub or any other SCM/VCS option.
We use a layered approach to help keep things secure. Client code lives in private repositories. Some codebases have additional access restriction requirements and we further restrict access to members of an authorized team. Sensitive data such as credentials generally live in config files that are kept out of the application code repos altogether. To deploy an application or update, we need to deploy both code and configurations.
We use a deployment service to deploy code and configurations to servers. We try to keep deployments as “hands-off” as possible. The deployment service can be given permission to deploy code directly from GitHub. It also must deploy configuration files as the application needs both code and configs. The deployment server can deploy config files from static files that have been manually loaded into the deployment service via a dashboard. This is not ideal, as it requires “hands-on” work to manage configs.
The config files and the sensitive data they contain create two problems:
- We don’t trust sensitive data in GitHub or any other SCM/VCS repo.
- Loading config files into a deployment service manually is an opportunity for error.
We need to solve both problems: managing the sensitive content of config files and making them available to the deployment service in a manner that doesn’t sacrifice quality assurance.
The rest of this article solves the first problem. A followup article will solve the second.
Managing Many Configs
As noted previously, we don’t trust sensitive information in GitHub or any other SCM/VCS repos. At the same time, we want the advantages of source code controls that allow us to safely manage changes to configs by using code review, merge, revert, etc. If we could somehow protect the sensitive information, we would increase the quality of the change control process through the use of SCM/VCS.
How can we encrypt the secure data in the configurations and safely use SCM/VCS?
Encrypted YAML
We manage server build and provisioning with Puppet. Through this work, we became familiarized with a similar configuration issue containing sensitive data. Configurations in Puppet are typically abstracted in Hiera yaml files. In turn, these provide values to class parameters that set the attributes for the various Puppet resources that are applied during server build and provisioning. Hiera files are mostly simple key-value pairs that specify things like filenames, file content, system packages, user accounts, services, scheduled jobs, and much more. Occasionally sensitive data is required, perhaps to configure a service that requires credentials. An example might be a connection to an email server or service provider—this type of configuration will require sensitive authentication credentials.
The sensitive Puppet Hiera configs problem has already been solved with encrypted yaml for Hiera. Encrypted yaml, or eyaml, provides a secure key-based facility to encrypt the sensitive data in yaml config files used by Hiera. This tool was designed specifically for managing yaml files with sensitive data. Fortunately, it also works on other config file formats such as dotenv, ini, and generally any text format.
Eyaml was developed for the Puppet ecosystem but can be used independently to encrypt sensitive data in text files. Many other tools provide encryption and decryption of text files but eyaml enables you to choose exactly which parts should be encrypted. When used to encrypt sensitive sections of a file in a Git repo, you can still use Git diff to easily see where there were changes. The encrypted sensitive data obscures what changed, but there is a simple way to decrypt the diff and see exact changes as long as you have the private key.
Install
Install the hiera-eyaml gem. This provides the eyaml tool.
$ sudo gem install hiera-eyaml
Create public and private keys with default names in the default location. (~/keys)
$ eyaml createkeys
Optionally create an eyaml config. This tells eyaml where to find your keys. You can also specify the paths to your keys on the command line but the config file is more convenient.
Config files will be read first from ~/.eyaml/config.yaml, then from /etc/eyaml/config.yaml and finally by anything referenced in the EYAML_CONFIG environment variable
For example, if your keys are in ~/keys/eyaml/, create ~/.eyaml/config.yaml with the following content. Replace /path/to/home with the path to your home directory. Note that eyaml does not expand ~ to your home directory, so spell it out in full.
---
pkcs7_private_key: '/path/to/home/keys/eyaml/private_key.pkcs7.pem'
pkcs7_public_key: '/path/to/home/keys/eyaml/public_key.pkcs7.pem'
Then set the environment EYAML_CONFIG to the path to your config.yaml. Again, replace /path/to/home with the path to your home directory.
$ export EYAML_CONFIG=/path/to/home/.eyaml/config.yaml
Example Workflow
We will use a conventional Git repo to manage our encrypted config files and we’ll use eyaml to edit the encrypted config. Given a plain text .env file named example.env, let’s encrypt the values for SECRET, MEANING_OF_LIFE, and PI.
SECRET="a very big and important secret"
MEANING_OF_LIFE=42
PI=3.14159
Create an empty file example.env.encrypted and begin an edit with eyaml.
$ touch example.env.encrypted; eyaml edit example.env.encrypted
The eyaml editor, on every open, will insert 11 lines of comments also known as the “preamble” at the top of your file. The preamble will remind you how to edit encrypted sections of your file and will be removed when you save your file.
Paste or type the original content of example.env into the editor after the preamble. Note the last line of the preamble includes a prototype of an encrypted value. In place of a value to be encrypted, insert the prototype leaving the value within the square brackets. For example, replace the value of PI “3.14159” with “DEC::PKCS7[3.14159]!”. Do this for the values of SECRET, MEANING_OF_LIFE, and PI. Your edit window, ignoring the preamble, should look like this.
SECRET=DEC::PKCS7["a very big and important secret"]!
MEANING_OF_LIFE=DEC::PKCS7[42]!
PI=DEC::PKCS7[3.14159]!
Save and close. Note this deletes the preamble. The contents of example.env.encrypted should look similar to this.
Note that the keys are still obvious while the values for SECRET, MEANING_OF_LIFE, and PI are obscured.
Commit example.env.encrypted. Reopen for edit with eyaml.
$ eyaml edit example.env.encrypted
Change the values for SECRET and MEANING_OF_LIFE.
SECRET=DEC(1)::PKCS7["a little secret"]!
MEANING_OF_LIFE=DEC(3)::PKCS7[24]!
PI=DEC(5)::PKCS7[3.14159]!
Save and close. Now run Git diff. Note that while it’s clear that the values of SECRET and MEANING_OF_LIFE changed but it’s not clear what the new values are.
Now try piping the output of diff into eyaml decrypt from stdin.
$ git diff example.env.encrypted | eyaml decrypt --stdindiff --git a/example.env.encrypted b/example.env.encrypted
index 1a31e8e..00e2f99 100644
--- a/example.env.encrypted
+++ b/example.env.encrypted
@@ -1,3 +1,3 @@
-SECRET="a very big and important secret"
-MEANING_OF_LIFE=42
+SECRET="a little secret"
+MEANING_OF_LIFE=24
PI=3.14159
Note that you can clearly see the changed values.
Keep These Caveats in Mind
Eyaml was designed to enable encryption of secrets in yaml files. Generally, there are no problems encrypting substrings or an entire line of text in other formats or just plain text files. However, do not attempt to encrypt nothing by using the empty prototype “DEC::PKCS7[]!” and do not attempt to encrypt text that spans more than one line. Both of these cases cause eyaml to assume yaml line continuation—probably not what you want.
Review
- Use eyaml to encrypt, decrypt, and edit config files.
- Replace the value to be encrypted with the prototype leaving the value within the square brackets.
- Safely commit encrypted config files after all sensitive data has been encrypted.
- DO NOT commit config files with unencrypted sensitive data.
- DO NOT commit your keys in your repo.
- You can use different keys for different repos when needed.
Check back here for a follow-up article with solutions for how to make config files available to deployment services in a manner that doesn’t sacrifice quality assurance.
Do you have more questions about config security? Send Stu, Highland’s DevOps Engineer, an email.
