It seems that every week there’s a security breach that resolves around the fact that secrets were stored poorly. As a responsible and security conscious engineer, you likely want to avoid the professional and reputational hazard associated with a sloppy security posture. You’ve likely heard of HashiCorp and you might have heard of their secrets management solution, Vault.

Why?

Secrets storage invariably costs companies anytime it is done incorrectly. It’s not a matter of “if” but “when”. Whether it’s because a disgruntled employee copied all of the secrets before being terminated or a breach into your GitHub account, exposing those secrets you thought were safe in that private repository. Proper secret storage is a good thing to master whether you are a startup or a Fortune 500 company. Additionally, a system that helps make secret rotation easier will save you headaches down the road.

What not do to do

While briefly illustrated above, I’ll dive into a few of the ways that people tend to store secrets and why using Vault is a better option.

Committing secrets in plain text in Git

This is likely something you know better than to do, but you’d be surprised how often it happens due to how easy it is (and it’s fairly easy to mistakenly do). It’s a very, very bad idea to do this given that anyone that can read the repository can read your secrets. Having a repository be “Private” isn’t enough here as all it takes is a hack of your GitHub account and your secrets are now also exposed. See this Dzone article for a more comprehensive writeup on why this is such a bad idea. If you’d like to prevent the accidental committing of secrets, tools like git-secrets can help with that.

Storing in Docker images

Similar to above, while your Docker image may be on a private registry - anyone that has access to the registry can access your secrets. Doing this also violates principle three of the twelve-factor app principles — Config and will make image re-use across environments particularly complicated and annoying.

Storing on your desktop and manually entered when deploying

While this sounds reasonable and sound, after all only you know the secret as it’s locked up on your personal workstation, it’s got it flaws as well. Shared with the above, the privacy, and security of your personal workstation is a false sense of security, many can and are hacked. In addition to the false sense of security, this method severely limits future automation capabilities. This option could be sane if you’re the one and only engineer on a product that only you will ever work on and you have a company size of one.

Why Vault then?

There are plenty of options available for secrets management and storage these days. HashiCorp Vault is an excellent option with a very security minded posture. It’s been around since April 2015 and is running in many production deployments. The CNCF Radar also puts Vault in the “Adopt” category, further signaling the maturity.

Vault additionally supports multiple backend and authentication methods. Whether your application is deployed in a datacenter or every major cloud, there is little chance you’ll find yourself missing something you need. If you do though, Vault is Open Source and supports a plugin model.

Working With Vault

Ultimately, Vault’s going to provide the most value once it’s in your customer-facing environment or Production. Before you get there though, there’s a few other ways to get quick hands on learning.

Local development

While secrets management has many angles and complexity to it, local development with Vault is easy. On top of being easy, it lets you iterate quickly and get a solid feel for how it will work without having to set up the datastore that you’d use in production.

Once you’ve installed Vault, you can now start a local server in “Dev” mode via vault server -dev. In dev mode, Vault stores its secrets in memory and listens on localhost. It’s secure but certainly not how you want to run in Production — again this is just for getting hands-on experience while requiring very little setup time.

Interacting with your local Vault

The vault binary is the server and the API client. I find using the vault client the easiest way of interacting with a running Vault server. If you’re curious how the requests are sent to the API, providing the flag -output-curl-string  when running the Vault client outputs the cURL equivalent. For example:

$ vault kv put -output-curl-string secret/hello foo=world
curl -X PUT -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" -d '{"data":{"foo":"world"},"options":{}}' http://127.0.0.1:8200/v1/secret/data/hello

As you could guess from the cURL response above, Vault offers an HTTP API. Beyond using a simple cURL, there are many Vault libraries for the major languages.

Key Value is what you want for secrets

As you get started and before you enable a different secrets backend, storing and retrieving secrets will typically be done through the key/value storage or kv. As a quick example, here’s how you write a secret and then retrieve it:

$ vault kv put secret/hello foo=world
Key              Value
---              -----
created_time     2021-05-01T18:50:48.977501Z
deletion_time    n/a
destroyed        false
version          1

$ vault kv get  secret/hello
====== Metadata ======
Key              Value
---              -----
created_time     2021-05-01T18:50:48.977501Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
foo    world

You can store multiple key/value pairs under a path and the kv backend supports multiple versions as well. Storing a second value under secret/hello illustrates this via the version metadata.

❯ vault kv put secret/hello foo=jon
Key              Value
---              -----
created_time     2021-05-01T18:52:33.488721Z
deletion_time    n/a
destroyed        false
version          2

To learn more about what you can do with the kv secrets engine, see the official Vault docs.

Moving towards Production

There are many options for how you can run Vault in production. My personal experience is running Vault within AWS on a classic stack of EC2/Autoscaling Group/ALB. How this is all automated and configured is deeper than I intend on covering here.

HCP Vault

While I’ve not used it personally, HashiCorp has a cloud platform that offers a hosted Vault offering. If I were starting without a secret management solution today, I’d seriously consider this option. The development tier is comparable, in terms of cost, to a 2 CPU / 4 gig instance in AWS (T4G.medium) without factoring in the costs of data transfer. As a bonus, you don’t need to do any of the infrastructure automation or hardening when going this route.

NOTE: As of May 2021, HCP Vault is limited only to running within AWS and specific regions. See the HCP Vault pricing page for the most up-to-date information.

Other Considerations

There are a few things I’d recommend considering as you experiment and plan to use Vault:

  1. What happens when it’s down? Since Vault’s hopefully the single source of truth for your secrets, what happens when it isn’t responding? A lot of this answer depends on how or when applications fetch secrets. It’s strongly recommended running Vault in a high-availability configuration in production, e.g., do not run the local server and expect to have a good time.
  2. Who can do what? Vault has a policy engine that allows fine-grained control to the various secrets and capabilities within it. This is an area where I’d recommend reading some best practices guides within the HashiCorp Learn site. In addition to the policies used to limit access to things like reading & writing secrets, also consider who can directly access the Vault server. Great Vault policies don’t last very long if everyone in the company can SSH into the server.
  3. What secret and auth engines do you need? I’d recommend starting as small as required here. Either of these aren’t complicated to set up but having too many concurrently enabled can make things needlessly complex, especially when starting new.

Next Steps

Hopefully, after reading this, you feel like you’ve got a handle on how Vault can improve your secret management posture. For other resources, I’ve found the HashiCorp Learn site an excellent place to start. The Vault documentation is also quite extensive and can provide information on the various storage, authentication, and plugin options.