Day Two Cloud 090: Winging it with HashiCorp Vault

This episode of Day Two Cloud featured yours truly talking Ethan through the world of HashiCorp Vault. It sprung from some conversations we’d had earlier about how he could better secure the secrets and API keys he uses in his Python scripts. We had a last minute guest cancellation, so we decided to wing it and do a whole episode without a script. I think it went well! In part because Ethan is an excellent interviewer, and also because all the Vault knowledge was fresh in my head. There is one thing that bothers me about the episode, and it’s too late to change it now. It’s about the master key.

All secure systems have to deal with a chicken and egg problem. You want to have a secure system holding your secrets. You need to apply encryption to that system, and encryption relies on keys. Now where and how do you store the root encryption key to secure the data stored by the system? You can’t store it on the system itself. Vault’s solution to the problem is to store the root encryption keys externally from Vault using one of two methods. I’ll come to those in a moment. But first, let’s talk about the keys at play:

• Unseal key – Key used to decrypt the Master key housed in backend storage. Stored as key shares or in an Cloud/HSM service.
• Master key – Used to decrypt the encryption keyring. Encrypted with the Unseal key and housed in backend storage.
• Encryption key – Actually a keyring (one or more keys) and used to encrypt Vault data in backend storage. Encrypted using the Master key.

The relationship looks a bit like this:

Unseal key –(encrypts)–> Master key –(encrypts)–> Encryption keys –(encrypts)–> Vault data

In the episode, I kept talking about the Master key and how it was broken into key shares or stored in an external Cloud/HSM. That’s not the case. The Unseal key is the key broken up into shares or stored in an external Cloud/HSM. It’s not a huge oversight, but I did muddy the waters on an already confusing topic.

To set the record straight, the Master key is stored in an encrypted state on the Vault storage backend. It is encrypted using the Unseal key.

The Unseal key is never stored in Vault. It is either stored on an external service Cloud/HSM or broken into shares using Shamir key sharing.

Whew! It feels good to get that off my chest. Listening to the episode I kept wishing I could reach back in time and remind past-Ned about the Unseal key.

While the Shamir key sharing process may seem fairly straightforward, you might wonder how the external Cloud/HSM works. The Unseal key is housed in the external service. When Vault wants to unseal, it submits the encrypted Master key to the external service to perform an unwrapping operation. The external service will unwrap (decrypt) the master key and send the unencrypted master key back to Vault. This means that your master key is transmitted unencrypted from the external service back to Vault. At least that is how I interpret it. In the Shamir key shares, the master key is never transmitted unencrypted, since the decryption operation happens inside the Vault barrier.

Of course all communication with the external Cloud/HSM is happening over a TLS secured channel, so in theory the transmission of the master key is safe. Still, it’s a little unnerving that such a valuable piece of data is relying solely on TLS. I’m not sure how else they could do it though!

Regardless, I think that is the only major gaffe of the episode. Otherwise, it is an accurate look into what Vault is and how it works. If you want to dive deeper, definitely check out my Pluralsight courses, my YouTube videos on Vault, or the Vault Certification Guide.