Vault secrets as environment variables

If you are a DevSecOps engineer working with Vault to provide a secure way to handle secrets and you are reading this post, then there are high chances that you have tried different solutions out there to inject environment variables without success. Do not worry. In this post I’ll show you how to beat this road blocker.

Secrets-as-env & Secrets-as-file

It is important to highlight that there are mostly two types of secrets; those that are read from environment variables (secrets-as-env) and those that are read from a secrets files (secrets-as-file). Vault works particularly fine with secrets-as-file, in which the agent-injector adds a vault-agent container inside your pod in order to load the secrets under the /vault/secrets/<secret-file> path.

The problem is not with secrets-as-file, the problem is mostly with secrets-as-env, since Vault does not provide an easy use case to inject secrets-as-env. If you read their documentation they will suggest to use template injection in which it “partially works” and it is kind of dark the way it is being handled. A lot of requests has been made in the GitHub and the community place for Vault without any release date. And I’ve tried several approaches of their documentation without success.

The good thing is that there are some third party (Open Source) Solutions that provides a full customized vault implementation in order to fix some problems. The tool that we are going to work in this post is called Bank Vault by BanzaiCloud, BUT we are only going to use their Mutating webhook to handle the secrets-as-env.

Solution overview

As mentioned before there are several third parties tools such as Patroni or Tozny that only allows read secrets as environment variables, and there are a bunch of other tools that can read secrets from a file (secrets-as-file).

Imagine that we are using kubernetes + Helm charts to deploy our solution. We will have a namespace called vault-infra which will contain the Official Vault Helm in one pod, and within the same namespace the Bank Vault webhook. The idea is to use the Vault Helm solution to handle everything related to secrets and the webhook will connect to that vault via service address to provide the extra capabilities to read environment variables.

So if you have an application in another namespace such as Redis that only reads secrets-as-file, you will be using the regular vault annotations to achieve it. If there is a case in which a third party only works with environment variables, you will connect that app to the Bank Vault webhook using their specific annotations.

In this way we can extend the capabilities of vault without affecting the (possible) already deployed vault infrastructure.

Figure 1. Solution overview

Bank Vault — Webhook annotations

As mentioned before, the annotations for Bank Vault are specific to Banzai, meaning that we need to add different pod annotations in order to work with the webhook. But those are not too different. Let’s take a look.

In your pod annotation you will need the following annotations.

• The vault-addr: that defines the service address of your Vaul Helm
• The vault-role: Which is the service account / Vault rol name
• The skip-verify: To avoid the TLS verification
• The vault-path: Which is the secret location path in your vault

vault.security.banzaicloud.io/vault-addr: "https://vaultname:443"
vault.security.banzaicloud.io/vault-role: "patroni"
vault.security.banzaicloud.io/vault-skip-verify: "true"
vault.security.banzaicloud.io/vault-path: "secret/patroni"

And that’s it! Those are the only annotations that you need.

Bank Vault — Loading secrets

In order to read the specific secrets we are going to use the following “path” “vault:SECRET_PATH#SECRET_KEY” and it will look like this.

env:
- name: DB_PASSWORD
  value: vault:secret/data/db#DB_SECRET_KEY

This means that we are going to use the secret path and then specify the KEY that will hold the secret for our environment variable. As you can see this is really easy to read.

How does it work?

I highly recommend checking this official article, but basically it is really secure since it does not store any values in etcd, or kubernetes secrets. It copies a binary called /vault/vault-env which is executed before the entry point of your pod, allowing to translate the path above to your specific secret.

If you check the environment variables of your running container you will see it just shows the path, but if you run the following command

kubectl exec -it patroni — /bin/sh -c "/vault/vault-env env"

You should be able to see your translated environment variables.

That’s it for now. And I hope this article was helpful.