Here is what you will learn from this article.<\/p>\n
Application Secret Formats<\/h2>\n
It is standard security practice to isolate secrets from code, and the developers should not worry about where the secrets come from. Generally, applications expect secrets in a file format in a specific location.<\/p>\n
The common formats in which applications expect secrets are,<\/p>\n
- \n
- A config file (Text file with newline strings.)<\/li>\n
- Json\/Yaml File<\/li>\n
- Environemnt variables<\/li>\n<\/ol>\n
The format depends on the application and teams who design the CI\/CD process.<\/p>\n
For example, a java spring boot application property file can be
application.properties<\/code> with the following contents.<\/p>\nspring.datasource.url=jdbc:mysql:\/\/localhost:3306\/myDb\nspring.datasource.username=myuser\nspring.datasource.password=secretpassword\nspring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver<\/code><\/pre>\nWhen it comes to CI\/CD process, you cannot commit the secrets to the source code or even have it as Kubernetes secret object. Therefore, you need an efficient mechanism like a vault injector for applications to consume secrets securely.<\/p>\n
I\u2019ll also show you how to use the Vault injector and all vault agent configurations to place or inject the secrets into a pod from the vault server in the required formats for the application.<\/p>\n
What is Vault Agent Injector?<\/h2>\n
Vault Agent Injector is a controller (custom implementation) that can add sidecar and init containers<\/a> to kubernetes pods<\/a> in runtime.<\/p>\n
The job of the init container is to authenticate and retrieve secrets from the vault server using the pod service account<\/a> place them in a shared location (In memory volume) where the application container can access them.<\/p>\n
![\"\"](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-7-57.png\")
<\/figure>\nYou can use this implementation for kubernetes standalone pods, deployments<\/a>, Statefuset, and Kubernetes jobs<\/a>.<\/p>\n
How Does Vault Injector Work<\/h2>\n
The Vault Agent Injector is a Kubernetes Mutation Webhook Controller<\/a>.<\/p>\n
Meaning, it is a custom piece of code (controller) and a webhook that gets deployed in kubernetes that intercepts pod events like create and update to check if any agent-specific annotation is applied to the pod.<\/p>\n
For example, if a pod gets deployed with an annotation.”
vault.hashicorp.com\/agent-inject: 'true'<\/code>“, here is what happens.<\/p>\n- \n
- Custom
MutatingWebhookConfiguration<\/code> sends a webhook with all pod information to the injector controller deployment.<\/li>\n- Then the controller modifies the Pod spec in runtime to introduce a sidecar and init container agents to the actual pod specification.<\/li>\n
- Controller then returns the modifed object for object validation.<\/li>\n
- After validation the modified pod spec gets deloyed with a sidecar and init container.<\/li>\n<\/ol>\n
![\"\"](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-329.png\")
<\/figure>\nSo when the pod comes up, it will have the application container, a sidecar, and a init container.<\/p>\n
The init container is responsible for retrieving the secrets<\/strong>. In addition, a sidecar container is required if your application uses dynamic secrets<\/strong>. Dynamic secrets are secrets that are created on-demand with expiration time. The sidecar container ensures that the latest secrets are present inside the pod after every secret renewal.<\/p>\n
If your application does not use dynamic secrets, then the sidecar container is not required.<\/p>\n
You can read more about dynamic secrets here<\/a>.<\/p>\n
Prerequisites & Github Repository<\/h2>\n
To follow this tutorial, you need a running vault server on kubernetes.<\/p>\n
If you don’t have a vault server, follow the vault server setup guide on Kubernetes<\/a>, which explains all the Kubberenetes components involved in vault setup.<\/p>\n
Vault server and vault injector Kubernetes manifests are part of the following Github repository.<\/p>\n
https:\/\/github.com\/scriptcamp\/kubernetes-vault<\/code><\/pre>\nDeploy Vault Agent Injector<\/h2>\n
As explained earlier, vault inject is a controller code that listens to a mutation webhook. The injector controller is responsible for modifying the pod spec to add sidecar and init containers.<\/p>\n
You can deploy the injector with one command.<\/p>\n
cd into the vault-injector-manifests directory of the cloned repository and execute the following command.<\/p>\n
kubectl apply -f .<\/code><\/pre>\nIf you are a helm user, you can install both vault server and injector using a single helm chart.<\/p>\n
helm repo add hashicorp https:\/\/helm.releases.hashicorp.com\nhelm repo update\nhelm install vault hashicorp\/vault<\/code><\/pre>\nAnyways, I will go through all the manifest files. Go to the next section if you want to skip the explanation.<\/p>\n
All the vault injector components get deployed in the
default<\/code> namespace.<\/p>\nLet’s get started.<\/p>\n
Create
rbac.yaml<\/code> and copy the following manifest. It creates the following.<\/p>\n- \n
vault-injector<\/code> service account<\/li>\n- vault-agent-injector-clusterrole with persmissions to
mutatingwebhookconfigurations<\/code> as we will be deploying a mutating webhook.<\/li>\n<\/ol>\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: vault-agent-injector\n namespace: default\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\n\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRole\nmetadata:\n name: vault-agent-injector-clusterrole\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault \nrules:\n- apiGroups: [\"admissionregistration.k8s.io\"]\n resources: [\"mutatingwebhookconfigurations\"]\n verbs: \n - \"get\"\n - \"list\"\n - \"watch\"\n - \"patch\"\n\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n name: vault-agent-injector-binding\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault \nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: vault-agent-injector-clusterrole\nsubjects:\n- kind: ServiceAccount\n name: vault-agent-injector\n namespace: default<\/code><\/pre>\nExecute the RBAC manifest.<\/p>\n
kubectl apply -f rbac.yaml<\/code><\/pre>\nCreate
mutating-webhook.yaml<\/code> and copy the following manifest. This webhook is responsible for intercepting and sending pod events to the injector controller. It sends the webhook to the injector controller’s service endpoint on\/mutate<\/code> path.<\/p>\n---\napiVersion: admissionregistration.k8s.io\/v1\nkind: MutatingWebhookConfiguration\nmetadata:\n name: vault-agent-injector-cfg\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\nwebhooks:\n - name: vault.hashicorp.com\n sideEffects: None\n admissionReviewVersions:\n - \"v1beta1\"\n - \"v1\"\n clientConfig:\n service:\n name: vault-agent-injector-svc\n namespace: default\n path: \"\/mutate\"\n caBundle: \"\"\n rules:\n - operations: [\"CREATE\", \"UPDATE\"]\n apiGroups: [\"\"]\n apiVersions: [\"v1\"]\n resources: [\"pods\"]\n failurePolicy: Ignore<\/code><\/pre>\nCreate the webhook.<\/p>\n
kubectl apply -f mutating-webhook.yaml<\/code><\/pre>\nCreate a
deployment.yaml<\/code> using the following manifest file.<\/p>\n- \n
- This deployment assumes you have vault server running on default namespace with service endpoint
http:\/\/vault.default.svc:8200<\/code><\/li>\n- we are using the latest vault injector image
hashicorp\/vault-k8s:0.11.0<\/code> (At the time of writing this blog)<\/li>\n<\/ol>\n---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: vault-agent-injector\n namespace: default\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\n component: webhook\nspec:\n replicas: 1\n selector:\n matchLabels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\n component: webhook\n template:\n metadata:\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\n component: webhook\n spec:\n \n affinity:\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n - labelSelector:\n matchLabels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: \"vault\"\n component: webhook\n topologyKey: kubernetes.io\/hostname\n serviceAccountName: \"vault-agent-injector\"\n hostNetwork: false\n securityContext:\n runAsNonRoot: true\n runAsGroup: 1000\n runAsUser: 100\n containers:\n - name: sidecar-injector\n \n image: \"hashicorp\/vault-k8s:0.11.0\"\n imagePullPolicy: \"IfNotPresent\"\n securityContext:\n allowPrivilegeEscalation: false\n env:\n - name: AGENT_INJECT_LISTEN\n value: :8080\n - name: AGENT_INJECT_LOG_LEVEL\n value: info\n - name: AGENT_INJECT_VAULT_ADDR\n value: http:\/\/vault.default.svc:8200\n - name: AGENT_INJECT_VAULT_AUTH_PATH\n value: auth\/kubernetes\n - name: AGENT_INJECT_VAULT_IMAGE\n value: \"hashicorp\/vault:1.8.0\"\n - name: AGENT_INJECT_TLS_AUTO\n value: vault-agent-injector-cfg\n - name: AGENT_INJECT_TLS_AUTO_HOSTS\n value: vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc\n - name: AGENT_INJECT_LOG_FORMAT\n value: standard\n - name: AGENT_INJECT_REVOKE_ON_SHUTDOWN\n value: \"false\"\n - name: AGENT_INJECT_CPU_REQUEST\n value: \"250m\"\n - name: AGENT_INJECT_CPU_LIMIT\n value: \"500m\"\n - name: AGENT_INJECT_MEM_REQUEST\n value: \"64Mi\"\n - name: AGENT_INJECT_MEM_LIMIT\n value: \"128Mi\"\n - name: AGENT_INJECT_DEFAULT_TEMPLATE\n value: \"map\"\n - name: AGENT_INJECT_TEMPLATE_CONFIG_EXIT_ON_RETRY_FAILURE\n value: \"true\"\n \n args:\n - agent-inject\n - 2>&1\n livenessProbe:\n httpGet:\n path: \/health\/ready\n port: 8080\n scheme: HTTPS\n failureThreshold: 2\n initialDelaySeconds: 5\n periodSeconds: 2\n successThreshold: 1\n timeoutSeconds: 5\n readinessProbe:\n httpGet:\n path: \/health\/ready\n port: 8080\n scheme: HTTPS\n failureThreshold: 2\n initialDelaySeconds: 5\n periodSeconds: 2\n successThreshold: 1\n timeoutSeconds: 5<\/code><\/pre>\nCreate the deployment.<\/p>\n
kubectl apply -f deployment.yaml<\/code><\/pre>\nCreate a
service.yaml<\/code> with the following manifest. The mutation webhook will use this service endpoint.<\/p>\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: vault-agent-injector-svc\n namespace: default\n labels:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\nspec:\n ports:\n - name: https\n port: 443\n targetPort: 8080\n selector:\n app.kubernetes.io\/name: vault-agent-injector\n app.kubernetes.io\/instance: vault\n component: webhook<\/code><\/pre>\nCreate the service endpoint.<\/p>\n
kubectl apply -f service.yaml<\/code><\/pre>\nNow we have all the vault agent injector components installed.<\/p>\n
Create Vault Secrets & Policy<\/h2>\n
To demonstrate vault agent injector functionality, I will create the following.<\/p>\n
- \n
- A set of secrets using vault kv engine<\/li>\n
- Vault policy to read the secrets.<\/li>\n
- Enable vault kubernetes authentication.<\/li>\n
- Create a vault role to bind vault policy and kubernetes service account (vault-auth).<\/li>\n
- Create a vault-auth kubernetes service account to be used for vault server authentication.<\/li>\n<\/ol>\n
Note:<\/strong> I assume you have unsealed and loged in to vault using the vault token. If not please follow the vault server setup guide mentioned in the pre-requites and perform steps till vault login.<\/p><\/blockquote>\n
Exec into vault pod.<\/p>\n
kubectl exec -it vault-0 -- \/bin\/sh <\/code><\/pre>\nEnable the vault kv engine (key-value store).<\/p>\n
vault secrets enable -version=2 -path=\"kv\" kv<\/code><\/pre>\nCreate two secrets under
kv\/dev\/apps\/service01<\/code> path.appkey<\/code> &apptoken<\/code><\/p>\nvault kv put kv\/dev\/apps\/service01 appkey=\"zsdkfjhj4534\" apptoken=\"zsdasdfaskfjhj4534\" <\/code><\/pre>\nCreate a vault policy named svc-policy that allowed read operation on secrets under
kv\/data\/dev\/apps\/service01<\/code> path.<\/p>\nvault policy write svc-policy - <<EOH\npath \"kv\/data\/dev\/apps\/service01\" {\n capabilities = [\"read\"]\n}\nEOH<\/code><\/pre>\nEnable Kubernetes authentication.<\/p>\n
Note: <\/strong>If you have done this as part of the vault setup on kubernetes, you can ignore the following two commands.<\/p><\/blockquote>\n
vault auth enable kubernetes<\/code><\/pre>\nvault write auth\/kubernetes\/config \\\n token_reviewer_jwt=\"$(cat \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/token)\" \\\n kubernetes_host=\"https:\/\/$KUBERNETES_PORT_443_TCP_ADDR:443\" \\ kubernetes_ca_cert=@\/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt<\/code><\/pre>\nCreate a vault role named
webapp<\/code> that bindssvc-policy<\/code> andvault-auth<\/code> kubernetes service account.<\/p>\nvault write auth\/kubernetes\/role\/webapp \\\n bound_service_account_names=vault-auth \\\n bound_service_account_namespaces=default \\\n policies=svc-policy \\\n ttl=72h<\/code><\/pre>\nNow, exit the pod exec session and create a kubernetes service account named
vault-auth<\/code>. Vault agents will use this service account to authenticate to the vault server and retrieve the required secrets.<\/p>\nkubectl create serviceaccount vault-auth<\/code><\/pre>\nInjecting Secrets With Vault Agents<\/h2>\n
Here is what you should know about injecting secrets with vault agents.<\/p>\n
- \n
- The vault agent injector uses pod annotations to decide whether vault agents should be injected into pods.<\/li>\n
- There are many supported annotations. Please refer the official documenation<\/a> to know about all the supported annotations.<\/li>\n
- If you are not using dyanmic secrets, you can disable the sidecar agent using an annotation.<\/li>\n
- By default the vault agents write the secrets to
\/vault\/secrets\/<\/code> path. Which is a shared in memory pod volume.<\/li>\n<\/ol>\nFirst, we will look at a vault injector example using a simple pod definition with the vault agent annotations and see if the sidecar and init container gets injected into the pods.<\/p>\n
Save the following manifest as
pod.yaml<\/code>. It deploys an Nginx container.<\/p>\n---\napiVersion: v1\nkind: Pod\nmetadata:\n name: webapp\n namespace: default\n annotations:\n vault.hashicorp.com\/agent-inject: 'true'\n vault.hashicorp.com\/role: 'webapp'\n vault.hashicorp.com\/agent-inject-secret-config.txt: 'kv\/dev\/apps\/service01' \nspec:\n containers:\n - image: nginx:latest\n name: nginx\n serviceAccountName: vault-auth<\/code><\/pre>\nAs you can see, we have three annotations.<\/p>\n
- \n
vault.hashicorp.com\/agent-inject: 'true'<\/code> – This annotation enables sidecar and init containers.<\/li>\nvault.hashicorp.com\/role: 'webapp<\/code>‘ : Assigns the webapp role we created earlier.<\/li>\nvault.hashicorp.com\/agent-inject-secret-config.txt: 'kv\/dev\/apps\/service01'<\/code>– Vault path where the secrets reside.<\/li>\n<\/ol>\nLet’s deploy the pod. It creates a pod named webapp<\/strong>.<\/p>\n
kubectl apply -f pod.yaml<\/code><\/pre>\nIf you look at the injector pod logs, you will see a log showing the request sent from mutation webhook to vault injector service endpoint.<\/p>\n
handler: Request received: Method=POST URL=\/mutate?timeout=10s<\/code><\/pre>\nTo validate vault agents, let’s describe the pod and check the events.<\/p>\n
kubectl describe pod webapp<\/code><\/pre>\nIf you check the output events, you will notice three containers getting created.<\/p>\n
![\"\"](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-3-63.png\")
<\/figure>\nNow let’s check the init container logs and see if it could connect to the vault server and retrieve secrets.<\/p>\n
kubectl logs webapp -c vault-agent-init<\/code><\/pre>\nOn successful execution, you will see the following output. You can see a message saying the secrets rendered to
\/vault\/secrets\/config.txt<\/code><\/p>\n![\"\"](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-4-55.png\")
<\/figure>\nNow you might wonder how did
\/vault\/secrets\/config.txt<\/code> path come in to picture?<\/p>\nBy default, the vault agent writes all the secrets in \/vault\/secrets\/ path, an in-memory volume that all the containers can access in the pod.<\/p>\n
The filename config.txt came from the annotation vault.hashicorp.com\/agent-inject-secret-config.txt<\/strong>. Any name you give after -secret<\/strong> will be considered as the secret file name.<\/p>\n
Now, let’s exec into the Nginx app container and see if we can access the secret file.<\/p>\n
kubectl exec webapp -c nginx -- cat \/vault\/secrets\/config.txt<\/code><\/pre>\nYou should be getting the following output.<\/p>\n
data: map[appkey:zsdkfjhj4534 apptoken:zsdasdfaskfjhj4534]\nmetadata: map[created_time:2021-08-08T11:29:42.495211138Z deletion_time: destroyed:false version:1]<\/code><\/pre>\nAs you can see, we have the secrets in the file. This means the vault agent is working as expected.<\/p>\n
However, the output is not in a format that<\/strong> the application can use<\/strong>. As discussed in the introduction, every application expects the secret config in a specific format. A text file with newline strings, a JSON file, or a YAML file.<\/p>\n
To achieve this, the vault agent provides templating where you can render secrets in required formats. I will show all the methods with an Nginx deployment as an example.<\/p>\n
Vault Agent Template Example<\/h2>\n
You can use vault templates to render secrets in required formats. In this example, we will see how to use templates in deployment annotation.<\/p>\n
Save the following manifest as
deployment-template.yaml<\/code>. It is a simple nginx deployment with vault agent configs.<\/p>\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: nginx\n labels:\n app: nginx\nspec:\n selector:\n matchLabels:\n app: nginx\n replicas: 1\n template:\n metadata:\n annotations:\n vault.hashicorp.com\/agent-inject: 'true'\n vault.hashicorp.com\/role: 'webapp'\n vault.hashicorp.com\/agent-pre-populate-only: 'true'\n vault.hashicorp.com\/agent-inject-secret-config.txt: 'kv\/dev\/apps\/service01'\n vault.hashicorp.com\/agent-inject-template-config.txt: |\n {{ with secret \"kv\/dev\/apps\/service01\" }}\n [DEFAULT]\n LogLevel = DEBUG\n [DATABASE]\n Address=127.0.0.1\n Port=3306\n User={{ .Data.data.appkey }}\n Password={{ .Data.data.apptoken }}\n Database=app\n {{ end }}\n labels:\n app: nginx\n spec:\n serviceAccountName: vault-auth\n containers:\n - name: nginx\n image: nginx:latest<\/code><\/pre>\nHere if you see, I have added an annotation named vault.hashicorp.com\/agent-pre-populate-only: ‘true’<\/strong>. It disables the sidecar agent, and only init container gets deployed in the pod along with Nginx.<\/p>\n
vault.hashicorp.com\/agent-inject-template-config.txt<\/strong> annotation contains the secret file template with newline config.<\/p>\n
{{ with secret \"kv\/dev\/apps\/service01\" }}\n[DEFAULT]\nLogLevel = DEBUG\n[DATABASE]\nAddress=127.0.0.1\nPort=3306\nUser={{ .Data.data.appkey }}\nPassword={{ .Data.data.apptoken }}\nDatabase=app\n{{ end }}<\/code><\/pre>\nIt starts with the
with<\/code> block with the secret path and ends withend<\/code> keywords. All the substitution should happen between these blocks.<\/p>\nHere we are substituting the user and password values using
{{ .Data.data.appkey }}<\/code> &{{ .Data.data.apptoken }}<\/code>. Where appkey<\/strong> and apptoken<\/strong> are the vault secret keys. Vault agent substitutes the actual values in run-time.<\/p>\nLet’s create the deployment.<\/p>\n
kubectl apply -f deployment-template.yaml<\/code><\/pre>\nIf you exec into the Nginx pod created by the deployment, you will see the rendered config.txt file with user and password substituted with actual values from the vault server.<\/p>\n
kubectl exec nginx-7c897d75cb-mvjft -c nginx \"--\" cat \/vault\/secrets\/config.txt<\/code><\/pre>\n![\"\"](\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/03\/image-5-65.png\")
<\/figure>\nVault Agent Configmap example<\/h2>\n
You can add the secret file template as a configmap also. Along with the template, you need to add a few vault agent configs as well.<\/p>\n
Here is an example deployment with configmap that contains the vault agent template and configs.<\/p>\n
apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: nginx-configmap\n labels:\n app: nginx\nspec:\n selector:\n matchLabels:\n app: nginx\n replicas: 1\n template:\n metadata:\n annotations:\n vault.hashicorp.com\/agent-inject: 'true'\n vault.hashicorp.com\/agent-pre-populate-only: 'true'\n vault.hashicorp.com\/agent-configmap: 'my-configmap'\n labels:\n app: nginx\n spec:\n serviceAccountName: vault-auth\n containers:\n - name: nginx\n image: nginx:latest\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: my-configmap\ndata:\n config-init.hcl: |\n \"auto_auth\" = {\n \"method\" = {\n \"config\" = {\n \"role\" = \"webapp\"\n }\n \"type\" = \"kubernetes\"\n }\n\n \"sink\" = {\n \"config\" = {\n \"path\" = \"\/home\/vault\/.token\"\n }\n\n \"type\" = \"file\"\n }\n }\n\n \"exit_after_auth\" = true\n \"pid_file\" = \"\/home\/vault\/.pid\"\n\n \"template\" = {\n \"contents\" = \"{{ with secret \\\"kv\/dev\/apps\/service01\\\" }} \\n User={{ .Data.data.appkey }} \\n Password={{ .Data.data.apptoken }} \\n config=testing\\n env=dev\\n {{ end }}\"\n \"destination\" = \"\/vault\/secrets\/db-creds\"\n }\n\n \"vault\" = {\n \"address\" = \"http:\/\/vault.default.svc:8200\"\n }<\/code><\/pre>\nIn the configmap spec, you can see the filename as
config-init.hc<\/code>l. It is for vault init agent. If you are also using a sidecar agent, then you need to create one moreconfig.hcl<\/code> in the configmap with the same configuration asconfig-init.hcl<\/code><\/p>\nIf you notice, other than the template content<\/strong>, we have other vault agent configs like vault address<\/strong>, role<\/strong>, type,<\/strong> etc., as part of the configmap.<\/p>\n
Also, we are using the same template content we used before, but here, we are using
\\n<\/code><\/strong> for next line.<\/p>\nYou can create the deployment and check the secret file.<\/p>\n
Vault Agent Environment Variable Example<\/h2>\n
There is no direct way using vault agents t make the secrets available as environment variables. However, there is a workaround to do it.<\/p>\n
We can use the regular vault template to create a file with an export command for all the environment variables. Then in the deployment, we can use the command arguments to source this file so that all the export commands get executed. The secrets will be available as environments variable for the application.<\/p>\n
Create a file named
deployment-env.yaml<\/code> using the following manifest. I used a basic ubuntu image for demonstration purposes.<\/p>\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: nginx\n labels:\n app: nginx\nspec:\n selector:\n matchLabels:\n app: nginx\n replicas: 1\n template:\n metadata:\n annotations:\n vault.hashicorp.com\/agent-inject: 'true'\n vault.hashicorp.com\/role: 'webapp'\n vault.hashicorp.com\/agent-pre-populate-only: 'true'\n vault.hashicorp.com\/agent-inject-secret-database-config: 'kv\/dev\/apps\/service01'\n vault.hashicorp.com\/agent-inject-template-database-config: |\n {{ with secret \"kv\/dev\/apps\/service01\" -}}\n export appkey=\"{{ .Data.data.appkey }}\"\n export apptoken=\"{{ .Data.data.apptoken }}\"\n {{- end }}\n labels:\n app: nginx\n spec:\n serviceAccountName: vault-auth\n containers:\n - name: nginx\n image: ubuntu:latest\n command: [\"\/bin\/bash\"]\n args:\n ['-c', 'source \/vault\/secrets\/database-config && env > \/vault\/secrets\/test && tail -f \/dev\/null']\n<\/code><\/pre>\nIf the above deployments,<\/p>\n
- \n
- In the annotation we have the template with a export commands for appkey<\/strong> and apptoken<\/strong>. You can have multiple export commands like this<\/li>\n
- In command arguments, we are sourcing the \/vault\/secrets\/database-config<\/li>\n
- Ther we are writing the env variables to
\/vault\/secrets\/test<\/code> for validation. This is just for testing this setup beacuae if you exec it a different session those variable will not be available.<\/li>\n- The tail -f \/dev\/null keeps the container running as we are using the base ubuntu image.<\/li>\n<\/ol>\n
Create the deployment.<\/p>\n
kubectl apply -f deployment-env.yaml<\/code><\/pre>\nNow, if you check the
\/vault\/secrets\/test<\/code> file, you will see both appkey<\/strong> and apptoken<\/strong> as environment variables. Replacenginx-767dbbd58b-bsxks<\/code> with your pod name.<\/p>\nkubectl exec -it nginx-767dbbd58b-bsxks -c nginx -- cat \/vault\/secrets\/test<\/code><\/pre>\nVault Injector Troubleshooting & Issues<\/h2>\n
Following are the issues I have faced during the vault injector setup.<\/p>\n
- \n
- Vault agent Template works only after a first pod restart:<\/strong>– The agent was not able to render the template when the pod gets deployed. However, when I restarted the pod, it worked. I suspect it to be a node resource issue as I was using small k8 nodes. Later when i increse the the node resources, i never got the error.<\/li>\n
- Templated rendered empty secret values: <\/strong>In vault documentation, for rendering secrets, the systax is given as
.Data.key<\/code>. It didnt work. Somewhere if found that we have to add data to the syntax. eg,.Data.data.apptoken<\/code><\/li>\n- error authenticating: error=”context deadline exceeded” : <\/strong> This happends when the vault agent is not able to connect to the vault server or the service account doest have persmissions to read the secrets. Check the vault URL, policy, role and service account mapping.<\/li>\n
- Private GKE connectivity issues: <\/strong>In few forums I found people discussing about connectivity issues (Timeout errors in MutatingWebhookConfiguration) from master to injector controller pods due to firewall issues. I can be solved by adding a custom firewall rule from master to nodes on port 8080. Check out the discussion thread here<\/a>.<\/li>\n<\/ol>\n
Conclusion<\/h2>\n
Using Kubernetes with a vault for secret management, a vault injector is a great way to introduce secrets to pods.<\/p>\n
This way application doesn’t need to be aware of the secret management system. It just has to consume the secrets from a designated file path.<\/p>\n
Also, the production vault server should be highly available. Vault server unavailability could result in application failure if there is a pod restart or scaling activity.<\/p>\n
Next, in this vault series, we will look at solutions like Bank vault and Kubernetes external secrets.<\/p>\n
You may be using a different approach in managing secrets or using solutions like Hashicorp vault.<\/p>\n
Either way, leave a comment.<\/p>\n
\n - Templated rendered empty secret values: <\/strong>In vault documentation, for rendering secrets, the systax is given as
- we are using the latest vault injector image
- vault-agent-injector-clusterrole with persmissions to
- Custom