Automating Kubernetes Secrets with Vault and the Vault Secrets Operator (VSO) on AWS
Managing secrets in Kubernetes can be a hassle. We want a secure, automated, and easy way to get secrets from HashiCorp Vault into our Kubernetes workloads.
This guide will walk you through a complete setup using:
- HashiCorp Vault: Deployed in Kubernetes and configured to use AWS KMS for auto-unseal. This avoids the manual process of unsealing Vault every time it restarts.
- Vault Secrets Operator (VSO): This operator will connect to Vault, authenticate using Kubernetes service accounts, and automatically create native Kubernetes
Secretobjects based on what’s stored in Vault. - Your Application: Your deployment can then consume these native Kubernetes secrets without needing to know anything about Vault.
Let’s get started.
Part 1: Deploying HashiCorp Vault with AWS KMS Auto-Unseal
First, we need a running Vault instance. We’ll use the official Helm chart and configure it to auto-unseal using an AWS KMS key.
Prerequisite: AWS IAM Role for EKS (IRSA)
To allow our Vault pod to communicate with AWS KMS, we’ll use IAM Roles for Service Accounts (IRSA). You’ll need to create an IAM role with two policies.
- IAM Permissions Policy: This policy allows the role to perform cryptographic operations with your specific KMS key.JSON
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:DescribeKey" ], "Resource": "arn:aws:kms:YOUR-REGION:YOUR-ACCOUNT-ID:key/YOUR-KMS-KEY-ID" } ] } - Trust Policy: This policy allows the Kubernetes service account (which we’ll create via Helm) to assume this IAM role.JSON
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::YOUR-ACCOUNT-ID:oidc-provider/oidc.eks.YOUR-REGION.amazonaws.com/id/YOUR-OIDC-PROVIDER-ID" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.YOUR-REGION.amazonaws.com/id/YOUR-OIDC-PROVIDER-ID:sub": "system:serviceaccount:vault:vaultv2" } } } ] }
Note: Replace all placeholders (like YOUR-REGION, YOUR-ACCOUNT-ID, YOUR-KMS-KEY-ID, etc.) with your specific values. The service account name vaultv2 and namespace vault are what we’ll define in the Helm chart.
Vault Helm values.yaml
Create a values.yaml file for the Vault Helm chart. This configuration enables standalone mode (good for a demo), sets up the AWS KMS seal, configures ingress, and annotates the service account to use the IRSA role you just created.
Update the following fields with your values:
seal.awskms.kms_key_idseal.awskms.regioningress.hosts[0].hostserviceAccount.annotations(update with your specific Role ARN)
YAML
# We're using standalone mode for this demo (non-HA)
server:
standalone:
enabled: true
# Base Vault configuration
config: |-
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "file" {
path = "/vault/data"
}
# AWS KMS auto-unseal configuration
# Update with your key ID and region
seal "awskms" {
kms_key_id = "c0537c82-b50d-4df9-a205-df8c000000"
region = "eu-north-1"
}
# Ingress configuration (using Traefik and cert-manager)
# Update your host and annotations as needed
ingress:
enabled: true
annotations:
cert-manager.io/cluster-issuer: "http"
traefik.ingress.kubernetes.io/router.tls: 'true'
hosts:
- host: vaultv2.yourdomain.com
paths: []
ingressClassName: "traefik"
# Service Account configuration for IRSA
# Update the annotations with your role ARN
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::63820000000:role/vault-k8s"
iam.amazonaws.com/role: "arn:aws:iam::638200000000:role/vault-k8s"
Install and Initialize Vault
Now, let’s install Vault using Helm.
Bash
# Add the HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
# Install or upgrade Vault
helm upgrade --install vaultv2 hashicorp/vault -n vault \
--create-namespace \
--values values.yaml
After installing, the Vault pod will start, but it won’t be ready. You must initialize it one time.
Bash
# Exec into the running Vault pod
kubectl exec -it vaultv2-0 -n vault -- /bin/sh
# Inside the pod, run the init command
/ # vault operator init
IMPORTANT: Securely save the output of this command, especially the Unseal Keys and the Initial Root Token. You will need the Root Token for the next steps.
To verify that auto-unseal is working, delete the pod:
Bash
kubectl delete pod vaultv2-0 -n vault
The pod will be recreated by Kubernetes. This time, it should start up, connect to AWS KMS, unseal itself, and become ready without any manual intervention.
Part 2: Installing the Vault Secrets Operator (VSO)
With Vault running, it’s time to install VSO. Its job is to watch for new VaultStaticSecret custom resources and create corresponding Kubernetes Secret objects.
VSO Helm values.yaml
This configuration is much simpler. We just need to tell VSO where to find our Vault server.
YAML
defaultVaultConnection:
enabled: true
# This address points to the internal Kubernetes service for Vault
address: "http://vaultv2-internal.vault.svc.cluster.local:8200"
# We're skipping TLS for this demo. Do not do this in production.
skipTLSVerify: true
Install VSO
Bash
helm upgrade --install vault-secrets-operator hashicorp/vault-secrets-operator \
-n vault-secrets-operator-system \
--create-namespace \
--values values.yaml
Part 3: Configuring Vault for Kubernetes Authentication
Now we need to configure Vault to trust VSO and our applications. We’ll do this by enabling the Kubernetes auth method, which allows pods to authenticate using their service account tokens.
First, get access to your Vault server.
Bash
# Set your root token from the 'vault operator init' step
export VAULT_TOKEN=hvs.F40000000000000
# Port-forward the Vault service to your local machine
kubectl port-forward service/vaultv2 8200:8200 -n vault &
Now, run this script to configure Vault. The script will:
- Enable the Kubernetes auth method.
- Configure it to find the Kubernetes API.
- Enable a KV-v2 (Key/Value version 2) secrets engine at the path
k8s/. - Create a policy named
test-readthat grants read access to secrets under the pathcompany/test/. - Create a role named
vault-testthat links thetest-readpolicy to thedefaultservice account in thevaultnamespace. - Put a test secret into Vault.
Bash
# 1. Enable Kubernetes auth
vault auth enable -path=kubernetes kubernetes
# 2. Configure the Kubernetes auth method
vault write auth/kubernetes/config \
kubernetes_host="https://kubernetes.default.svc/"
# 3. Enable a KV-v2 secrets engine at the path "k8s"
vault secrets enable -path=k8s kv-v2
# 4. Create a policy for reading test secrets
# Note: For KV-v2, the API path includes "/data/", which is what the policy needs.
echo "Creating Vault policy 'test-read'..."
vault policy write test-read - << EOF
path "k8s/data/company/test/*" {
capabilities = ["read"]
}
EOF
# 5. Create a role linking the policy to a K8s service account
echo "Creating Vault role 'vault-test'..."
vault write auth/kubernetes/role/vault-test \
bound_service_account_names=default \
bound_service_account_namespaces=vault \
policies=test-read \
ttl=1h
# 6. Add a test secret
echo "Adding test secret..."
vault kv put k8s/data/company/test/secret1 value="itsasecret"
Part 4: Creating Your First Secret with VSO
We’re all set! Now for the final step: telling VSO to fetch our secret. We do this by applying two YAML manifests.
VaultAuth: Tells VSO how to authenticate to Vault (using thevault-testK8s role we just created).VaultStaticSecret: Tells VSO what secret to get and where to put it.
Create a file named vso-secret.yaml:
YAML
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: vso-auth
namespace: vault
spec:
method: kubernetes
mount: kubernetes # This is the auth path we enabled
kubernetes:
role: vault-test # This is the role we created
serviceAccount: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: deploy-secrets
namespace: vault
spec:
type: kv-v2
mount: k8s # The mount path of our KV-v2 engine
path: company/test/secret1 # The path *within* the engine
destination:
name: deploy-secrets # Name of the K8s secret to create
create: true
# Link to the VaultAuth configuration
vaultAuthRef: vso-auth
# How often to check for updates
refreshAfter: 30s
Apply the manifest:
Bash
kubectl apply -f vso-secret.yaml
A Quick Note on KV-v2 Paths
This is a common point of confusion. Notice the different paths we used:
- Vault CLI Command:
vault kv put k8s/data/company/test/secret1- The CLI knows
kvcommands for a KV-v2 engine need to interact with the/data/API path.
- The CLI knows
- VSO
VaultStaticSecret:mount: k8spath: company/test/secret1- VSO is smart enough to know that for a
kv-v2type, it needs to combine themountandpathto ask Vault fork8s/data/company/test/secret1.
You did this correctly in your original post, and it’s important to understand why it’s correct!
Part 5: Verification
Let’s see if it worked. First, check the status of the VaultStaticSecret resource:
Bash
kubectl describe vaultstaticsecret.secrets.hashicorp.com deploy-secrets -n vault
Look at the Status and Events sections. You should see a message indicating the secret has been synchronized.
If the status looks good, check that the native Kubernetes Secret was created:
Bash
kubectl get secret deploy-secrets -n vault -o yaml
You should see the data field populated with your base64-encoded secret. Your application can now mount deploy-secrets just like any other Kubernetes secret.
You now have a fully automated secrets management pipeline!