Seamless Secret Management with Kubernetes Vault Secrets Operator

There are multiple ways of accessing secrets in HashiCorp Vault from your Kubernetes workloads. Which approach should you use? Well, it depends. In this blog, I will focus on accessing secrets using the Vault Secrets Operator. If you're curious about how to set up a highly available external Vault cluster, feel free to check out my previous blog post for a detailed guide.

Challenge

Vault offers a complete solution for secrets lifecycle management, but that requires developers and operators to learn a new tool. Instead, developers want a cloud native way to access the secrets through Kubernetes and have no need to understand Vault in great depth. Vault Secrets Operator (VSO) updates Kubernetes native secrets. The user accesses Kubernetes native secrets managed on the backend by HashiCorp Vault.

Solution

A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.

The Vault Secrets Operator is a Kubernetes operator that syncs secrets between Vault and Kubernetes natively without requiring the users to learn details of Vault use.

Currently, Vault secrets operator is available and supports kv-v1 and kv-v2, TLS certificates in PKI and full range of static and dynamic secrets.

The Vault Secrets Operator syncs the secrets between Vault and the Kubernetes secrets in a specified namespace. Within that namespace, applications have access to the secrets. The secrets are still managed by Vault, but accessed through the standard way on Kubernetes.

Prerequisites

Kubernetes Cluster. I'm using k3s.
Vault Cluster. I'm using Vault Community.

Configure Vault

Pre-Configuration

Retrieve the CA Certificate from Kubernetes Cluster.

cat /etc/rancher/k3s/k3s.yaml | grep certificate-authority-data | awk '{print $2}' | base64 --decode > ca.crt

# or

cat ~/.kube/config | grep certificate-authority-data | awk '{print $2}' | base64 --decode > ca.crt

Setting up vault authentication with Kubernetes

cat<<EOF > vault-sa.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
EOF

kubect apply -f vault-sa.yaml

ServiceAccount for Vault Authentication: We’ll start by creating a vault-auth ServiceAccount, which will be used by Vault to authenticate with the Kubernetes API.
Secret for the ServiceAccount: Kubernetes automatically generates a Secret that contains the authentication token for the vault-auth ServiceAccount. Vault will use this token to authenticate.
Set Up RBAC Permissions: Vault needs permission to verify ServiceAccount tokens. To achieve this, we create a ClusterRoleBinding that binds the vault-auth ServiceAccount to the system:auth-delegator ClusterRole, allowing it to review service tokens.

Retrieve authentication token

TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

For KUBE_HOST, since I am using an external Vault (outside of the Kubernetes cluster), I only need to define the connection URL to the Kubernetes API on port 6443. For example:

export KUBE_HOST="https://10.20.10.201:6443"

Configure Vault

Enable the Kubernetes auth method.

vault auth enable -path=vso kubernetes

Configure the auth method.

vault write auth/vso/config \
      token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
      kubernetes_host="$KUBE_HOST" \
      [email protected] \
      disable_issuer_verification=true

Enable the kv v2 Secrets Engine.

vault secrets enable -path=kvv2 kv-v2

Create a vault policy.

vault policy write wordpress-ro - <<EOF
path "kvv2/data/wordpress" {
   capabilities = ["read"]
}
path "kvv2/metadata/wordpress" {
   capabilities = ["read"]
}
EOF

$ vault policy list
default
wordpress-ro
root

Create a role in Vault to enable access to secrets within the kv v2 secrets engine.

vault write auth/vso/role/vso-role \
      bound_service_account_names=default \
      bound_service_account_namespaces=default \
      policies=wordpress-ro \
      ttl=24h

Notice that the bound_service_account_namespaces is default, limiting which namespace the secret is synced to.

$ vault list auth/vso/role
Keys
----
vso-role

$ vault read auth/vso/role/vso-role
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
bound_service_account_names                 [default]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [default]
policies                                    [wordpress-ro]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                0s
token_policies                              [wordpress-ro]
token_ttl                                   24h
token_type                                  default
ttl                                         24h

Create a secret.

vault kv put kvv2/wordpress password=":pa55word:"

vault kv get -format=json kvv2/wordpress  | jq ".data.data"
---
{
  "password": ":pa55word:"
}

Install the Vault Secrets Operator

Use helm to deploy the Vault Secrets Operator.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault-secrets-operator hashicorp/vault-secrets-operator

Define the Vault Connection. We need to create a VaultConnection resource that tells your Kubernetes cluster how to communicate with your Vault server.

cat<<EOF > vault-connection.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  # address to the Vault server.
  address: https://rj-vault.rjhaikal.my.id:8200
  skipTLSVerify: false
EOF

kubectl apply -f vault-connection.yaml

Deploy and Sync a Secret

Set up Kubernetes authentication for the secret.

cat<<EOF > vault-auth.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: vso
  kubernetes:
    role: vso-role
    serviceAccount: default
EOF

kubectl apply -f vault-auth.yaml

Create the secret names mysql-pass in the default namespace.

cat<<EOF > vault-static-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
spec:
  vaultAuthRef: vault-auth
  mount: kvv2
  type: kv-v2
  path:  wordpress
  refreshAfter: 10s
  destination:
    create: true
    name: mysql-pass
EOF

kubectl apply -f vault-static-secret.yaml

You will find a field called refreshAfter. It controls how often the secret is checked for updates. In this example, I set up to refresh after 10 seconds.

$ kubectl get vaultstaticsecret
NAME                  AGE
vault-static-secret   4s

$ kubectl describe vaultstaticsecret vault-static-secret
...
Events:
  Type    Reason         Age   From               Message
  ----    ------         ----  ----               -------
  Normal  SecretSynced   15s   VaultStaticSecret  Secret synced
  Normal  SecretRotated  15s   VaultStaticSecret  Secret synced

Verify and retrive secret.

$ kubectl get secret mysql-pass
NAME         TYPE     DATA   AGE
mysql-pass   Opaque   2      79s

To decode and view the actual secret data, use the following command:

kubectl get secret mysql-pass -o jsonpath='{.data._raw}' | base64 --decode | jq '.data'

{
  "password": ":pa55word:"
}

Or use the k9s tool to display secret. Open a new terminal and start up k9s.

k9s

Type in :secrets and press enter. Now the secrets named mysql-pass is displayed, highlight it. Display the secret by pressing the x key.

Rotate the Secret

Go to vault instance, and rotate the secret with a new value.

vault kv put kvv2/wordpress password=":pa55word:-v2"

vault kv get -format=json kvv2/wordpress  | jq ".data.data"
---
{
  "password": ":pa55word:-v2"
}

Wait about 10 seconds before continuing to the next step, to allow for the secret to refresh.

Return to k9s, and escape back to the secret page and press x again to display the updated secret.

Deploy Wordpress

Download wordpress manifests.

curl -LO https://k8s.io/examples/application/wordpress/mysql-deployment.yaml
curl -LO https://k8s.io/examples/application/wordpress/wordpress-deployment.yaml

Create ingress file.

cat<<EOF > ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
# Since I'm using k3s with Cilium, if you're using NGINX IngressClass, uncomment the following line:
#  ingressClassName: nginx
  rules:
  - host: wordpress.rjhaikal.my.id
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: wordpress
            port:
              number: 80
EOF

Create kustomize file.

cat<<EOF > kustomization.yaml
# Not necessary because the secret has already been created by Vault.
# secretGenerator:
# - name: mysql-pass
#   literals:
#   - password=:pa55word:-v2
resources:
  - mysql-deployment.yaml
  - wordpress-deployment.yaml
  - ingress.yaml
namespace: default
EOF

Apply manifests.

kubectl apply -k ./

kubectl get pod,ing,pvc
NAME                                                             READY   STATUS    RESTARTS   AGE
pod/vault-secrets-operator-controller-manager-5456b96fbb-4z5x2   2/2     Running   0          52m
pod/wordpress-7d7846bd75-8rsm4                                   1/1     Running   0          13s
pod/wordpress-mysql-6dd978bc8b-sn6sb                             1/1     Running   0          13s

NAME                                          CLASS    HOSTS                      ADDRESS        PORTS   AGE
ingress.networking.k8s.io/wordpress-ingress   cilium   wordpress.rjhaikal.my.id   10.20.10.170   80      13s

NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
mysql-pv-claim   Bound    pvc-a7da0472-6b4b-4b17-a27e-18f02886217b   20Gi       RWO            local-path     <unset>                 13s
wp-pv-claim      Bound    pvc-3f02bd2f-0c75-41d4-9e93-d273c695c5eb   20Gi       RWO            local-path     <unset>                 13s