Using Vault with Kubernetes Authentication

Updated on October 20, 2022

This guide will walk you through configuring HashiCorp Vault running on a VKE cluster to exchange service accounts for a scoped client Vault token. This can be useful when you want your services running on a Kubernetes cluster to self-authenticate against Vault without the need to pass around Vault credentials.

Prerequisites

Before you begin, you should have the following:

A running VKE cluster
HashiCorp Vault installed with the Web UI started
Kubernetes administration experience

Auth Delegators

The first thing you want to set up is a ClusterRoleBinding that has a roleRef which uses system:auth-delegator

This role allows delegated authentication and authorization checks. Add-on API servers commonly use this for unified authentication and authorization.

apiVersion: rbac.authorization.k8s.io/v1beta1  
kind: ClusterRoleBinding  
metadata:  
    name: role-tokenreview-binding  
    namespace: default  
roleRef:  
    apiGroup: rbac.authorization.k8s.io     
    kind: ClusterRole  
    name: system:auth-delegator  
subjects:  
- kind: ServiceAccount     
    name: vault-auth     
    namespace: default

Note: Change the namespace to something more appropriate than default.

Next, create a service account which will get bound to this ClusterRoleBinding. Ensure the ServiceAccount namespace matches the ServiceAccount specified in the ClusterRoleBinding.

apiVersion: v1
kind: ServiceAccount
metadata:
    name: vault-auth
    namespace: default

Required information

Before you continue to set up Vault, gather some extra data from the cluster and the newly created service account. In your terminal, collect the following information in your environment.

Service Account Token

$ export VAULT_SA_NAME=$(kubectl get sa vault-auth --output jsonpath="{.secrets[*]['name']}")

Service Account JWT

$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME --output 'go-template={{ .data.token }}'| base64 --decode)

Service Account CA CRT

$ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

K8 Hostname

$ export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')

JWT Issue

If the cluster is on VKE 1.21+, you must pass the following:

https://kubernetes.default.svc

This is required for version 1.21 and later because those versions use a different auth method.

Vault Setup

Now you can set up your Kubernetes auth on Vault.

Connect to the Vault Web UI.
In the Access section, click enable new method to navigate to a selection screen.
Click Kubernetes.
The path that you should use is kubernetes. This is the default path, and there is no need to change it unless you want to add a different Kubernetes auth engine.
On the next page, fill in three fields:

Kubernetes Host: equal to echo ${K8S_HOST}
Kubernetes SA CA Certificate: equal to echo ${SA_CA_CRT}
Token Reviewer JWT (in Kubernetes Options): equal to echo ${SA_JWT_TOKEN}

With this configured, there is now an accessible v1/auth/kubernetes/login endpoint. This is where your services must send requests to get a client token.

Role Setup

Policies and roles are created on a per-service basis. Here is an example of the setup required to get a scoped client token.

Policy creation

Go to the Policies section of Vault and click create ACL Policy. Then, give it a name and a policy. Here is an example policy for a path that would already exist on Vault. Note that these paths must be created as new engines.

path "my-app/*" {  
    capabilities = ["read", "list"]  
}

Service Account

While every namespace has a default service account that gets attached to a pod if you do not provide one, we will create one for the sake of this guide and the want/need to scope pods to Vault ACLs.

kubectl create sa {MY SA NAME} -n {NAMESPACE}

To assign this service account to your pods that will interact with Vault, define the serviceAccountName field in your spec definition.

spec:
    serviceAccountName: {MY SA NAME}

In the Vault access section, click view configuration on your Kubernetes auth, leading to a page with two tabs. Click the Roles tab, then Create Role. Complete the form with the following:

Name: The name of this role. A good choice is the name of the service that uses it.
Bound service account names: The service account name.
Bound service account namespace: The namespace that the service account lives in.
Generated Tokens Policies: Name of the policy to use.

Service integration

Now, your services have scoped identities and no longer require the root client token to be passed to them. Instead, you will have to set up the services to call /v1/auth/kubernetes/login with a JSON body that consists of role and jwt.

Role: Name of role on Vault.
JWT: The service account token that is mounted to the pod. This can be found at /run/secrets/kubernetes.io/serviceaccount/token inside of the pod.

Wrapping up

After successful validation, Vault returns a JSON body describing your auth status, which includes your scoped client token. This eliminates the need to store any information within your applications about Vault, as it is entirely driven by the native login functionality of VKE.

Useful links

To learn more about Vault and Kubernetes auth, see these resources.