In a previous article, I wrote about the Key Vault FlexVolume driver for Kubernetes. I demonstrated how to use it to mount an HTTPS certificate from Azure Key Vault onto Kubernetes pods. Since then, the FlexVolume driver has been deprecated in favor of the Container Storage Interface (CSI) secrets store driver and Azure Key Vault provider.

The CSI Standard

The Container Storage Interface (CSI) is the latest evolution in storage plugins for Kubernetes. It is defined by a standard design to overcome the shortcomings of the FlexVolume plugin. It is an “out of tree” plugin, meaning that it is decoupled from Kubernetes so that CSI drivers can be developed and versioned separately from Kubernetes.

The Secrets Store CSI Driver

This driver’s design is a “secrets driver + provider” model where the secrets store CSI driver provides the implementation for mounting a volume and delivering secrets to pods. Providers implement access to a particular secrets store. Currently, supported providers include:

• Azure Key Vault
• HashiCorp Vault
• Google Secret Manager

Multiple providers can run in the same cluster simultaneously.

Besides mounting secrets to a pod volume, this driver also allows you to map from the secret store to Kubernetes secrets optionally. This is useful when instead of terminating TLS at the pod level, you are using an ingress controller such as NGINX that requires the HTTPS certificate to be a Kubernetes secret.

The SecretProviderClass Resource

With the FlexVolume driver for Key Vault, all the Key Vault and secret settings were declared in the YAML defining the volume mount in a deployment.

The Secret Store CSI Driver uses a custom Kubernetes resource called a SecretProviderClass to define the secret store and secret mount settings. Then the volume mount definition refers to the SecretProviderClass name. This results in a much cleaner deployment YAML and a decoupling of the secrets provider configuration from a particular volume mount.

Installing with Helm

Installing the Secrets Store CSI Driver and Azure provider is straightforward with the Helm package manager and the provided Helm charts. This installs the driver as a Kubernetes daemonset that will be available on all nodes so that any pods can utilize it in the cluster.

Mounting a Certificate for HTTPS

In addition to secrets such as passwords and API keys, Azure Key Vault can securely store and provide private key certificates such as those used for HTTPS. As we demonstrated with the FlexVolume driver for Key Vault, we can mount a certification to our pods and use them to bootstrap Kestrel in ASP.NET Core for HTTPS.
Let’s look at how we would do the same thing with the CSI Secret Store Driver and Azure Provider. You can see the full working example in the aks-csi-keyvault-certs GitHub repo and see more detailed instructions in the README.

First, we create our SecretProviderClass resource definition:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    tenantId: "[*** YOUR KEY VAULT TENANT ID ***]"
    keyvaultName: "[*** YOUR KEY VAULT NAME ***]"
    objects:  |
      array:
        - |
          objectName: aks-https
          objectAlias: https.pfx.base64
          objectType: secret        # object types: secret, key or cert
          objectFormat: pfx         # for .NET Core 3.1 we want the PFX format
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty

We name our class azure-kvname, which we will use in our volume definition. In the object property, we can define 1-N secrets to be mounted as files on the volume. In this case, our secret has these properties:

In our Kubernetes Deployment YAML, we then define our volume like so:

      volumes:
      - name: aks-keyvault-aspnetcore-httpscert
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-kvname"
          nodePublishSecretRef:
            name: kvcreds

In this demo, we use an Azure Active Directory service principal to authenticate to Key Vault, whose credentials are stored as a Kubernetes secret. The nodePublishSecretRef option provides the name of the Kubernetes secret containing these credentials.

Then in our deployment YAML, we define the volume mount for our pods:

        volumeMounts:
        - name: aks-keyvault-aspnetcore-httpscert
          mountPath: /certs
          readOnly: true

Given our volume mountPath and the objectAlias in our SecretProviderClass, the certificate will be available in our pods using the path /certs/https.pfx.base64.

Keeping Secrets

The Secrets Store CSI Driver and Azure Key Vault provider for Kubernetes are a great way to deliver secrets to your containerized applications. If you are currently using the FlexVolume driver for Azure Key Vault, you should strongly consider updating to the CSI driver to take advantage of the latest innovations and features it provides.