In a previous article, I wrote about the Key Vault FlexVolume driver for Kubernetes. I demonstrated how to use it to mount an HTTPS certificate from Azure Key Vault onto Kubernetes pods. Since then, the FlexVolume driver has been deprecated in favor of the Container Storage Interface (CSI) secrets store driver and Azure Key Vault provider.
The CSI Standard
The Container Storage Interface (CSI) is the latest evolution in storage plugins for Kubernetes. It is defined by a standard design to overcome the shortcomings of the FlexVolume plugin. It is an “out of tree” plugin, meaning that it is decoupled from Kubernetes so that CSI drivers can be developed and versioned separately from Kubernetes.
The Secrets Store CSI Driver
This driver’s design is a “secrets driver + provider” model where the secrets store CSI driver provides the implementation for mounting a volume and delivering secrets to pods. Providers implement access to a particular secrets store. Currently, supported providers include:
- Azure Key Vault
- HashiCorp Vault
- Google Secret Manager
Multiple providers can run in the same cluster simultaneously.
Besides mounting secrets to a pod volume, this driver also allows you to map from the secret store to Kubernetes secrets optionally. This is useful when instead of terminating TLS at the pod level, you are using an ingress controller such as NGINX that requires the HTTPS certificate to be a Kubernetes secret.
The SecretProviderClass Resource
With the FlexVolume driver for Key Vault, all the Key Vault and secret settings were declared in the YAML defining the volume mount in a deployment.
The Secret Store CSI Driver uses a custom Kubernetes resource called a SecretProviderClass to define the secret store and secret mount settings. Then the volume mount definition refers to the SecretProviderClass name. This results in a much cleaner deployment YAML and a decoupling of the secrets provider configuration from a particular volume mount.
Installing with Helm
Installing the Secrets Store CSI Driver and Azure provider is straightforward with the Helm package manager and the provided Helm charts. This installs the driver as a Kubernetes daemonset that will be available on all nodes so that any pods can utilize it in the cluster.
Mounting a Certificate for HTTPS
In addition to secrets such as passwords and API keys, Azure Key Vault can securely store and provide private key certificates such as those used for HTTPS. As we demonstrated with the FlexVolume driver for Key Vault, we can mount a certification to our pods and use them to bootstrap Kestrel in ASP.NET Core for HTTPS.
Let’s look at how we would do the same thing with the CSI Secret Store Driver and Azure Provider. You can see the full working example in the aks-csi-keyvault-certs GitHub repo and see more detailed instructions in the README.
First, we create our SecretProviderClass resource definition:
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1 kind: SecretProviderClass metadata: name: azure-kvname spec: provider: azure parameters: tenantId: "[*** YOUR KEY VAULT TENANT ID ***]" keyvaultName: "[*** YOUR KEY VAULT NAME ***]" objects: | array: - | objectName: aks-https objectAlias: https.pfx.base64 objectType: secret # object types: secret, key or cert objectFormat: pfx # for .NET Core 3.1 we want the PFX format objectVersion: "" # [OPTIONAL] object versions, default to latest if empty
We name our class azure-kvname, which we will use in our volume definition. In the object property, we can define 1-N secrets to be mounted as files on the volume. In this case, our secret has these properties:
|The name of the certificate in Key Vault.
|This will be used as the file name on the volume.
|We use “secret”, which will get us the private key certificate.
|Unfortunately, even if we have stored the certificate in Key Vault as a PFX, Azure CSI provider will automatically convert it to PEM format. .NET Core 3.1 does not support PEM out of the box, so setting this to “pfx” ensures we get a PFX.
In our Kubernetes Deployment YAML, we then define our volume like so:
volumes: - name: aks-keyvault-aspnetcore-httpscert csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "azure-kvname" nodePublishSecretRef: name: kvcreds
In this demo, we use an Azure Active Directory service principal to authenticate to Key Vault, whose credentials are stored as a Kubernetes secret. The nodePublishSecretRef option provides the name of the Kubernetes secret containing these credentials.
Then in our deployment YAML, we define the volume mount for our pods:
volumeMounts: - name: aks-keyvault-aspnetcore-httpscert mountPath: /certs readOnly: true
Given our volume mountPath and the objectAlias in our SecretProviderClass, the certificate will be available in our pods using the path /certs/https.pfx.base64.
The Secrets Store CSI Driver and Azure Key Vault provider for Kubernetes are a great way to deliver secrets to your containerized applications. If you are currently using the FlexVolume driver for Azure Key Vault, you should strongly consider updating to the CSI driver to take advantage of the latest innovations and features it provides.