Hardening AKS: How to prevent pulling from unknown container registries using Azure Policies

Introduction

In this post, I'll demonstrate how pulling from foreign registries can be restricted on your Azure Kubernetes Cluster.

By limiting the sources from which Docker images can be pulled, you enhance the overall security of your Azure Kubernetes Cluster, as it reduces the risk of running potentially malicious or vulnerable images. It can further lead to resource optimizations as it will reduce network bandwidth.

Although there are other ways to solve this use case, I'll only outline the Azure way of protecting your cluster by leveraging Azure Policy, as it is the most native. Let's dive in 🤓

Using Azure Policies with AKS

Doing it the Azure way implies leveraging Azure Policy for this task. We will therefore need the following:

An AKS cluster running version 1.14 or higher
The Azure Policy Add-on for AKS
A Gatekeeper v3 installation
An Azure Policy

Enabling the Azure Policy Add-on for AKS

The nice thing about the Azure Policy Add-on is that installing it also does install Gatekeeper for you. Gatekeeper is an admission controller webhook for the Open Policy Agent.

🔎 Admission controllers are Kubernetes components that intercept requests to the Kubernetes API server before they are processed. They are used to enforce policies, perform custom validation, and modify requests before they are allowed to proceed.

Let's start by checking the running Kubernetes version, which should be 1.14 or higher.

$ az aks show -n aks-azureblue -g rg-demo --query kubernetesVersion
"1.24.9"

According to the documentation, you must ensure your subscription has the Microsoft-PolicyInsights resource provider activated before installing the Azure Policy Add-on.

$ az provider register --namespace Microsoft.PolicyInsights

Next, we can enable the Add-On, which can take a minute or two to complete.

$ az aks enable-addons --addons azure-policy --name aks-azureblue --resource-group rg-demo

And finally, verify if the installation went successful

$ kubectl get pods -n kube-system | Select-String azure-policy
azure-policy-7f88886d48-8v26s                                     1/1     Running   0          6m31s
azure-policy-webhook-756cc8b7dd-lg86n                             1/1     Running   0          6m31s

$ kubectl get pods -n gatekeeper-system
NAME                                     READY   STATUS    RESTARTS   AGE
gatekeeper-audit-6647475577-wxndx        1/1     Running   0          7m52s
gatekeeper-controller-5ddfc9b86b-5mp7r   1/1     Running   0          7m52s
gatekeeper-controller-5ddfc9b86b-mgvrw   1/1     Running   0          7m52s

Assign Azure Policy

For this post, we are particularly interested in a built-in policy described as "Kubernetes cluster containers should only use allowed images."

🔎 For a list of Azure Kubernetes built-in Policies navigate to the official documentation.

Its name is febd0533-8e55-448f-b837-bd0e06f16469. Let's retrieve the description.

$ az policy definition show --name febd0533-8e55-448f-b837-bd0e06f16469 --query description

"Use images from trusted registries to reduce the Kubernetes cluster's exposure risk to unknown vulnerabilities, security issues and malicious images. This policy is generally available for Kubernetes Service (AKS), and preview for Azure Arc enabled Kubernetes. For more information, see https://aka.ms/kubepolicydoc."

Retrieving the policies description

Last but not least, an Azure Policy needs to be assigned. Unfortunately, I can only demonstrate Azure PowerShell here since passing JSON as a string with Azure CLI didn't work for me.

# Define parameters for Azure Policy
$param = @{
    "effect" = "deny";
    "excludedNamespaces" = "kube-system", "gatekeeper-system", "azure-arc", "playground";
    "allowedContainerImagesRegex" = "crazureblue\.azurecr\.io\/.+$";
}

# Set a name and display name for the assignment 
$name = 'restrict-container-registries'

# Retrieve the Azure Policy object 
$policy = Get-AzPolicyDefinition -Name 'febd0533-8e55-448f-b837-bd0e06f16469'

# Retrieve the resource group for scope assignment 
$scope = Get-AzResourceGroup -Name rg-demo 

# Assign the policy 
New-AzPolicyAssignment -DisplayName $name -name $name -Scope $scope.ResourceId -PolicyDefinition $policy -PolicyParameterObject $param

Create a policy assignment using Azure PowerShell

As you can see from the $param hash map, I am using a regex of crazureblue\.azurecr\.io/.+$. This will allow image definitions in the following format.

crazureblue.azurecr.io/namespace/sub-namespace/repository:tag
crazureblue.azurecr.io/namespace/repository:tag
crazureblue.azurecr.io/repository:tag

Also, the excluded namespaces should be noted. I've excluded an additional playground beside the system namespaces. To white-list multiple registries, you can use a regex of (crazureblue.azurecr.io|docker.io)/.+$, which would allow the following formats.

crazureblue.azurecr.io/repository:tag
crazureblue.azurecr.io/.../repository:tag
docker.io/repository:tag
docker.io/.../repository:tag

Verify policy setup

After the policy got successfully assigned, let's try creating a pod with an image from Docker hub, e.g., so

💡Please note, it can take up to 20 minutes for policy assignments to sync into the cluster💡

apiVersion: v1
kind: Pod
metadata:
  name: webserver
  namespace: demo
  labels:
    name: webserver
spec:
  containers:
  - name: webserver
    image: nginx:latest
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    ports:
      - containerPort: 8080

Pod definition with image from Docker Hub

As expected, the operation gets stopped by the admission webhook.

$ kubectl apply -f c:\temp\pod.yaml

Error from server (Forbidden): error when creating "C:\\temp\\pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev2containerallowedimag-ed3f60290a38f6a6500d] Container image nginx:latest for container webserver has not been allowed.

Now, let's move the image to our private Azure Container Registry.

$ docker pull nginx:latest 
$ docker tag nginx:latest crazureblue.azurecr.io/nginx:latest
$ az acr login --name crazureblue 
$ az push crazureblue.azurecr.io/nginx:latest

And now, creating the pod with a private image source in the demo namespace will succeed 🙌🏻

apiVersion: v1
kind: Pod
metadata:
  name: webserver
  namespace: demo
  labels:
    name: webserver
spec:
  containers:
  - name: webserver
    image: crazureblue.azurecr.io/nginx:latest
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    ports:
      - containerPort: 8080

Behind the scenes

The Azure Policy Add-On translates Azure Policies into Gatekeeper constraint templates and constraints. It checks every 15 minutes with the Azure Policy service assignment changes and creates, updates, or deletes the constraints.

You can get a list of the installed constraint templates by issuing

$ kubectl get constrainttemplates

NAME                               AGE
k8sazurev1blockdefault             14h
k8sazurev1ingresshttpsonly         14h
k8sazurev1serviceallowedports      14h
k8sazurev2blockautomounttoken      14h
k8sazurev2blockhostnamespace       14h
k8sazurev2containerallowedimages   14h
k8sazurev2noprivilege              14h
k8sazurev3allowedcapabilities      14h
k8sazurev3allowedusersgroups       14h
k8sazurev3containerlimits          14h
k8sazurev3disallowedcapabilities   14h
k8sazurev3enforceapparmor          14h
k8sazurev3hostfilesystem           14h
k8sazurev3hostnetworkingports      14h
k8sazurev3noprivilegeescalation    14h
k8sazurev3readonlyrootfilesystem   14h

And to see details for the constraint template that we are using to restrict container registries. The output shows the hint to the Azure Policy.

$ kubectl get constrainttemplate/k8sazurev2containerallowedimages -o yaml

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  annotations:
    azure-policy-definition-id-1: /providers/Microsoft.Authorization/policyDefinitions/febd0533-8e55-448f-b837-bd0e06f16469
    constraint-template: https://store.policy.core.windows.net/kubernetes/container-allowed-images/v2/template.yaml
    constraint-template-installed-by: azure-policy-addon
  creationTimestamp: "2023-02-20T07:10:55Z"
  generation: 1
  labels:
    managed-by: azure-policy-addon
  name: k8sazurev2containerallowedimages
...

And here is the constraint with hints to the Azure Policy Assignment

$ kubectl get k8sazurev2containerallowedimages.constraints.gatekeeper.sh/azurepolicy-k8sazurev2containerallowedimag-1ae7d556007bf4e129c0 -o yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAzureV2ContainerAllowedImages
metadata:
  annotations:
    azure-policy-assignment-id: /subscriptions/<subscription>/resourceGroups/rg-demo/providers/Microsoft.Authorization/policyAssignments/restrict-container-registries
    azure-policy-definition-id: /providers/Microsoft.Authorization/policyDefinitions/febd0533-8e55-448f-b837-bd0e06f16469
    azure-policy-definition-reference-id: ""
    azure-policy-setdefinition-id: ""
    constraint-installed-by: azure-policy-addon
  creationTimestamp: "2023-02-20T17:56:36Z"
  generation: 1
  labels:
    managed-by: azure-policy-addon
  name: azurepolicy-k8sazurev2containerallowedimag-1ae7d556007bf4e129c0
  resourceVersion: "2847096"
  uid: 6b13e8a3-1bb6-42a6-bdc3-5de9597a332c
spec:
  enforcementAction: deny
  match:
    excludedNamespaces:
    - kube-system
    - gatekeeper-system
    - azure-arc
    - playground
    kinds:
    - apiGroups:
      - ""
      kinds:
      - Pod
  parameters:
    excludedContainers: []
    imageRegex: crazureblue\.azurecr\.io\/.+$
...

You should not manually edit these templates or constraints as the Azure Policy will overwrite your changes roughly every 15 minutes with settings from the Azure Policies.

That's it for today, thanks for reading, and happy hardening! 🔐💥🤓

Summary

By installing Azure Policy Add-On, Gatekeeper gets installed
The Azure Policy Add-on creates Gatekeeper constraints behind the scenes
You can limit the policy to specific namespaces only by using namespaces parameter
You can exclude namespaces by using the excludedNamespaces parameter
Policy assignment can take up to 20 minutes to become effective
You should not manually edit the constraints as they will get overwritten
When passing parameters to the Azure Policies I recommend creating a PowerShell hash-map object instead of fiddling around with JSON strings
Properly escaping JSON with Azure CLI and PowerShell is a pain