Azure Kubernetes Services: policy and Open Service Mesh add-ons quick tip

Did you know that enabling both the azure-policy and the open-service-mesh (OSM) built-in AKS add-ons results in a non-functioning OSM? Azure ships with two default policy initiatives: Kubernetes cluster pod security baseline standards for Linux-based workloads and Kubernetes cluster pod security restricted standards for Linux-based workloads.

The primary purpose of these two initiatives is to harden your AKS clusters. The AKS policy add-on relies on an admission controller called Gatekeeper. Its role is to audit/deny deployments that violate some rules. In the AKS context, the policy add-on translates Azure policies into constraint templates interpreted and enforced by Gatekeeper. Suppose you apply the built-in initiatives in deny mode to your clusters and want to use OSM simultaneously. In that case, you will encounter a show stopper since OSM will immediately be blocked.

It is because OSM is based on the K8s sidecar pattern, which injects a sidecar container next to the application. To do this, OSM relies on init containers, which rewrite IP tables to let the sidecar intercept the inbound and outbound traffic of the application container. OSM’s init container requires NET_ADMIN and NET_RAW capabilities to do so. It violates the policy that Kubernetes cluster containers should only use allowed capabilities that are part of the built-in initiatives. Other service meshes such as Istio and Linkerd integrate with CNI plugins and do not rely on the init container to rewrite IP tables. It is not yet available for OSM. Therefore, the only possibility with OSM is to exempt the init container from the policy evaluation, and here comes the tip!

The built-in initiatives do not allow you to exempt this container only. They allow you to exempt namespaces but not specific containers. Since the purpose of a service mesh is to inject almost every namespace, you need another level of granularity, meaning the container or the container image itself. Fortunately, you can create your initiative and override specific policy parameters such as excludedContainers and excludedImages. It only allows this clear init container to leverage NET_ADMIN and NET_RAW, which is better than a total exemption or no policy.