Which network option to choose with Azure Kubernetes Services (AKS)?

AKS allows you to choose between two network options: Kubernetes (native K8s) or another CNI (Container Network Interface) plugin. Both modes come with a series of benefits and drawbacks. Kubernetes is a tempting option because it is IP-friendly, so your clusters won’t consume many IPs. Kubernetes uses NAT (Network Address Translation) to NAT pod IPs to node IPs. In short, only nodes receive an actual IP address.

In the IPv4 world, companies run shorter and shorter available IP addresses, which may lead them to choose Kubernetes. Conversely, any other CNI option (Azure CNI or Bring Your Own CNI) will allocate one IP per running pod. Until recently, Azure CNI pre-allocated 30 IPs per node, no matter how many pods were running on the node.

So, with a CNI, every pod gets a routable IP, like a VM. Pods are therefore sensitive to NSGs, and all the network plumbing offered by Azure. I do not consider this a real benefit. Unlike VMs, pods are volatile and can be re-scheduled to any node or even on the same node with a different IP, so you are unlikely to allow pod IPs directly in a firewall rule. No matter Kubernet or CNI, you can organize your node pools and subnets to isolate sensitive components into their subnet, leverage NSGs, etc.

In theory, Kubernetes is a little slower than CNI because it performs the NAT operations, but this extra latency seems invisible in practice. What could drive your choice for one or the other is mainly the following factors:

• If you plan to run monster clusters, a CNI plugin is a better option because when using Kubernetes, AKS clusters are currently limited to 400 nodes.
• If you plan to use the Virtual Kubelets, the Azure CNI plugin is required
• Suppose you plan to use any ecosystem solution (a Service Mesh, for example). In that case, that is based on the Sidecar pattern and rewrites IP tables to proxy the application container’s inbound & outbound traffic. With Kubernetes, this IP table rewriting will be handled by init containers, requiring extra RBAC permissions such as NET_ADMIN. In AKS, you can only lock down that permission to specific containers, but if you can’t live with that exception, then a CNI plugin would be a better option.