AI News Hub Logo

AI News Hub

Kubernetes CNI Complete Guide: Flannel vs Cilium vs Calico + Cloud Provider CNIs

DEV Community
Pendela BhargavaSai

K3s v1.29+  |  Flannel v0.24+  |  Cilium v1.15+  |  Calico v3.27+  |  AWS VPC CNI v1.18+  |  Azure CNI v1.5+  |  GKE Dataplane V2 (Cilium-based) A definitive comparison of every major Kubernetes CNI โ€” open-source plugins (Flannel, Calico, Cilium, Weave, Antrea, Multus) and cloud-managed defaults (AWS VPC CNI on EKS, Azure CNI on AKS, and GKE's Dataplane V2 on GKE) โ€” across architecture, performance, network policy, observability, encryption, and when to choose each. CNI Identity Core Approach Default On ๐ŸŸข Flannel Simple Overlay VXLAN tunnel, zero policy K3s ๐ŸŸ  Calico Policy Powerhouse BGP routing, iptables/eBPF Self-managed ๐Ÿ”ต Cilium eBPF Native Kernel eBPF, replaces kube-proxy GKE (Dataplane V2) ๐ŸŸก Weave Net Mesh Overlay Gossip-based mesh routing Self-managed ๐ŸŸฃ Antrea VMware-backed OVS dataplane, Antrea policies Self-managed ๐Ÿ”ถ AWS VPC CNI Cloud-native Native VPC IP assignment EKS ๐Ÿ”ท Azure CNI Cloud-native Azure VNET IP assignment AKS โ™ฆ๏ธ GKE CNI / Dataplane V2 Cloud-native + eBPF Cilium-based eBPF on GKE GKE What Is a CNI? 2.1 Flannel โ€” Simple Overlay 2.2 Cilium โ€” eBPF Native 2.3 Calico โ€” BGP + Flexible Dataplane 2.4 Weave Net โ€” Mesh Overlay 2.5 Antrea โ€” OVS-based CNI 2.6 Multus โ€” Meta CNI Cloud Provider CNIs 3.1 AWS VPC CNI โ€” EKS Default 3.2 Azure CNI โ€” AKS Default 3.3 GKE Dataplane V2 โ€” GKE Default Data Plane Comparison Network Policy Observability Performance Benchmarks Encryption Multi-Cluster Resource Usage Full Feature Comparison When to Choose Each K3s-Specific Setup Migration Guide on K3s Conclusion The Container Network Interface (CNI) is the plugin layer every Kubernetes cluster depends on for: Assigning IP addresses to pods from a defined CIDR range Creating virtual Ethernet (veth) pairs between pod namespaces and the host Programming cross-node routing so pods on Node A can reach pods on Node B Optionally enforcing NetworkPolicy resources to control traffic flow Cloud providers like AWS, Azure, and GCP have built proprietary CNI plugins that deeply integrate with their underlying VPC/VNET networking primitives โ€” providing native IP assignment, cloud-aware routing, and tight integration with cloud IAM, load balancers, and security groups. ๐Ÿ’ก K3s Key Flag --flannel-backend=none --disable-network-policy. This leaves the CNI slot open for Calico or Cilium to fill. Flannel's design philosophy: do one thing well. A user-space daemon (flanneld) manages subnet allocation, while the kernel's own VXLAN and bridge code handles all actual forwarding. No policy, no observability โ€” just connectivity. Pod A (eth0: 10.244.0.2) Pod B (eth0: 10.244.0.5) โ”‚ โ”‚ โ”‚ veth pair โ”‚ veth pair โ–ผ โ–ผ cni0 Linux bridge (kernel) โ”‚ iptables PREROUTING / FORWARD / POSTROUTING โ”‚ VXLAN encapsulation โ€” UDP 8472 โ”‚ flanneld (user-space) โ† etcd / K8s API โ”‚ Physical NIC โ†’ Node B Available backends: Backend Transport Use Case vxlan UDP encap (default) Works across any network, even routers host-gw Direct routing Fastest, requires L2 adjacency between nodes wireguard-native Encrypted WireGuard tunnel When you need encryption udp Legacy user-space Fallback only โ€” very slow Network Policy: Flannel enforces zero NetworkPolicy. Resources are silently ignored. You must pair it with Calico (Canal) to get policy โ€” adding a second DaemonSet, version compatibility risk, and split ownership between two projects. Flannel Encryption: Flannel encrypts cross-node traffic only โ€” pod-to-pod on the same node travels through the cni0 bridge unencrypted. No auto key rotation; restart flanneld to rotate keys. { "Network": "10.244.0.0/16", "Backend": { "Type": "wireguard" } } Best for: Dev/CI clusters, Raspberry Pi, edge nodes, K3s defaults. Cilium compiles and injects eBPF programs into the Linux kernel at TC/XDP hook points. There is no bridge, no iptables โ€” packets are forwarded via bpf_redirect() at line rate, and policy is enforced via O(1) BPF map lookups. Pod A (eth0) Pod B (eth0) โ”‚ โ”‚ โ”‚ veth pair โ”‚ โ–ผ โ–ผ TC eBPF hook โ”€โ”€โ”€โ”€ bpf_redirect() โ”€โ”€โ”€โ”€ TC eBPF hook โ”‚ BPF maps: identity ยท policy ยท NAT ยท LB โ”‚ cilium-agent โ€” compiles eBPF, watches K8s API โ”‚ Physical NIC โ€” GENEVE / native routing Datapath modes: Mode Encapsulation Requirement tunnel: geneve GENEVE (default) Any network topology native-routing None L2 adjacency or BGP underlay wireguard WireGuard transparent Kernel โ‰ฅ 5.6 ipsec IPsec FIPS-regulated environments Network Policy: 4.3 Cilium โ€” L3 Through L7, No Sidecar Cilium enforces standard NetworkPolicy and extends it with CiliumNetworkPolicy (CNP) for Layer 7 rules โ€” no sidecar required: # CiliumNetworkPolicy โ€” L7 HTTP rule apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-get-only spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: app: frontend toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: GET path: "/api/v1/.*" โœ… Per-flow visibility on every packet โœ… Live service dependency map (Hubble UI) โœ… L7 HTTP / DNS / Kafka / gRPC flows โœ… Drop reason per endpoint โœ… Rich Prometheus metrics # Enable Hubble and UI cilium hubble enable --ui # Watch live flows in a namespace hubble observe --namespace production --follow # Show only policy drops with reason hubble observe --verdict DROPPED --follow # Sample output: # 12:34:01: default/frontend โ†’ default/backend FORWARDED TCP:SYN # 12:34:02: default/attacker โ†’ default/backend DROPPED Policy denied Cilium Encryption: Cilium WireGuard + IPsec # WireGuard with strict mode (drops unencrypted packets) cilium install \ --encryption wireguard \ --encryption-strict-mode true # IPsec for FIPS-regulated environments cilium install --encryption ipsec Best for: Large-scale production, L7 policy, observability (Hubble), zero-trust, multi-cluster. Calico uses BGP (Border Gateway Protocol) to distribute pod routes across nodes โ€” no encapsulation by default. Each node acts as a BGP peer, advertising its pod CIDR to other nodes and upstream routers. Calico's data plane is pluggable: iptables, eBPF, or even Windows HNS. Pod A (eth0: 192.168.0.2) Pod B (eth0: 192.168.1.2) โ”‚ โ”‚ โ”‚ veth pair โ”‚ veth pair โ–ผ โ–ผ Host routing table (no bridge needed) โ”‚ iptables / eBPF policy enforcement โ”‚ Felix (per-node agent) โ† Typha (fan-out) โ”‚ BIRD (BGP daemon) โ€” peers with other nodes โ”‚ Physical NIC โ€” direct IP routing (no encap) Key Calico components: Component Role Felix Per-node agent; programs iptables/eBPF rules and routes BIRD Open-source BGP daemon; advertises pod subnets to peers Typha Fan-out proxy for the K8s datastore; recommended at 50+ nodes calico-kube-controllers Garbage-collects stale Calico resources Network Policy: 4.2 Calico โ€” L3/L4 Policy Leader Calico is widely regarded as the gold standard for L3/L4 NetworkPolicy. It supports standard NetworkPolicy resources plus its own GlobalNetworkPolicy and NetworkSet CRDs: # Calico GlobalNetworkPolicy โ€” cluster-wide deny-all apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default-deny-all spec: selector: all() types: - Ingress - Egress # Calico NetworkSet โ€” group external CIDRs apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: trusted-external spec: nets: - 203.0.113.0/24 - 198.51.100.0/24 โš ๏ธ Calico does not support L7 HTTP/gRPC policy natively in OSS. For that you need its optional Envoy-based Application Layer Policy (ALP), which adds a sidecar and complexity. Calico Encryption: Calico supports WireGuard for node-to-node encryption, enabled with a single patch: kubectl patch felixconfiguration default \ --type merge \ --patch '{"spec":{"wireguardEnabled":true}}' Starting in Calico v3.26, same-node pod traffic encryption is also supported via host-to-pod WireGuard options. Best for: BGP-integrated DCs, Windows node support, bare-metal L3, robust L3/L4 policy. Weave Net uses a gossip protocol to build a full mesh topology between all cluster nodes without any central store. It wraps packets in a sleeve (VXLAN-like) tunnel and can optionally encrypt all traffic with NaCl. Weave is simpler to operate than Calico/Cilium but is no longer under active development (archived by Weaveworks in 2023). Pod A (eth0) โ”‚ weave bridge โ”‚ weave daemon (gossip mesh peer discovery) โ”‚ Sleeve / Fast Datapath (VXLAN kernel bypass) โ”‚ Node B weave daemon โ”‚ Pod B (eth0) Key characteristics: Feature Detail Discovery Gossip โ€” no external etcd needed Datapath Sleeve (user-space) or Fast Datapath (kernel VXLAN) Encryption NaCl (enabled per-pod connection) NetworkPolicy โœ… Standard K8s policy supported Status โš ๏ธ Archived/maintenance mode (use Cilium or Calico for new clusters) โš ๏ธ Important: Weaveworks ceased active development in 2023. Weave Net is community-maintained but no longer receives feature updates. It is not recommended for new clusters โ€” migrate to Cilium or Calico. Best for: Legacy clusters already running Weave with migration on the roadmap. Antrea is a CNI backed by VMware (now Broadcom) that uses Open vSwitch (OVS) as its dataplane. It supports both Linux and Windows nodes and provides its own AntreaNetworkPolicy and ClusterNetworkPolicy CRDs with tiered policy enforcement. Antrea integrates well with NSX-T for enterprise SD-WAN environments. Pod A (eth0) โ”‚ OVS (Open vSwitch) bridge โ”‚ antrea-agent (per-node DaemonSet) โ”‚ antrea-controller (centralized) โ”‚ Encap: Geneve / VXLAN / GRE (configurable) โ”‚ Node B OVS bridge โ†’ Pod B Key features: Feature Antrea Dataplane Open vSwitch (OVS) Windows support โœ… Full (OVS on Windows) NetworkPolicy โœ… K8s standard + AntreaNetworkPolicy CRDs Tiered policy โœ… (Emergency / Security / Application tiers) Encryption โœ… IPsec / WireGuard Observability โœ… Antrea Octant plugin, Prometheus metrics NSX-T integration โœ… Enterprise add-on eBPF support โœ… AntreaProxy (partial eBPF) Best for: VMware/NSX-T environments, Windows-heavy clusters, tiered network policy. Multus is not a standalone CNI โ€” it is a meta CNI that allows pods to attach multiple network interfaces simultaneously. A pod can have its primary network (managed by Flannel/Calico/Cilium) and secondary interfaces (SR-IOV, DPDK, Macvlan) for specialized workloads like telco NFV or HPC. Pod with Multiple NICs: eth0 (primary) โ† Flannel/Calico/Cilium (cluster network) net1 (secondary) โ† SR-IOV (high-throughput direct NIC) net2 (secondary) โ† Macvlan (storage network) Multus reads NetworkAttachmentDefinition CRDs and delegates to the correct CNI for each interface. # NetworkAttachmentDefinition for secondary interface apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: sriov-net spec: config: | { "type": "sriov", "name": "sriov-net", "ipam": { "type": "static" } } Best for: Telco/NFV workloads, HPC, pods that need to straddle multiple network segments. Cloud-managed Kubernetes services ship their own CNI plugins that are deeply integrated with the underlying cloud networking fabric. These provide first-class VPC routing, cloud IAM integration, and managed lifecycle โ€” but are typically locked to their respective cloud. Amazon EKS uses the Amazon VPC CNI plugin (aws-node DaemonSet) by default. Instead of an overlay, it assigns real VPC secondary IP addresses directly to pods from Elastic Network Interfaces (ENIs) attached to the worker node. Worker Node (EC2 instance) โ”‚ โ”œโ”€โ”€ Primary ENI (node IP: 10.0.1.10) โ”‚ โ””โ”€โ”€ eth0 โ”‚ โ”œโ”€โ”€ Secondary ENI (attached by vpc-cni) โ”‚ โ”œโ”€โ”€ 10.0.1.20 โ†’ Pod A (eth0 via veth) โ”‚ โ”œโ”€โ”€ 10.0.1.21 โ†’ Pod B (eth0 via veth) โ”‚ โ””โ”€โ”€ 10.0.1.22 โ†’ Pod C (eth0 via veth) โ”‚ โ””โ”€โ”€ vpc-cni (aws-node DaemonSet) manages ENI lifecycle via EC2 API How pod IPs work: Each EC2 instance can attach multiple ENIs; each ENI holds multiple secondary IPs vpc-cni pre-warms a pool of secondary IPs per node via EC2 API calls Pods receive a real VPC IP โ€” routable natively across the VPC, peered VPCs, VPNs, and Direct Connect โ€” with no overlay Pod density limits per node (examples): Instance Type Max ENIs Max IPs (pod limit) t3.medium 3 17 m5.large 3 29 m5.xlarge 4 58 m5.4xlarge 8 234 c5.18xlarge 15 750 โš ๏ธ Important: Default pod density is capped by the ENI/IP limit per instance type. For IP-constrained environments, use VPC CNI with prefix delegation (ENABLE_PREFIX_DELEGATION=true) to assign /28 prefixes instead of individual IPs, dramatically increasing pod density. Key features: Feature AWS VPC CNI IP assignment Native VPC secondary IPs from ENIs Overlay โœ— None โ€” native VPC routing NetworkPolicy โœ— Not built-in โ€” requires Calico or Cilium add-on Security Groups โœ… Security Groups for Pods (SGP) โ€” per-pod AWS SGs IPv6 โœ… Supported Prefix delegation โœ… /28 prefix per ENI (more pods per node) Windows nodes โœ… Supported Custom networking โœ… Pods in different subnet than node eBPF acceleration โœ… via Cilium add-on (EKS + Cilium mode) Enabling Network Policy on EKS: Calico (most common) โ€” install as an add-on alongside vpc-cni Cilium in chained mode โ€” replaces policy enforcement, keeps VPC IP routing Amazon VPC CNI Network Policy (AWS-native, GA as of 2024) โ€” uses eBPF for policy enforcement # Enable AWS-native network policy controller (EKS add-on) aws eks create-addon \ --cluster-name my-cluster \ --addon-name vpc-cni \ --configuration-values '{"nodeAgent":{"enablePolicyEventLogs":"true"}}' When to choose AWS VPC CNI: โœ… Running EKS โ€” it is the default and AWS-managed โœ… Need pods directly reachable from on-premises via Direct Connect / VPN โœ… Need per-pod AWS Security Groups (SGP feature) โœ… Compliance requires no overlay network โš ๏ธ Watch instance type ENI limits for large pod densities Azure Kubernetes Service (AKS) offers multiple CNI modes. The default for most production clusters is Azure CNI, which assigns pod IPs directly from the Azure Virtual Network (VNET) subnet โ€” similar in concept to AWS VPC CNI but using Azure's networking primitives. AKS CNI Modes: Mode Description Default? kubenet Basic overlay; nodes get VNET IPs, pods get private overlay IPs (NAT) Legacy default Azure CNI Pods get real VNET IPs from a pre-allocated subnet Current recommended default Azure CNI Overlay Pods get overlay IPs (larger scale, fewer VNET IPs needed) Recommended for large clusters Azure CNI + Cilium Azure CNI routing + Cilium eBPF dataplane + Hubble Recommended for policy/observability Bring Your Own CNI Disable Azure CNI; install Calico, Flannel, etc. Advanced Azure CNI (traditional): AKS Worker Node (Azure VM) โ”‚ โ”œโ”€โ”€ Primary NIC (node IP: 10.240.0.4) โ”‚ โ””โ”€โ”€ VNET: 10.240.0.0/16 โ”‚ โ””โ”€โ”€ Pod IPs pre-allocated from subnet: โ”œโ”€โ”€ 10.240.0.10 โ†’ Pod A โ”œโ”€โ”€ 10.240.0.11 โ†’ Pod B โ””โ”€โ”€ 10.240.0.12 โ†’ Pod C azure-vnet (CNI plugin) programs routes in Azure SDN Azure CNI Overlay (recommended for scale): # Create AKS cluster with Azure CNI Overlay + Cilium dataplane az aks create \ --resource-group myRG \ --name myAKS \ --network-plugin azure \ --network-plugin-mode overlay \ --network-dataplane cilium \ --pod-cidr 192.168.0.0/16 Key features: Feature kubenet Azure CNI Azure CNI Overlay Azure CNI + Cilium Pod IPs Overlay (NAT) Real VNET IPs Overlay (Azure SDN) Overlay (Azure SDN) IP exhaustion risk Low High Low Low Direct pod routing โœ— โœ… โœ… (via Azure SDN) โœ… NetworkPolicy Basic Azure Network Policy / Calico Azure NP / Calico โœ… Cilium (eBPF) Windows nodes โœ… โœ… โœ… โš ๏ธ Partial Hubble observability โœ— โœ— โœ— โœ… Max pods/node 110 250 250 250 Network Policy options on AKS: Azure Network Policy Manager (NPM) โ€” iptables-based, Azure-native, limited feature set Calico โ€” add-on, full L3/L4 policy, most commonly used Cilium โ€” available with Azure CNI Overlay mode, eBPF enforcement + Hubble When to choose Azure CNI: โœ… Running AKS โ€” Azure CNI Overlay is the modern recommended choice โœ… Need pods directly reachable from on-premises via ExpressRoute โœ… Want Hubble observability โ†’ use Azure CNI Overlay + Cilium dataplane โœ… Large clusters (100+ nodes) โ†’ use Overlay mode to avoid VNET IP exhaustion โš ๏ธ Traditional Azure CNI requires pre-allocating pod IPs per node โ€” plan subnet size carefully Google Kubernetes Engine (GKE) introduced Dataplane V2 in 2021, which is based on Cilium's eBPF engine. It is the default for new GKE clusters and brings production-grade eBPF networking, built-in NetworkPolicy enforcement, and a subset of Hubble observability โ€” all managed by Google. GKE networking modes: Mode Description Default? Legacy (iptables) kube-proxy + iptables, no Dataplane V2 Older clusters Dataplane V2 Cilium eBPF, managed by GKE, no full Cilium control plane Default for new clusters Dataplane V2 + Hubble Same + network telemetry via Hubble Optional add-on Architecture: GKE Node (GCE VM) โ”‚ โ”œโ”€โ”€ Alias IP range (VPC-native pod CIDRs) โ”‚ Pods get real VPC IPs, routed via Google SDN โ”‚ โ””โ”€โ”€ Dataplane V2 (Cilium eBPF engine) โ”œโ”€โ”€ TC eBPF hooks on veth interfaces โ”œโ”€โ”€ BPF maps for policy, NAT, LB โ”œโ”€โ”€ kube-proxy replaced by eBPF โ””โ”€โ”€ Hubble telemetry (if enabled) GKE uses VPC-native networking (alias IP ranges) โ€” pods get real VPC CIDRs routed natively through Google's Andromeda SDN. Dataplane V2 sits on top, adding eBPF policy enforcement and observability. Enabling Dataplane V2 on GKE: # Create GKE cluster with Dataplane V2 (default for new clusters) gcloud container clusters create my-cluster \ --enable-dataplane-v2 \ --enable-ip-alias \ --location us-central1 # Enable Hubble observability add-on gcloud container clusters update my-cluster \ --enable-dataplane-v2-flow-observability \ --location us-central1 Key features: Feature GKE Dataplane V2 Dataplane Cilium eBPF (managed subset) kube-proxy replacement โœ… eBPF NetworkPolicy โœ… eBPF-enforced (L3/L4) FQDN policy โœ… (GKE 1.28+) Hubble observability โœ… Optional add-on L7 policy โš ๏ธ Not exposed (managed limitations) Pod IPs Real VPC IPs (alias ranges) Windows nodes โœ… Multi-cluster โœ… via GKE Fleet / Anthos Managed lifecycle โœ… Google manages upgrades Dataplane V2 vs self-managed Cilium on GKE: Aspect GKE Dataplane V2 Self-managed Cilium on GKE Management Google-managed You manage Helm values/upgrades Feature exposure Subset of Cilium Full Cilium feature set Hubble Basic (add-on) Full Hubble UI + Relay Cluster Mesh โœ— (use GKE Fleet) โœ… L7 CNP โœ— โœ… Support GKE SLA Community / Isovalent ๐Ÿ’ก GKE Recommendation: For most workloads, Dataplane V2 is the right choice โ€” Google manages it, it's eBPF-based, and it covers L3/L4 policy. If you need full CiliumNetworkPolicy L7 rules or Cluster Mesh, consider self-managed Cilium on GKE with --network-plugin=cni and disabling kube-proxy. When to choose GKE Dataplane V2: โœ… Running GKE โ€” it is the default and Google-managed โœ… Want eBPF performance without managing Cilium yourself โœ… NetworkPolicy enforcement at scale (eBPF O(1) lookups) โœ… Need basic Hubble network telemetry โš ๏ธ For full L7 policy or Cluster Mesh, self-manage Cilium on GKE instead Services Flannel (iptables) Calico (iptables) Calico (eBPF) Cilium (eBPF) AWS VPC CNI Azure CNI GKE DPv2 100 ~10 ms ~10 ms \ --set k8sServicePort=6443 \ --set bpf.masquerade=true \ --set ipam.mode=kubernetes Feature Cilium Calico eBPF Basic CNI โ‰ฅ 4.9 Any kube-proxy replacement โ‰ฅ 5.2 โ‰ฅ 5.3 WireGuard encryption โ‰ฅ 5.6 โ‰ฅ 5.6 XDP acceleration โ‰ฅ 5.10 โ‰ฅ 5.10 โœ… Ubuntu 22.04 ships kernel 5.15, Debian 12 ships 6.1, Raspberry Pi OS Bookworm ships 6.1 โ€” all satisfy every requirement. All migrations follow the same pattern: drain โ†’ clean CNI state โ†’ restart K3s with --flannel-backend=none โ†’ install new CNI โ†’ uncordon # Step 1: Drain the node kubectl drain --ignore-daemonsets --delete-emptydir-data # Step 2: Remove Flannel state on the node systemctl stop k3s ip link delete flannel.1 2>/dev/null || true ip link delete cni0 2>/dev/null || true rm -rf /var/lib/cni /etc/cni/net.d # Step 3: Set flannel-backend: none in /etc/rancher/k3s/config.yaml, then restart systemctl start k3s # Step 4: Install Calico operator kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml # Step 5: Uncordon kubectl uncordon # Steps 1โ€“3 same as above (drain, clean, restart with flannel-backend=none) # Step 4: Install Cilium helm repo add cilium https://helm.cilium.io/ helm install cilium cilium/cilium \ --namespace kube-system \ --set kubeProxyReplacement=true \ --set k8sServiceHost= \ --set k8sServicePort=6443 # Step 5: Uncordon kubectl uncordon ๐Ÿ’ก Pro Tip: For single-node K3s lab environments, a clean reinstall is always faster and safer than a live migration. Run k3s-uninstall.sh, reinstall with the correct flags, then Helm install your chosen CNI โ€” total time is about 10 minutes. ๐ŸŸข Flannel โ€” A masterpiece of minimalism. One job, done perfectly, with near-zero operational overhead. The right choice when simplicity and RAM constraints matter more than policy or observability. ๐ŸŸ  Calico โ€” The policy-first CNI. BGP-native routing, mature L3/L4 NetworkPolicy, Windows node support, and a pluggable data plane. The right choice when you need robust policy enforcement, prefer no-encap routing, or operate in an environment with existing BGP infrastructure. ๐Ÿ”ต Cilium โ€” The platform CNI. eBPF-native with O(1) service lookup, L7-aware policy with no sidecar, Hubble observability, full pod-traffic encryption, and Cluster Mesh multi-cluster. The most capable networking layer available in Kubernetes today. ๐ŸŸก Weave Net โ€” Once a popular choice for simplicity and built-in encryption. Now archived โ€” migrate to Cilium or Calico for any new or long-running cluster. ๐ŸŸฃ Antrea โ€” The VMware-native CNI. OVS dataplane, tiered policy, Windows support, and NSX-T integration. The right choice in Tanzu or NSX environments. ๐Ÿ”ท Multus โ€” Not a CNI replacement but a CNI multiplier. Essential for telco/NFV workloads needing multiple pod network interfaces. Cloud Provider CNIs ๐Ÿ”ถ AWS VPC CNI (EKS) โ€” Native VPC IP assignment with no overlay. Pods are first-class VPC citizens. Add Calico or the AWS-native policy controller for NetworkPolicy. Choose prefix delegation for high pod density. ๐Ÿ”ท Azure CNI (AKS) โ€” Use Azure CNI Overlay for most production workloads to avoid IP exhaustion, and add the Cilium dataplane for eBPF policy + Hubble observability. Azure CNI traditional still works, but requires careful subnet pre-planning. โ™ฆ๏ธ GKE Dataplane V2 (GKE) โ€” Google's managed Cilium eBPF layer. The default for new GKE clusters. Handles NetworkPolicy at scale with eBPF O(1) lookups. Add the Hubble observability add-on for network telemetry. Self-manage Cilium on GKE only if you need L7 CNP or Cluster Mesh. Bottom line: If you run a managed Kubernetes service, use the cloud-default CNI and layer policy/observability on top. If you run self-managed clusters, Cilium is the most capable long-term investment, with Calico as the pragmatic choice if BGP integration or Windows nodes are required. The networking layer of your cluster is not where you want to cut corners at scale. Choose based on where your cluster is going โ€” not just where it is today. Cilium K3s Installation Guide Cilium Network Performance Calico on K3s โ€” Official Docs Flannel GitHub Flannel Networking Antrea GitHub AWS VPC CNI GitHub Azure CNI Overlay Docs GKE Dataplane V2 Overview Cilium Cluster Mesh Docs Calico GlobalNetworkPolicy Reference Antrea Network Policy Guide Written for K3s v1.29+, Cilium v1.15+, Calico v3.27+, Flannel v0.24+, AWS VPC CNI v1.18+, Azure CNI v1.5+, GKE 1.28+. Benchmark figures are representative โ€” always test with your own hardware and workload before production decisions.