Cilium ClusterMesh: Scaling Kubernetes Multi-Cluster Networking

Managing a single Kubernetes cluster is hard enough. Once you start scaling to multiple clusters—whether across regions, clouds, or just separate VPCs—the networking becomes a nightmare. You’re suddenly dealing with overlapping CIDRs, complex ingress routing, and service discovery that doesn't span boundaries.

I’ve spent the last few years dealing with this exact friction. That’s where Cilium ClusterMesh comes in. It’s not just another overlay; it’s a high-performance way to handle Kubernetes multi-cluster networking that leverages eBPF to bypass the kernel's overhead.

Why Cilium ClusterMesh?

Most traditional solutions rely on heavy service meshes or complex VPN/tunneling setups that kill your performance. Cilium uses eBPF to hook directly into the Linux kernel, allowing packets to move between clusters with near-native speed.

When you implement Cilium ClusterMesh, you’re essentially creating a flat, identity-aware network across all your clusters. It handles cross-cluster connectivity by assigning global identities to pods, meaning you can enforce security policies across clusters as if they were running on the same node.

Prerequisites

Before we dive into the config, ensure you have:

• Two or more Kubernetes clusters (v1.24+ recommended).
• Cilium CLI (v0.15+) installed.
• Helm (v3.0+) for deployment.
• Connectivity between nodes (Cilium handles the encryption, but the nodes need to see each other).

Setting Up the Mesh

First, you need to enable ClusterMesh on your clusters. I prefer using the Cilium CLI for this because it automates the certificate generation and the creation of the clustermesh-apiserver.

Run this on both clusters:

Bash
cilium clustermesh enable --context cluster1
cilium clustermesh enable --context cluster2

This command installs the necessary components to allow the clusters to communicate. It sets up a shared identity store so that when a pod in Cluster A tries to reach a service in Cluster B, the network knows exactly what that pod is allowed to do.

Connecting the Clusters

Now, we need to establish the peering. This is the part where most people get tripped up. You need to connect the clusters so they can exchange service information.

Bash
# Connect cluster2 to cluster1
cilium clustermesh connect --context cluster1 --destination-context cluster2

Once connected, Cilium automatically syncs the service metadata. You can verify the status by running:

Bash
cilium clustermesh status --context cluster1

You should see all connected clusters listed with a "Success" status.

Enabling Global Service Discovery

The real magic of Cilium ClusterMesh is service discovery. You don't want to manage separate DNS entries for every cluster. You want a global view.

To expose a service globally, you just add a specific annotation to your Kubernetes Service definition:

YAML
apiVersion: v1
kind: Service
metadata:
  name: my-backend-service
  annotations:
    io.cilium/global-service: "true"
spec:
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

When you apply this, Cilium creates a global service object. Any pod in any cluster in the mesh can now reach this service via the standard DNS name: my-backend-service.default.svc.cluster.local. Cilium handles the load balancing across the local and remote endpoints automatically.

Hard-Won Lessons

I’ve learned a few things the hard way while running this in production:

1. MTU Mismatch is Real: If you’re using different cloud providers or different VPC configurations, ensure your MTU settings are consistent. I’ve wasted hours debugging dropped packets that were simply too large for the tunnel.
2. Encryption: If you’re crossing public internet boundaries, enable IPsec or WireGuard encryption in Cilium. It’s a simple flag in your Helm values, and it saves you from needing an extra layer of VPN complexity.
3. Identity Sync: Remember that Cilium identities are based on labels. If your labels don't match between clusters, your network policies won't apply correctly. Keep your label schemas consistent across the entire fleet.

Why This Matters

By moving to eBPF networking via Cilium, you’re effectively removing the "network tax" that comes with multi-cluster setups. You get observability, security, and connectivity in one package. When a developer asks, "Can my service talk to the database in the other region?" you can finally say "Yes," without needing to configure a dozen ingress gateways or service mesh proxies.

It’s cleaner, it’s faster, and it’s significantly easier to debug than traditional overlay networks. If you're managing more than two clusters, start looking into this—it’ll save your weekends.