Kubernetes Multi-cluster Service Discovery with Submariner Guide

Why Cross-Cluster Networking is a Nightmare

We’ve all been there. Your application outgrows a single cluster, or you need to split workloads across regions for compliance. Suddenly, your simple http://service-name calls stop working because the DNS resolution is trapped inside a single cluster boundary.

You could hack it with NodePorts or LoadBalancers, but that's a security and management disaster. That’s where Submariner comes in. It’s the industry standard for connecting Kubernetes clusters at the network layer.

The Architecture: How Submariner Works

Submariner creates a flat, encrypted overlay network between your clusters. It uses a "Gateway" node in each cluster to route traffic to the remote cluster's pod CIDR.

When you use the ServiceExport API, Submariner’s service-discovery component watches for these resources and automatically creates the necessary DNS entries in the remote clusters. It’s essentially a global service registry that doesn't force you to rewrite your application code.

Prerequisites

Before we dive in, make sure you're running:

• Kubernetes: v1.24+
• Submariner: v0.16.0
• subctl: The CLI tool for managing the installation.

Step 1: Installing Submariner

First, grab the subctl binary. I usually stick to the official release page. Once you've got it, initialize your broker. The broker acts as the central hub for cluster metadata.

Bash
# Initialize the broker in your management cluster
subctl deploy-broker

Now, join your clusters to the broker. You'll need the broker-info.subm file generated in the previous step.

Bash
subctl join broker-info.subm --clusterid cluster-a --clustercidr 10.42.0.0/16
subctl join broker-info.subm --clusterid cluster-b --clustercidr 10.43.0.0/16

Verify the connection:

Bash
subctl show connections

If you see a status of connected, you’ve successfully built the tunnel.

Step 2: Enabling Service Discovery

Submariner doesn't expose services by default. You have to opt-in using the ServiceExport resource. This is a custom resource definition (CRD) that tells Submariner, "Hey, make this service available to the rest of the fleet."

Let’s say you have a backend-api service in cluster-a.

YAML
apiVersion: net.submariner.io/v1alpha1
kind: ServiceExport
metadata:
  name: backend-api
  namespace: default

Apply this file:

Bash
kubectl apply -f service-export.yaml

Step 3: Accessing the Service

Once you've exported the service, Submariner automatically creates a corresponding ServiceImport resource in the other clusters. You can now access your service using a global DNS name:

service-name.namespace.svc.clusterset.local

For my backend-api example, the internal DNS from cluster-b will be: backend-api.default.svc.clusterset.local

This is the beauty of it. Your developers don't need to change their code or use hardcoded IPs. They just update their connection string to the clusterset.local domain, and the underlying network handles the routing.

Lessons from the Field

Don't ignore your MTU settings. When you wrap traffic in an overlay (especially IPsec), you lose overhead. If you're running on AWS or GCP, ensure your VPC subnets don't overlap with the Pod CIDRs you define in Submariner. I’ve spent countless hours debugging packet drops only to realize the VPC route table was fighting with the Submariner gateway.

Also, keep an eye on your Gateway nodes. If they go down, the inter-cluster traffic dies. I recommend labeling your gateway nodes with submariner.io/gateway=true and ensuring they have sufficient CPU/RAM to handle the encryption overhead.

Wrapping Up

Kubernetes multi-cluster networking is complex, but Submariner abstracts away the heavy lifting. By leveraging ServiceExport and a unified DNS namespace, you can treat your disparate clusters as one cohesive unit. Start small—connect two clusters in a staging environment—and watch how much easier it becomes to manage cross-cluster service discovery.