Kubernetes CRDs and Controller-Runtime: A Practical Guide to Operators

If you’ve ever found yourself manually syncing configurations or orchestrating complex state across your cluster, you’ve hit the ceiling of standard Kubernetes objects. That’s where Kubernetes CRDs and the Controller-Runtime library come in, allowing you to codify your operational knowledge into a custom operator.

In this guide, we’ll move beyond basic YAML manifests. We’ll build a custom automation engine using the Go Operator SDK to manage resources exactly how your infrastructure requires.

Why Build Custom Operators?

Standard controllers handle things like Deployments and Services, but they don't know your business logic. When you need to manage Kubernetes Secret Management, you might rely on existing operators, but for internal platform requirements, custom automation is often the only way to maintain consistency.

Think of an operator as a control loop. It watches the state of your cluster, compares it to your desired state, and executes actions to bridge the gap. By using the Controller-Runtime library, you get battle-tested scaffolding that handles the heavy lifting of caching, leader election, and event filtering.

Getting Started with the Go Operator SDK

First, ensure you have the Operator SDK (I'm using v1.30.0) and Go (1.21+) installed. We’ll initialize a project to define our custom API.

Bash
mkdir my-operator && cd my-operator
operator-sdk init --domain mahamudul.dev --repo github.com/mhrubel/my-operator
operator-sdk create api --group apps --version v1alpha1 --kind AppConfig

This command generates the boilerplate for your Kubernetes CRDs and the controller scaffolding. The api/v1alpha1/appconfig_types.go file is where you define your schema.

Implementing the Reconciliation Loop

The heart of your operator is the Reconcile function. This is where the magic happens. You don't just "do" things; you reconcile the state.

Go
func (r *AppConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Fetch the AppConfig instance
    appConfig := &appsv1alpha1.AppConfig{}
    if err := r.Get(ctx, req.NamespacedName, appConfig); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Add your custom automation logic here
    log.Info("Reconciling AppConfig", "name", appConfig.Name)

    // 3. Update status or create dependent resources
    return ctrl.Result{}, nil
}

The key here is idempotency. Your logic should be able to run hundreds of times without causing side effects. If your operator manages complex node lifecycle tasks, you might consider integrating Kubernetes Cluster API patterns to ensure you aren't fighting with existing cluster state controllers.

Advanced Patterns for Production

Once your basic controller is working, you need to think about observability and stability. You can’t debug a distributed system by guessing. I strongly recommend setting up Kubernetes Observability early in your development process so you can trace reconciliation requests through your custom controllers.

Also, consider how your CRDs interact with the rest of the cluster. If you’re managing resources that cross namespace boundaries, look at how Kubernetes Multi-Tenancy handles hierarchy to avoid permission nightmares.

Best Practices for Your Controller

• Use Finalizers: If your custom automation creates external resources (like cloud load balancers), use finalizers to ensure cleanup when a CRD is deleted.
• Watch Dependent Resources: Use builder.Owns(&appsv1.Deployment{}) in your controller setup to trigger a reconciliation whenever a managed resource changes.
• Limit Scope: Keep your controllers focused. One operator per domain is better than a "god-operator" that manages everything.

Summary

Building Kubernetes Operators is the ultimate way to level up your platform engineering game. By leveraging Controller-Runtime and the Go Operator SDK, you move from manual scripts to a declarative, self-healing system. Start small, focus on idempotency, and always prioritize observability to ensure your custom automation stays reliable under pressure.