By apipark —

How to Make Your Controller Watch for CRD Changes

How to Make Your Controller Watch for CRD Changes
controller to watch for changes to crd
XRoute.AI
XPack.AI

Kubernetes has revolutionized how we deploy, manage, and scale applications in modern cloud-native environments. Its declarative nature and powerful extension mechanisms allow users to define desired states for their systems, with Kubernetes constantly working to reconcile the actual state with the desired state. At the heart of this reconciliation process lie controllers, automated loops that observe specific resources and take action to ensure the system behaves as expected. While Kubernetes provides a rich set of built-in resources like Deployments, Services, and Pods, its true power lies in its extensibility through Custom Resources (CRs) and Custom Resource Definitions (CRDs). These allow users to define their own high-level abstractions, tailored to their application's specific needs, effectively transforming Kubernetes into a domain-specific operating system.

However, merely defining a new CRD isn't enough. To breathe life into these custom resources, you need a controller that understands them, watches for their changes, and acts upon them. This is the essence of building a Kubernetes operator—a pattern for packaging, deploying, and managing a Kubernetes-native application. An operator encapsulates operational knowledge, automating tasks that would otherwise require manual intervention, such as scaling, upgrades, backups, and failure recovery, all driven by the state of your custom resources.

This comprehensive guide will meticulously walk you through the intricate process of developing a Kubernetes controller that actively watches for changes in your Custom Resources. We will demystify the core components involved, from the Kubernetes API Server's watch mechanism to the client-go library's Informers and Workqueues. By the end of this journey, you will possess a profound understanding of how to architect and implement robust, production-ready controllers, empowering you to extend Kubernetes to meet virtually any operational challenge.

1. Understanding Kubernetes Custom Resources and Controllers

Before we dive into the mechanics of watching for changes, it's paramount to establish a solid foundation of what Custom Resources and controllers fundamentally are within the Kubernetes ecosystem. These two concepts are inextricably linked, forming the bedrock of Kubernetes extensibility.

1.1 The Extensibility of Kubernetes: Custom Resource Definitions (CRDs)

Kubernetes, by design, is incredibly flexible. While it comes with a predefined set of fundamental objects like Pods, Services, Deployments, and ConfigMaps, its true power in extending its capabilities lies in Custom Resource Definitions (CRDs). A CRD is a special kind of resource that allows you to define your own resource types, making them first-class citizens in the Kubernetes API. When you create a CRD, you are essentially telling Kubernetes, "Hey, I'm introducing a new kind of object that I want you to manage, and here's how it should look."

Imagine you're developing an application that requires a specific database instance, say a PostgreSQL instance, to be provisioned and managed. Instead of manually creating Pods, Persistent Volumes, and Services, you could define a PostgreSQL CRD. This CRD would specify the schema for your custom PostgreSQL resource, including fields like database version, storage size, replication factor, and desired users. Once the CRD is registered with the Kubernetes API Server, you can then create instances of this PostgreSQL custom resource, just like you would create a Deployment or a Service.

Structure of a CRD:

A CRD manifest typically contains several key sections:

  • apiVersion and kind: Standard Kubernetes API object identifiers. For CRDs, apiVersion is usually apiextensions.k8s.io/v1 and kind is CustomResourceDefinition.
  • metadata: Contains standard object metadata like name. The name field for a CRD follows the format <plural-name>.<group>. For instance, postgress.database.example.com.
  • spec: This is the core of your CRD, defining its behavior and schema.
    • group: The API group to which your custom resource belongs (e.g., database.example.com). This helps organize and avoid naming conflicts.
    • names: Defines how your custom resource will be referred to. This includes plural (e.g., postgress), singular (e.g., postgres), kind (e.g., Postgres), and optionally shortNames (e.g., pg). The kind must be CamelCase and unique within the group.
    • scope: Specifies whether your custom resource is Namespaced (like Pods and Deployments) or Cluster (like Nodes and PersistentVolumes). Most custom resources are Namespaced.
    • versions: This is a crucial section where you define the versions of your custom resource and their schemas.
      • Each version (name) can have served: true (meaning the API server will serve this version) and storage: true (meaning this is the version stored in etcd). There must be exactly one storage version.
      • schema: This uses OpenAPI v3 validation schema to describe the structure of your custom resource's spec and status fields. This is incredibly powerful as it allows Kubernetes to validate incoming custom resource objects against your defined schema, preventing malformed or invalid configurations. You can define required fields, data types, ranges, patterns, and more.
      • subresources: Allows you to define status and scale subresources. The status subresource enables independent updates to the status field, improving concurrency. The scale subresource integrates with horizontal pod autoscalers.

Why CRDs are Powerful:

CRDs fundamentally extend the Kubernetes API itself. This means:

  1. kubectl Integration: You can use kubectl to create, get, list, update, and delete your custom resources just like built-in ones (e.g., kubectl get postgres).
  2. Declarative Management: You define the desired state of your custom resources using YAML, and Kubernetes works to achieve and maintain that state.
  3. Kubernetes-Native Tooling: All existing Kubernetes tools, like RBAC, kubectl, client libraries, and even API gateway integrations, understand and can interact with your custom resources. This unified approach simplifies operations.
  4. Abstraction: CRDs allow you to abstract away complex underlying infrastructure or application logic. Users interact with a simple, high-level custom resource, while the controller (which we'll discuss next) handles the intricate details.
  5. Ecosystem Integration: Operators built on CRDs integrate seamlessly into the broader Kubernetes ecosystem, leveraging existing patterns for monitoring, logging, and security.

1.2 The Heart of Automation: Kubernetes Controllers

A Kubernetes controller is a control loop that continuously monitors the state of your cluster and makes changes to drive the current state towards the desired state. Think of it like a thermostat in your home: it constantly reads the current temperature, compares it to your desired setting, and then turns the heating or cooling on or off as needed to maintain that desired temperature.

In Kubernetes, controllers watch specific resource types (e.g., Deployments, Pods, or your Custom Resources) through the Kubernetes API Server. When a change occurs (a resource is created, updated, or deleted), the controller is notified. It then fetches the current state of the relevant resources, compares it to the desired state (often defined in the resource's spec field), and executes reconciliation logic to bridge any gap. This might involve creating new resources, updating existing ones, or deleting obsolete ones.

Key Components of a Controller:

Every effective Kubernetes controller, especially those built using the client-go library, typically comprises several interconnected components that work in harmony:

  1. Informers: These are the primary mechanism for a controller to "watch" for changes. Informers abstract away the complexities of directly interacting with the Kubernetes API Server's watch API. Instead of sending repeated requests, Informers establish a long-lived connection to the API Server, receive events (Add, Update, Delete) for a specific resource type, and maintain a local, in-memory cache of these resources. This cache is crucial for performance, as the controller can query it directly rather than making numerous calls to the API Server.
  2. Event Handlers: When an Informer detects a change and updates its cache, it triggers event handlers. These handlers are functions you define, which receive the changed object (or old and new objects for updates) and typically add a "key" representing that object (e.g., namespace/name) to a workqueue.
  3. Workqueue: This is a robust, thread-safe queue that decouples the event handling logic from the reconciliation logic. When an event handler adds a key to the workqueue, the item waits to be processed. The workqueue ensures that each item is processed exactly once concurrently, handles retries with back-off mechanisms for transient errors, and prevents duplicate processing of the same item.
  4. Reconcile Function (Worker): This is the core business logic of your controller. One or more worker goroutines continuously pull items (keys) from the workqueue. For each key, the worker retrieves the corresponding object from the Informer's local cache. It then executes the reconciliation logic:
    • Fetch Current State: Retrieve all relevant resources from the cache or directly from the API Server.
    • Compare Desired vs. Actual: Analyze the spec of the custom resource (desired state) and compare it with the actual state of the child resources it manages (e.g., Deployments, Services).
    • Act: Create, update, or delete child resources to match the desired state.
    • Update Status: Update the status field of the custom resource to reflect the actual state and any conditions.
    • Error Handling: If an error occurs, the item might be re-added to the workqueue for a retry with exponential back-off.

By combining CRDs with controllers, you effectively create an autonomous system that understands and manages your application-specific infrastructure. This forms the foundation of the operator pattern, enabling powerful automation and simplifying complex operational tasks within Kubernetes.

2. The Mechanics of Watching CRD Changes

Understanding the conceptual roles of CRDs and controllers is the first step. The next, more technical step, is to delve into the actual mechanisms by which a controller observes changes in Custom Resources. This involves the Kubernetes API Server's event stream, the client-go library's Informers, and the crucial role of Workqueues.

2.1 The Kubernetes API Server: The Source of Truth

At the core of all Kubernetes operations is the API Server. It is the front end of the Kubernetes control plane, exposing the Kubernetes API and serving as the primary interface for users, external components, and internal cluster components to interact with the cluster. All custom resources, once defined by a CRD, are stored and managed by the API Server in its backing data store, etcd.

When a Custom Resource is created, updated, or deleted, these changes are persisted in etcd via the API Server. For a controller to react to these changes, it needs a mechanism to be notified. This is where the Kubernetes Watch API comes into play.

The Watch API:

The Kubernetes API Server provides a Watch API endpoint for every resource type. Instead of polling the API Server repeatedly (which would be inefficient and place a heavy load on the server), clients can establish a long-lived HTTP connection to the Watch API. This connection allows the API Server to stream events to the client whenever a change occurs to a watched resource.

Each event typically contains:

  • Type: Indicates the type of change – ADDED, MODIFIED, or DELETED.
  • Object: The actual resource object that was added, modified, or deleted.

While directly using the Watch API is possible, it presents several challenges for controller developers:

  1. Connection Management: What happens if the connection drops? The client needs to intelligently re-establish it, potentially from a specific resource version to avoid missing events.
  2. Resource Versions: To ensure consistency and avoid processing the same event twice or missing events, clients need to keep track of the resourceVersion for each object.
  3. Initial Listing: When a watch connection is established, the client typically needs an initial list of all existing resources to build its initial state, followed by events for subsequent changes.
  4. Performance: Directly watching numerous resource types can lead to a large number of open connections and event processing overhead for each controller instance.

These complexities are precisely why client-go provides a higher-level abstraction: Informers.

2.2 Informers: The Efficient Watchers

Informers are a foundational building block in client-go for building robust and performant Kubernetes controllers. They elegantly abstract away the complexities of the Watch API and provide a cached, event-driven mechanism for controllers to observe resources.

An Informer essentially performs two crucial tasks:

  1. Listing and Watching: It first performs a LIST operation to fetch all existing resources of a specific type. Then, it establishes a WATCH connection to the API Server, starting from the resourceVersion returned by the LIST operation. This ensures that no events are missed between the initial list and the start of the watch.
  2. Local Cache (Indexer): As events stream in (ADDED, MODIFIED, DELETED), the Informer updates its local, in-memory cache, known as an Indexer. This cache stores the latest state of all watched resources. The Indexer typically allows efficient retrieval of objects by namespace and name, or even by custom indices.

Why Informers are Essential:

  • Performance: By maintaining a local cache, Informers significantly reduce the load on the Kubernetes API Server. Controllers can query the cache directly for resource information, avoiding expensive and slow API calls for every reconciliation cycle. This is particularly critical in clusters with many controllers or high churn rates.
  • Reliability: Informers handle connection drops, re-establish watches with appropriate resourceVersions, and manage potential desynchronization between the cache and the API Server. They ensure that the cache eventually converges with the true state of the API Server.
  • Event-Driven: Instead of polling, Informers push events to registered handlers. This reactive model is more efficient and responsive.
  • Shared Informers: In a controller, you often need to watch multiple resource types (e.g., your custom resource, plus Deployments, Services, and Pods that it manages). A SharedInformerFactory allows multiple controllers or components within the same process to share a single Informer for a given resource type. This means only one LIST and WATCH connection is established for each resource type, further conserving resources and reducing API Server load.

When an Informer processes an event, it calls predefined Event Handlers:

  • AddFunc(obj interface{}): Called when a new resource is added.
  • UpdateFunc(oldObj, newObj interface{}): Called when an existing resource is modified.
  • DeleteFunc(obj interface{}): Called when a resource is deleted.

These handlers are where your controller begins its reconciliation process, typically by enqueuing the object's identifier into a workqueue.

2.3 Workqueues: Decoupling and Rate Limiting

The Workqueue (workqueue.RateLimitingInterface in client-go) is another critical component that acts as a buffer and a processing mechanism between the Informer's event handlers and the controller's reconciliation logic. Its primary role is to decouple the event reception from the event processing, ensuring robustness, fairness, and efficient resource utilization.

When an event handler (from an Informer) detects a change in a Custom Resource (or any other watched resource), it doesn't immediately trigger the reconciliation logic. Instead, it adds a "key" representing the changed object (typically namespace/name) to the workqueue.

Why Workqueues are Needed:

  1. Decoupling: Event handlers are often executed concurrently within the Informer's goroutine. Direct, synchronous reconciliation in the handler could block the Informer, leading to missed events or cache staleness. The workqueue ensures that event processing is asynchronous and doesn't impede the Informer.
  2. Concurrency Control: You might have multiple worker goroutines processing items from the workqueue. The workqueue ensures that each item is processed by only one worker at a time, preventing race conditions when reconciling a specific object.
  3. Idempotency: Controllers are designed to be idempotent; processing an event multiple times should have the same effect as processing it once. However, the workqueue helps manage the flow. If multiple events for the same object arrive quickly, the workqueue can deduplicate them, only processing the object once for a given batch of changes, thus reducing redundant work.
  4. Rate Limiting and Back-off: This is one of the most powerful features. If an error occurs during reconciliation (e.g., a temporary network issue, a conflicting update), the controller needs to retry. The workqueue allows you to re-add an item with a delay (exponential back-off), preventing a controller from constantly retrying a failing operation and overwhelming the API Server. This is crucial for stability.
  5. Fairness: The workqueue ensures that all items eventually get processed, and typically prioritizes older items, contributing to overall system fairness.

How it Works:

  1. Enqueueing: When an AddFunc, UpdateFunc, or DeleteFunc is triggered by an Informer, it typically calls workqueue.Add(key) or workqueue.AddRateLimited(key) to place the key (e.g., default/my-app-01) into the queue.
  2. Processing: Controller worker goroutines continuously call workqueue.Get() to retrieve the next item. Get() blocks until an item is available.
  3. Reconciliation: The worker then processes the item, which involves fetching the object from the Informer's cache using the key and executing the Reconcile function.
  4. Completion/Retry:
    • If reconciliation is successful, the worker calls workqueue.Forget(key) to remove the item from the queue's tracking.
    • If reconciliation fails (e.g., due to a transient error), the worker calls workqueue.AddRateLimited(key) to re-add the item to the queue with a delay, allowing for retries. It also calls workqueue.Done(key) to mark the item as processed for the current attempt.

The combination of the Kubernetes API Server's event stream, Informers for efficient watching and caching, and Workqueues for robust and rate-limited processing forms the backbone of any reliable Kubernetes controller. These components ensure that your controller can react promptly and resiliently to changes in your Custom Resources, driving your desired state effectively.

3. Building a Controller: A Practical Guide

Now that we've explored the theoretical underpinnings, let's roll up our sleeves and walk through the practical steps of building a controller. We'll use Go and the client-go library, which is the idiomatic way to develop Kubernetes controllers. For this example, we'll create a simple MyApp Custom Resource that manages a Kubernetes Deployment and Service.

3.1 Setting Up Your Development Environment

Before writing any code, ensure your development environment is properly configured.

  • Go Language: Install Go (version 1.16 or later is recommended).
  • Kubernetes Tools:
    • kubectl: For interacting with your Kubernetes cluster.
    • minikube or kind: For running a local Kubernetes cluster for development and testing. kind (Kubernetes in Docker) is often preferred for controller development due to its simplicity and speed.
  • Git: For version control.

Let's initialize a new Go module:

mkdir my-controller
cd my-controller
go mod init github.com/your-org/my-controller # Replace with your module path

Next, fetch the necessary client-go dependencies. client-go is not directly imported; instead, it's typically pulled as a dependency by higher-level frameworks or by explicitly importing its packages as needed.

go get k8s.io/client-go@kubernetes-1.28.0 # Use a specific version matching your cluster
go get k8s.io/apimachinery@kubernetes-1.28.0

(Note: Replace kubernetes-1.28.0 with a version tag that matches your Kubernetes cluster's major.minor version to avoid compatibility issues.)

3.2 Defining Your Custom Resource (CRD)

First, we need to define our MyApp Custom Resource. This CRD will specify the schema for our application, including the desired image, replica count, and port.

Create a file named config/crd/bases/mycrd.example.com_myapps.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.mycrd.example.com
spec:
  group: mycrd.example.com
  names:
    kind: MyApp
    listKind: MyAppList
    plural: myapps
    singular: myapp
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              type: object
              properties:
                image:
                  type: string
                  description: The container image for the application.
                replicas:
                  type: integer
                  minimum: 1
                  description: The desired number of replicas for the application.
                port:
                  type: integer
                  minimum: 1
                  maximum: 65535
                  description: The port the application listens on.
              required:
                - image
                - replicas
                - port
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
                  description: The number of available replicas running the application.
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      lastTransitionTime:
                        type: string
                        format: date-time
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    required:
                      - lastTransitionTime
                      - message
                      - reason
                      - status
                      - type
      subresources:
        status: {}

Next, define the Go types for your MyApp custom resource. These types will allow your controller to strongly type the objects it retrieves from the Kubernetes API and interact with their spec and status fields.

Create a directory api/v1alpha1 and a file api/v1alpha1/myapp_types.go:

package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// MyAppSpec defines the desired state of MyApp
type MyAppSpec struct {
    Image    string `json:"image"`
    Replicas int32  `json:"replicas"`
    Port     int32  `json:"port"`
}

// MyAppStatus defines the observed state of MyApp
type MyAppStatus struct {
    AvailableReplicas int32             `json:"availableReplicas,omitempty"`
    Conditions        []metav1.Condition `json:"conditions,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// MyApp is the Schema for the myapps API
type MyApp struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyAppSpec   `json:"spec,omitempty"`
    Status MyAppStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// MyAppList contains a list of MyApp
type MyAppList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []MyApp `json:"items"`
}

func init() {
    SchemeBuilder.Register(&MyApp{}, &MyAppList{})
}

(Note: The +kubebuilder markers are conventions if you were using controller-runtime or operator-sdk, but for a plain client-go example, they serve as useful comments for metav1.TypeMeta and subresource definitions.)

You'll also need to define SchemeBuilder and register your types. Create api/v1alpha1/zz_generated.deepcopy.go and api/v1alpha1/groupversion_info.go. For simplicity in this manual walkthrough, we'll manually create groupversion_info.go and skip auto-generation of deepcopy.go as it's typically handled by controller-gen.

api/v1alpha1/groupversion_info.go:

// Package v1alpha1 contains API Schema definitions for the mycrd v1alpha1 API group
// +kubebuilder:object:generate=true
// +groupName=mycrd.example.com
package v1alpha1

import (
    "k8s.io/apimachinery/pkg/runtime/schema"
    "sigs.k8s.io/controller-runtime/pkg/scheme"
)

var (
    // SchemeGroupVersion is group version used to register these objects
    SchemeGroupVersion = schema.GroupVersion{Group: "mycrd.example.com", Version: "v1alpha1"}

    // SchemeBuilder is used to add go types to the GroupVersionKind scheme
    SchemeBuilder = &scheme.Builder{GroupVersion: SchemeGroupVersion}

    // AddToScheme adds the types in this group-version to the given scheme.
    AddToScheme = SchemeBuilder.AddToScheme
)

(You'd also need a register.go and other boilerplate for a complete API project, but for client-go, the key is the types and the CRD manifest.)

3.3 Implementing the Controller Logic (Go with client-go)

Now, let's write the controller. This will involve setting up clients, informers, a workqueue, and the reconciliation loop.

Create main.go in your project root:

package main

import (
    "context"
    "flag"
    "fmt"
    "os"
    "time"

    "github.com/your-org/my-controller/api/v1alpha1" // Import your CRD types

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/schema"
    "k8s.io/apimachinery/pkg/util/intstr"
    "k8s.io/apimachinery/pkg/util/wait"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/workqueue"
    "k8s.io/klog/v2"

    // Import for CRD client
    "k8s.io/client-go/dynamic"
)

const (
    controllerAgentName = "my-app-controller"
    // Label key to identify resources managed by this controller
    myAppLabelKey = "mycrd.example.com/myapp"
)

// Controller struct defines our controller
type Controller struct {
    kubeclientset kubernetes.Interface // Client for built-in Kubernetes types (Deployments, Services)
    dynamicClient dynamic.Interface    // Client for Custom Resources (MyApp)
    myAppsLister  cache.Indexer        // Indexer for MyApps (our CRD)
    deploymentsLister cache.Indexer    // Indexer for Deployments
    servicesLister cache.Indexer      // Indexer for Services
    workqueue     workqueue.RateLimitingInterface
    informerFactory cache.SharedInformerFactory // For built-in types
    crdInformer     cache.SharedIndexInformer   // For MyApp CRD
    scheme          *runtime.Scheme             // Scheme for type conversions
}

// NewController creates a new sample controller
func NewController(
    kubeclientset kubernetes.Interface,
    dynamicClient dynamic.Interface,
    myAppsInformer cache.SharedIndexInformer,
    deploymentInformer cache.SharedIndexInformer,
    serviceInformer cache.SharedIndexInformer,
    scheme *runtime.Scheme) *Controller {

    klog.Info("Setting up event handlers...")

    controller := &Controller{
        kubeclientset: kubeclientset,
        dynamicClient: dynamicClient,
        myAppsLister:  myAppsInformer.GetIndexer(),
        deploymentsLister: deploymentInformer.GetIndexer(),
        servicesLister:    serviceInformer.GetIndexer(),
        workqueue:     workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
        crdInformer:   myAppsInformer,
        scheme:        scheme,
    }

    // Event handlers for our MyApp CRD
    myAppsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: controller.handleAddMyApp,
        UpdateFunc: func(oldObj, newObj interface{}) {
            controller.handleUpdateMyApp(oldObj, newObj)
        },
        DeleteFunc: controller.handleDeleteMyApp,
    })

    // Event handlers for Deployments and Services owned by our MyApps
    deploymentInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: controller.handleObject,
        UpdateFunc: func(old, new interface{}) {
            newDepl := new.(*appsv1.Deployment)
            oldDepl := old.(*appsv1.Deployment)
            if newDepl.ResourceVersion == oldDepl.ResourceVersion {
                // Periodic resync will send update events for the same object.
                // We do not want to process these if the object's resource version
                // has not changed, it means **no actual change** has happened.
                return
            }
            controller.handleObject(new)
        },
        DeleteFunc: controller.handleObject,
    })

    serviceInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: controller.handleObject,
        UpdateFunc: func(old, new interface{}) {
            newSvc := new.(*corev1.Service)
            oldSvc := old.(*corev1.Service)
            if newSvc.ResourceVersion == oldSvc.ResourceVersion {
                return
            }
            controller.handleObject(new)
        },
        DeleteFunc: controller.handleObject,
    })

    return controller
}

// Run will set up the event handlers for types we are interested in, as well
// as syncing informer caches and starting workers. It will block until ctx.Done() is closed.
func (c *Controller) Run(ctx context.Context, threadiness int) error {
    defer workqueue.ShutDown()

    klog.Info("Starting MyApp controller")

    // Wait for the caches to be synced before starting workers
    klog.Info("Waiting for informer caches to sync")
    if !cache.WaitForCacheSync(ctx.Done(), c.crdInformer.HasSynced, c.deploymentsLister.HasSynced, c.servicesLister.HasSynced) {
        return fmt.Errorf("failed to sync caches")
    }
    klog.Info("Informer caches synced")

    // Start workers
    for i := 0; i < threadiness; i++ {
        go wait.UntilWithContext(ctx, c.runWorker, time.Second)
    }

    klog.Info("Started workers")
    <-ctx.Done()
    klog.Info("Shutting down workers")

    return nil
}

// runWorker is a long-running function that will continually call the
// processNextWorkItem function in a loop.
func (c *Controller) runWorker(ctx context.Context) {
    for c.processNextWorkItem(ctx) {
    }
}

// processNextWorkItem will read a single item from the workqueue and
// attempt to process it, by calling the reconcile function.
func (c *Controller) processNextWorkItem(ctx context.Context) bool {
    obj, shutdown := c.workqueue.Get()

    if shutdown {
        return false
    }

    // We wrap this block in a func so we can defer c.workqueue.Done.
    err := func(obj interface{}) error {
        defer c.workqueue.Done(obj)
        var key string
        var ok bool
        // We expect strings to come off the workqueue. These are of the
        // form namespace/name. We do this as the delayed nature of the
        // workqueue means the items in the informer cache may no longer be
        // the same as when the item was added to the workqueue.
        if key, ok = obj.(string); !ok {
            // As the item in the workqueue is actually invalid, we call
            // Forget() to ensure it does not get re-added.
            c.workqueue.Forget(obj)
            klog.Errorf("expected string in workqueue but got %#v", obj)
            return nil
        }
        // Run the reconcile, passing it the namespace/name string.
        if err := c.reconcile(ctx, key); err != nil {
            // Put the item back on the workqueue with an error rate limit.
            c.workqueue.AddRateLimited(key)
            return fmt.Errorf("error reconciling '%s': %s, requeueing", key, err.Error())
        }
        // If no error occurs we Forget this item so it does not get re-added
        // to workqueue.
        c.workqueue.Forget(obj)
        klog.Infof("Successfully reconciled '%s'", key)
        return nil
    }(obj)

    if err != nil {
        klog.Error(err)
        return true
    }

    return true
}

// enqueueMyApp takes a MyApp resource and converts it into a namespace/name
// string which is then put onto the work queue. This method should *not* be
// passed object which may be a snapshot of the **API** server's state for
// the object because objects in the informer cache are immutable.
func (c *Controller) enqueueMyApp(obj interface{}) {
    var key string
    var err error
    if key, err = cache.MetaNamespaceKeyFunc(obj); err != nil {
        klog.Error(err, "couldn't get key for object")
        return
    }
    c.workqueue.Add(key)
}

// handleAddMyApp is called when a MyApp is added.
func (c *Controller) handleAddMyApp(obj interface{}) {
    klog.Info("MyApp ADDED detected. Enqueuing...")
    c.enqueueMyApp(obj)
}

// handleUpdateMyApp is called when a MyApp is updated.
func (c *Controller) handleUpdateMyApp(oldObj, newObj interface{}) {
    oldMyApp := oldObj.(*v1alpha1.MyApp)
    newMyApp := newObj.(*v1alpha1.MyApp)

    // If the resource versions are the same, it means no actual change occurred.
    // This happens with periodic resyncs.
    if oldMyApp.ResourceVersion == newMyApp.ResourceVersion {
        return
    }

    // Check if spec changed. If only status changed, we might not need to reconcile immediately,
    // but generally, it's safer to always reconcile on update.
    // For this example, we'll enqueue on any update.
    klog.Info("MyApp MODIFIED detected. Enqueuing...")
    c.enqueueMyApp(newObj)
}

// handleDeleteMyApp is called when a MyApp is deleted.
func (c *Controller) handleDeleteMyApp(obj interface{}) {
    klog.Info("MyApp DELETED detected. Enqueuing...")
    // Deletion events often come with a finalizer logic or garbage collection.
    // We'll still enqueue to ensure cleanup if needed.
    c.enqueueMyApp(obj)
}

// handleObject will take any resource that is a Deployment or Service owned by a MyApp
// and figures out which MyApp resource it belongs to and enqueues that MyApp.
func (c *Controller) handleObject(obj interface{}) {
    var object metav1.Object
    var ok bool
    if object, ok = obj.(metav1.Object); !ok {
        tombstone, ok := obj.(cache.DeletedFinalStateUnknown)
        if !ok {
            klog.Errorf("error decoding object, invalid type")
            return
        }
        object, ok = tombstone.Obj.(metav1.Object)
        if !ok {
            klog.Errorf("error decoding object tombstone, invalid type")
            return
        }
        klog.Infof("Recovered deleted object '%s' from tombstone", object.GetName())
    }
    klog.Infof("Processing object: %s", object.GetName())
    if ownerRef := metav1.Get=''OwnerReference(object.GetOwnerReferences(), v1alpha1.SchemeGroupVersion.WithKind("MyApp").Kind); ownerRef != nil {
        // Enqueue the owning MyApp
        c.enqueueMyApp(object)
        return
    }
    // If no owner reference, it's not managed by us.
}

// reconcile is the core logic of our controller. It takes the key (namespace/name)
// of a MyApp and ensures the cluster state matches the MyApp's spec.
func (c *Controller) reconcile(ctx context.Context, key string) error {
    namespace, name, err := cache.SplitMetaNamespaceKey(key)
    if err != nil {
        klog.Errorf("invalid resource key: %s", key)
        return nil
    }

    // 1. Get the MyApp resource from the informer's cache
    obj, exists, err := c.myAppsLister.GetByKey(key)
    if err != nil {
        klog.Errorf("failed to get MyApp '%s' from lister: %v", key, err)
        return err
    }

    var myApp *v1alpha1.MyApp
    if exists {
        // Convert the unstructured object (from dynamic client) to our MyApp type
        // This requires some careful type assertion/conversion
        // For simplicity, we assume the object in the indexer is already type-asserted or unmarshaled properly.
        // In a real controller, you'd unmarshal the unstructured.Unstructured object from dynamic client
        // or ensure your lister is configured for your specific type.
        // For now, let's assume obj is directly a *v1alpha1.MyApp if you use typed client for CRDs.
        // For dynamic client, we retrieve Unstructured, then convert.
        // This example implicitly assumes a typed client for myAppsLister for simplicity,
        // but typically with dynamic client you'd use unstructured.Unstructured and then convert.

        // To correctly get MyApp from dynamic client, we'd do something like:
        // myAppObj, err := c.dynamicClient.Resource(v1alpha1.SchemeGroupVersion.WithResource("myapps")).Namespace(namespace).Get(ctx, name, metav1.GetOptions{})
        // if err != nil { ... }
        // myApp = &v1alpha1.MyApp{}
        // err = runtime.DefaultUnstructuredConverter.FromUnstructured(myAppObj.UnstructuredContent(), myApp)
        // if err != nil { ... }

        // However, since `c.myAppsLister` is populated by `crdInformer` which is fed by `dynamicClient`,
        // the objects in `c.myAppsLister` are `Unstructured` objects.
        unstructuredObj, ok := obj.(runtime.Unstructured)
        if !ok {
            klog.Errorf("object is not Unstructured: %#v", obj)
            return fmt.Errorf("object %s is not Unstructured", key)
        }
        myApp = &v1alpha1.MyApp{}
        err = runtime.DefaultUnstructuredConverter.FromUnstructured(unstructuredObj.UnstructuredContent(), myApp)
        if err != nil {
            klog.Errorf("failed to convert Unstructured to MyApp: %v", err)
            return err
        }
    } else {
        // The MyApp object has been deleted from the cluster.
        // Perform cleanup of child resources if they still exist.
        klog.Infof("MyApp '%s' in workqueue no longer exists in cache. Assuming it was deleted. Performing cleanup.", key)
        // This is where you'd clean up associated Deployments/Services if owner references
        // and garbage collection didn't handle it. For this example, let's assume GC works.
        return nil
    }

    // 2. Manage the Deployment
    deployment, err := c.kubeclientset.AppsV1().Deployments(namespace).Get(ctx, name, metav1.GetOptions{})
    if errors.IsNotFound(err) {
        klog.Infof("Creating Deployment for MyApp '%s'", name)
        deployment = c.newDeployment(myApp)
        _, err = c.kubeclientset.AppsV1().Deployments(namespace).Create(ctx, deployment, metav1.CreateOptions{})
    } else if err != nil {
        klog.Errorf("failed to get Deployment '%s': %v", name, err)
        return err
    } else {
        // Check if deployment needs to be updated
        if deployment.Spec.Replicas != &myApp.Spec.Replicas ||
            deployment.Spec.Template.Spec.Containers[0].Image != myApp.Spec.Image {
            klog.Infof("Updating Deployment for MyApp '%s'", name)
            updatedDepl := c.newDeployment(myApp) // Generate new deployment spec
            deployment.Spec = updatedDepl.Spec     // Update only the spec, keep metadata
            _, err = c.kubeclientset.AppsV1().Deployments(namespace).Update(ctx, deployment, metav1.UpdateOptions{})
        }
    }
    if err != nil {
        klog.Errorf("failed to create/update Deployment for MyApp '%s': %v", name, err)
        return err
    }

    // 3. Manage the Service
    service, err := c.kubeclientset.CoreV1().Services(namespace).Get(ctx, name, metav1.GetOptions{})
    if errors.IsNotFound(err) {
        klog.Infof("Creating Service for MyApp '%s'", name)
        service = c.newService(myApp)
        _, err = c.kubeclientset.CoreV1().Services(namespace).Create(ctx, service, metav1.CreateOptions{})
    } else if err != nil {
        klog.Errorf("failed to get Service '%s': %v", name, err)
        return err
    } else {
        // Check if service needs to be updated (e.g. port change)
        if service.Spec.Ports[0].Port != myApp.Spec.Port {
            klog.Infof("Updating Service for MyApp '%s'", name)
            updatedSvc := c.newService(myApp)
            service.Spec.Ports = updatedSvc.Spec.Ports
            _, err = c.kubeclientset.CoreV1().Services(namespace).Update(ctx, service, metav1.UpdateOptions{})
        }
    }
    if err != nil {
        klog.Errorf("failed to create/update Service for MyApp '%s': %v", name, err)
        return err
    }

    // 4. Update the MyApp's Status
    // Only update status if the observed state differs from reported status
    if myApp.Status.AvailableReplicas != deployment.Status.AvailableReplicas {
        myAppCopy := myApp.DeepCopy() // Always work on a copy to avoid modifying objects in cache
        myAppCopy.Status.AvailableReplicas = deployment.Status.AvailableReplicas
        // Update condition, e.g., if deployment is ready
        if deployment.Status.AvailableReplicas == myApp.Spec.Replicas {
            myAppCopy.Status.Conditions = []metav1.Condition{
                {
                    Type:               "Ready",
                    Status:             metav1.ConditionTrue,
                    LastTransitionTime: metav1.Now(),
                    Reason:             "DeploymentAvailable",
                    Message:            fmt.Sprintf("Deployment has %d available replicas", deployment.Status.AvailableReplicas),
                },
            }
        } else {
            myAppCopy.Status.Conditions = []metav1.Condition{
                {
                    Type:               "Ready",
                    Status:             metav1.ConditionFalse,
                    LastTransitionTime: metav1.Now(),
                    Reason:             "DeploymentNotReady",
                    Message:            fmt.Sprintf("Deployment has %d/%d available replicas", deployment.Status.AvailableReplicas, myApp.Spec.Replicas),
                },
            }
        }

        // Patch the status subresource
        _, err = c.dynamicClient.Resource(v1alpha1.SchemeGroupVersion.WithResource("myapps")).Namespace(myAppCopy.Namespace).
            UpdateStatus(ctx, myAppCopy.DeepCopyObject().(runtime.Unstructured), metav1.UpdateOptions{})
        if err != nil {
            klog.Errorf("failed to update status for MyApp '%s': %v", name, err)
            return err
        }
        klog.Infof("MyApp '%s' status updated to %d available replicas.", name, myAppCopy.Status.AvailableReplicas)
    }

    return nil
}

// newDeployment creates a new Deployment for a MyApp resource.
func (c *Controller) newDeployment(myApp *v1alpha1.MyApp) *appsv1.Deployment {
    labels := map[string]string{
        "app":         "myapp",
        "controller":  myApp.Name,
        myAppLabelKey: myApp.Name,
    }
    return &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myApp.Name,
            Namespace: myApp.Namespace,
            Labels:    labels,
            OwnerReferences: []metav1.OwnerReference{
                *metav1.NewControllerRef(myApp, v1alpha1.SchemeGroupVersion.WithKind("MyApp")),
            },
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &myApp.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: labels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: labels,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name:  "myapp",
                            Image: myApp.Spec.Image,
                            Ports: []corev1.ContainerPort{
                                {
                                    ContainerPort: myApp.Spec.Port,
                                },
                            },
                        },
                    },
                },
            },
        },
    }
}

// newService creates a new Service for a MyApp resource.
func (c *Controller) newService(myApp *v1alpha1.MyApp) *corev1.Service {
    labels := map[string]string{
        "app":         "myapp",
        "controller":  myApp.Name,
        myAppLabelKey: myApp.Name,
    }
    return &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      myApp.Name,
            Namespace: myApp.Namespace,
            Labels:    labels,
            OwnerReferences: []metav1.OwnerReference{
                *metav1.NewControllerRef(myApp, v1alpha1.SchemeGroupVersion.WithKind("MyApp")),
            },
        },
        Spec: corev1.ServiceSpec{
            Selector: labels,
            Ports: []corev1.ServicePort{
                {
                    Protocol:   corev1.ProtocolTCP,
                    Port:       myApp.Spec.Port,
                    TargetPort: intstr.FromInt32(myApp.Spec.Port),
                },
            },
            Type: corev1.ServiceTypeClusterIP,
        },
    }
}

func main() {
    klog.InitFlags(nil)
    flag.Parse()

    // Get a kubeconfig file location from the command line argument
    var kubeconfig *string
    if home := os.Getenv("HOME"); home != "" {
        kubeconfig = flag.String("kubeconfig", fmt.Sprintf("%s/.kube/config", home), "(optional) absolute path to the kubeconfig file")
    } else {
        kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
    }

    // Build the Kubernetes client configuration
    config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
    if err != nil {
        klog.Warningf("Error building kubeconfig: %s. Falling back to in-cluster config.", err.Error())
        config, err = rest.InClusterConfig()
        if err != nil {
            klog.Fatalf("Error building in-cluster config: %s", err.Error())
        }
    }

    // Create a typed client for built-in resources
    kubeClient, err := kubernetes.NewForConfig(config)
    if err != nil {
        klog.Fatalf("Error building kubernetes clientset: %s", err.Error())
    }

    // Create a dynamic client for Custom Resources
    dynamicClient, err := dynamic.NewForConfig(config)
    if err != nil {
        klog.Fatalf("Error building dynamic client: %s", err.Error())
    }

    // Create a scheme to register our types
    scheme := runtime.NewScheme()
    _ = corev1.AddToScheme(scheme)
    _ = appsv1.AddToScheme(scheme)
    _ = v1alpha1.AddToScheme(scheme) // Register our custom types

    // Set up informers for built-in resources (Deployments, Services)
    kubeInformerFactory := cache.NewSharedInformerFactory(kubeClient, time.Second*30) // Resync every 30 seconds

    // Set up informer for our MyApp CRD using dynamic client
    myAppGVR := v1alpha1.SchemeGroupVersion.WithResource("myapps") // GroupVersionResource for MyApps
    crdInformer := cache.NewSharedIndexInformer(
        &cache.ListWatch{
            ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
                return dynamicClient.Resource(myAppGVR).List(context.TODO(), options)
            },
            WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
                return dynamicClient.Resource(myAppGVR).Watch(context.TODO(), options)
            },
        },
        &v1alpha1.MyApp{}, // We use our specific type here for the informer, but objects come as Unstructured
        time.Second*30,    // Resync period
        cache.Indexers{},  // No custom indexers for now
    )

    controller := NewController(
        kubeClient,
        dynamicClient,
        crdInformer,
        kubeInformerFactory.Apps().V1().Deployments().Informer(),
        kubeInformerFactory.Core().V1().Services().Informer(),
        scheme,
    )

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // Start all informers
    go kubeInformerFactory.Start(ctx.Done())
    go crdInformer.Run(ctx.Done())

    if err = controller.Run(ctx, 2); err != nil { // Run with 2 worker threads
        klog.Fatalf("Error running controller: %s", err.Error())
    }
}

This main.go file outlines the entire controller logic:

  1. Client Initialization: It sets up kubernetes.Interface for standard Kubernetes resources and dynamic.Interface for Custom Resources. The dynamic.Interface is crucial because custom resources are Unstructured objects from the perspective of the generic Kubernetes API.
  2. Scheme Registration: It registers our MyApp Go types with a runtime scheme, allowing for type conversions.
  3. Informer Setup:
    • A SharedInformerFactory is created for standard Kubernetes resources (Deployments, Services).
    • A NewSharedIndexInformer is specifically set up for our MyApp CRD, using the dynamicClient's List/Watch functions. The cache.NewSharedIndexInformer requires a concrete Go type (&v1alpha1.MyApp{}), but the objects it internally receives are Unstructured which then need conversion in the reconciliation loop.
  4. Event Handlers:
    • handleAddMyApp, handleUpdateMyApp, handleDeleteMyApp: These are registered with the MyApp informer. They simply enqueue the namespace/name key of the changed MyApp object into the workqueue.
    • handleObject: This handler is registered for Deployment and Service informers. It checks if the changed Deployment/Service has an OwnerReference pointing back to a MyApp. If it does, it enqueues the owning MyApp to trigger a reconciliation. This is vital for reacting to changes in managed child resources.
  5. Workqueue: A workqueue.NewRateLimitingQueue is initialized to handle items for reconciliation.
  6. Reconcile Function: This is the heart of the controller.
    • It retrieves the MyApp object from the myAppsLister (which is the cache of our CRD informer). Note the conversion from runtime.Unstructured to v1alpha1.MyApp.
    • It then checks for the existence of the corresponding Deployment and Service.
    • If they don't exist, they are created.
    • If they exist but their spec (image, replicas, port) doesn't match the MyApp's spec, they are updated.
    • Crucially, metav1.NewControllerRef is used to establish OwnerReferences, ensuring that Kubernetes' garbage collector automatically cleans up Deployments and Services when their owning MyApp is deleted.
    • Finally, it updates the status field of the MyApp to reflect the current state of its child resources (e.g., availableReplicas). This ensures the MyApp itself provides feedback to users.
  7. Run Loop: The Run method starts the informers, waits for their caches to sync, and then starts multiple worker goroutines (runWorker) that continuously pull items from the workqueue and call the reconcile function.

3.4 Deployment of Your Controller and CRD

To deploy your controller, you'll need a few YAML manifests.

1. Deploy the CRD:

kubectl apply -f config/crd/bases/mycrd.example.com_myapps.yaml

Verify: kubectl get crd myapps.mycrd.example.com

2. Create RBAC for Your Controller:

Your controller needs permissions to interact with MyApp CRs, Deployments, and Services.

config/rbac/role.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-app-controller-role
rules:
  - apiGroups: ["mycrd.example.com"]
    resources: ["myapps", "myapps/status"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["services", "pods"] # Pods for status checks, services for management
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"] # For publishing events

config/rbac/service_account.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-controller-sa
  namespace: default # Or the namespace where you deploy your controller

config/rbac/role_binding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-app-controller-rb
subjects:
  - kind: ServiceAccount
    name: my-app-controller-sa
    namespace: default # Must match the namespace of the ServiceAccount
roleRef:
  kind: ClusterRole
  name: my-app-controller-role
  apiGroup: rbac.authorization.k8s.io

Apply RBAC:

kubectl apply -f config/rbac/service_account.yaml
kubectl apply -f config/rbac/role.yaml
kubectl apply -f config/rbac/role_binding.yaml

3. Deploy Your Controller:

You'll need to build your Go application into a Docker image and deploy it as a Kubernetes Deployment.

First, build the Docker image:

docker build -t your-repo/my-controller:v1 . # Replace with your repo
docker push your-repo/my-controller:v1

config/controller/deployment.yaml:

apiVersion: apps.k8s.io/v1
kind: Deployment
metadata:
  name: my-app-controller
  namespace: default
  labels:
    app: my-app-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app-controller
  template:
    metadata:
      labels:
        app: my-app-controller
    spec:
      serviceAccountName: my-app-controller-sa
      containers:
        - name: controller
          image: your-repo/my-controller:v1 # Your image here
          imagePullPolicy: IfNotPresent
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          # Add resource limits, probes for production readiness

Apply Controller Deployment:

kubectl apply -f config/controller/deployment.yaml

4. Create an Instance of Your Custom Resource:

Now, let's test our controller by creating a MyApp instance.

config/samples/myapp_v1alpha1_myapp.yaml:

apiVersion: mycrd.example.com/v1alpha1
kind: MyApp
metadata:
  name: my-nginx-app
  namespace: default
spec:
  image: nginx:latest
  replicas: 3
  port: 80

Apply your Custom Resource:

kubectl apply -f config/samples/myapp_v1alpha1_myapp.yaml

Observe the magic:

kubectl get myapp my-nginx-app -o yaml
kubectl get deployment my-nginx-app
kubectl get service my-nginx-app

You should see your controller create a Deployment and a Service named my-nginx-app, and the status of your MyApp object should reflect the availableReplicas of the deployment. If you modify the replicas or image in the MyApp's spec and re-apply, the controller will detect the change and update the Deployment. If you delete the MyApp, the Deployment and Service will be garbage collected automatically thanks to OwnerReferences.

This practical guide provides a robust starting point for building your own Kubernetes controllers. While this example focuses on basic reconciliation, the principles extend to far more complex scenarios, forming the backbone of powerful Kubernetes operators.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇
Install APIPark – it’s free

4. Advanced Controller Concepts and Best Practices

Building a basic controller is a significant achievement, but creating a production-grade operator requires delving into more advanced concepts and adhering to best practices. These elements ensure your controller is resilient, efficient, and maintainable.

4.1 Idempotency and Edge Cases

A fundamental principle for any Kubernetes controller is idempotency. This means that applying the same reconciliation logic multiple times, with the same desired state, should always result in the same actual state without causing unintended side effects. Your reconcile function must be designed to compare the desired state (from your CR's spec) with the current actual state and only perform actions if there's a discrepancy. This prevents redundant API calls and ensures stability.

Consider edge cases:

  • Concurrent Updates: Multiple controllers (or even human operators) might try to modify the same resource simultaneously. Your controller should handle conflicts gracefully (e.g., using resourceVersion for optimistic locking or retrying on conflict errors).
  • Partial Failures: What if a child resource creation fails? The workqueue's rate-limiting retry mechanism is crucial here. The controller should re-enqueue the item and try again later.
  • Deletion Race Conditions: A resource might be deleted while your controller is processing it. Your reconcile function must be able to detect that the resource no longer exists and cease operations, or perform cleanup if it was the last known state before deletion. The handleObject logic for child resources demonstrates this, looking for DeletedFinalStateUnknown tombstones.

4.2 Owner References and Garbage Collection

We briefly touched upon OwnerReferences in the practical example. This is a critical Kubernetes mechanism for managing the lifecycle of related resources. When you set an OwnerReference on a child resource (e.g., a Deployment) pointing to a parent Custom Resource (e.g., your MyApp), you establish a parent-child relationship.

The Kubernetes garbage collector leverages this. When the owning MyApp is deleted, the garbage collector automatically deletes all its dependents (Deployments, Services, etc.) that have ownerReference.blockOwnerDeletion: true and ownerReference.controller: true set. This prevents orphaned resources and simplifies cleanup, making your operator more robust. Always ensure your controller sets proper OwnerReferences when creating child resources.

4.3 Finalizers: Graceful Deletion

While OwnerReferences handle cascading deletion, sometimes a controller needs to perform specific cleanup logic before a Custom Resource is fully removed from the Kubernetes cluster. For example, if your MyApp provisions external cloud resources (like a database instance in AWS RDS), you'd want to de-provision those resources before the MyApp object itself is gone. This is where Finalizers come in.

A finalizer is a string added to the metadata.finalizers list of an object. When an object with finalizers is deleted, Kubernetes doesn't immediately remove it. Instead, it sets the object's metadata.deletionTimestamp and sends a DELETE event. Your controller, upon observing this deletionTimestamp, knows it's time to perform cleanup. Once the cleanup is complete, your controller removes the finalizer string. Only when the finalizers list is empty will Kubernetes finally delete the object.

Implementing finalizers typically involves:

  1. Adding a finalizer to your CR's metadata when it's created (or on the first update).
  2. In your reconcile loop, checking for deletionTimestamp.
  3. If deletionTimestamp is set and your finalizer is present, execute cleanup logic.
  4. Once cleanup is done, remove your finalizer from the metadata and update the CR.

4.4 Controller Runtime and Operator SDK

While building a controller directly with client-go provides a deep understanding of its inner workings, it involves a lot of boilerplate code. Frameworks like Controller Runtime and Operator SDK abstract away much of this complexity, providing higher-level APIs and tools to streamline controller development.

Feature client-go (Raw) Controller Runtime Operator SDK
Learning Curve High (requires deep understanding of informers, workqueues, dynamic client, etc.) Medium (simplified client-go interactions, opinionated structure) Low (generates scaffold, integrates with KubeBuilder)
Boilerplate Extensive (manual setup of informers, workqueues, event handlers, leader election) Significantly reduced (automatic informer/manager setup, reconciliation loop) Minimal (scaffolds project, CRD, controller code)
Clients kubernetes.Interface, dynamic.Interface client.Client (unified, cached client), dynamic.Client client.Client
Reconcile Loop Manual processNextWorkItem and reconcile Reconciler interface, managed by Manager Reconciler interface, code generation
Custom CRDs Requires dynamic client, manual type conversion Integrated scheme.AddToScheme, direct typed access Integrated code generation for CRD types
Testing Manual setup of fake clients, complex Built-in test environment for controllers Integrated testing tools
Community/Ecosystem Fundamental, but raw De facto standard for Go controllers, active Built on Controller Runtime, adds CLI and opinionated structure
Use Cases Deep dives, highly specialized or minimal footprint controllers Most common Go controllers, operators Rapid operator development, complex application lifecycle management

Controller Runtime: This library provides core components for building Kubernetes controllers, including a Manager that handles shared informers, caches, and dependency injection. It offers a powerful client.Client that unifies access to both built-in and custom resources and manages caching for you. It significantly reduces the amount of boilerplate needed for common tasks like leader election, metrics, and health checks.

Operator SDK: Built on top of Controller Runtime (and leveraging KubeBuilder), Operator SDK provides a command-line interface (CLI) that helps developers quickly scaffold operator projects. It generates CRD definitions, Go types, controller boilerplate, and deployment manifests. It also offers advanced features like Helm and Ansible operator support, and OLM (Operator Lifecycle Manager) integration.

For most new controller development in Go, using Controller Runtime (or Operator SDK which uses Controller Runtime) is highly recommended. It allows you to focus on your core reconciliation logic rather than managing low-level client-go details.

4.5 Error Handling and Observability

A robust controller must be observable. This involves:

  • Structured Logging: Use klog (or a structured logging library) to output informative logs. Include contextual information like the resource namespace and name, the operation being performed, and any errors encountered. This is invaluable for debugging in production.
  • Metrics: Expose Prometheus metrics from your controller. Track:
    • The number of successful/failed reconciliations.
    • The duration of reconciliation cycles.
    • Workqueue depth and processing time.
    • Informers' cache sync status. This allows you to monitor the health and performance of your controller over time.
  • Events API: Kubernetes has a built-in Event API. Your controller can publish Kubernetes events (e.g., "MyAppCreated", "DeploymentScaled", "ReconciliationFailed") associated with your Custom Resources or child resources. These events are visible via kubectl describe and provide a timeline of important actions and status changes, making it easier for users to understand what your operator is doing.
  • Health Checks: Implement liveness and readiness probes for your controller pod to ensure it's running correctly and ready to serve requests.

4.6 Performance Considerations

As your cluster grows and your controller manages more resources, performance becomes critical.

  • Efficient Caching: Informers are your best friend here. Leverage them fully. Avoid direct API Server calls within your reconcile loop unless absolutely necessary (e.g., for very fresh data or to ensure a specific API call occurs). Rely on the Informer's local cache (lister) for GET and LIST operations.
  • Batching Updates: If your controller needs to make multiple changes to child resources or update the CR status, try to batch these into a single API call (e.g., using strategic merge patches) rather than multiple individual updates, where possible, to reduce API Server load.
  • Watch Filters: For specific use cases, you can apply filters to your informers (e.g., labelSelector, fieldSelector) to only watch for resources that match certain criteria, reducing the amount of data processed.
  • Rate Limiting: The workqueue's rate-limiting is essential for error handling but also prevents your controller from overwhelming the API Server during periods of high churn or instability. Configure it appropriately.
  • Resource Management: Ensure your controller pod has appropriate CPU and memory requests and limits to prevent resource exhaustion or throttling.

By adopting these advanced concepts and best practices, you can develop Kubernetes controllers that are not only functional but also reliable, efficient, and easy to operate in production environments.

5. Kubernetes Controllers in the Broader Ecosystem

Kubernetes controllers are not isolated components; they operate within a rich ecosystem, interacting with various parts of the cluster and often with external systems. Understanding this broader context is key to designing powerful and well-integrated operators.

5.1 The Role of Controllers in Cloud-Native Operations

The advent of Kubernetes controllers, especially those based on Custom Resources, has profoundly reshaped cloud-native operations. They move operational knowledge from human playbooks and scripts into automated, declarative code living directly within the Kubernetes cluster. This shift offers several advantages:

  • Self-Healing Systems: Controllers constantly monitor for discrepancies between desired and actual states. If a managed resource fails (e.g., a database pod crashes), the controller automatically detects it and takes corrective action, reducing manual intervention and improving system uptime.
  • Application-Specific Automation: Instead of managing generic Kubernetes primitives, operators allow you to define high-level application abstractions (like MyApp, PostgreSQL, KafkaTopic). The controller then understands how to provision, configure, and manage all the underlying Kubernetes resources required for that application.
  • Consistent Deployments: Operators ensure that applications are deployed and managed consistently across different environments, adhering to predefined best practices encoded in the controller's logic.
  • Reduced Operational Burden: By automating routine tasks like scaling, upgrades, backups, and disaster recovery, operators free up human operators to focus on more complex, high-value work.

This paradigm transforms Kubernetes from a container orchestrator into a powerful application platform capable of managing virtually any workload, internal or external.

5.2 Connecting with External Systems

One of the most compelling use cases for Kubernetes controllers is their ability to act as a bridge between the Kubernetes cluster and external systems. While controllers primarily manage resources within Kubernetes, they often need to interact with external APIs, cloud providers, databases, or even on-premises systems to fulfill the desired state defined by a Custom Resource.

For instance, a controller managing a CloudDatabase CR might:

  • Call a cloud provider's API (e.g., AWS RDS API) to provision a new database instance.
  • Update a DNS record in an external DNS service.
  • Integrate with an external identity provider for user management.
  • Interact with a specialized machine learning service or a large language model.

This interaction with external APIs introduces a new layer of complexity and a critical need for robust API management.

When a controller needs to interact with various external AI models or REST services, an efficient and robust API gateway becomes indispensable. An API gateway acts as a single entry point for all client requests, routing them to the appropriate backend service. It can handle common tasks such as authentication, rate limiting, traffic management, and protocol translation, freeing the controller from these concerns.

Products like APIPark provide an open-source AI gateway and API management platform that simplifies the integration, deployment, and management of diverse APIs, including over 100 AI models. This ensures that controllers can reliably and securely communicate with necessary external resources, benefiting from features like unified API formats for AI invocation, prompt encapsulation into REST API, and comprehensive end-to-end API lifecycle management. By offloading these responsibilities to a dedicated API gateway, the controller can focus purely on its core reconciliation logic, enhancing modularity, security, and scalability. This external API management is particularly critical when a controller must interact with a multitude of heterogenous services, each with its own authentication and API specificities.

5.3 Kubernetes as an Application Platform

The combination of CRDs and controllers elevates Kubernetes beyond a mere container orchestrator; it transforms it into a flexible, extensible application platform. By defining custom resources that represent entire applications or complex infrastructure components, developers can abstract away the underlying operational details. This allows users to deploy and manage sophisticated applications using simple, declarative manifests, just as they manage a single Pod or Deployment.

This concept of "Kubernetes-native" applications implies that the application's lifecycle (provisioning, scaling, updating, healing, deleting) is fully managed by Kubernetes controllers, driven by the desired state defined in custom resources. This approach fosters a powerful ecosystem where:

  • Vendor Ecosystem: Cloud providers and software vendors can offer their services as Kubernetes operators, allowing users to consume them directly from within their clusters.
  • Internal Platforms: Enterprises can build internal platforms by defining custom resources for their unique services and automating their management with operators.
  • Unified Control Plane: Operations teams gain a unified control plane for managing both infrastructure and applications, using kubectl and the Kubernetes API for everything.

In essence, CRDs provide the nouns (what you want to manage), and controllers provide the verbs (how to manage it). Together, they enable Kubernetes to speak the language of your applications and infrastructure, making it an incredibly powerful and adaptable platform for the cloud-native era.

Conclusion

The journey of understanding and implementing a Kubernetes controller that watches for Custom Resource Definition (CRD) changes is a profound exploration into the heart of Kubernetes' extensibility. We've traversed from the fundamental concepts of CRDs as extensions of the Kubernetes API to the intricate mechanics of how controllers observe and reconcile desired states using Informers and Workqueues. The practical guide meticulously illustrated how to build a controller in Go with client-go, demonstrating the creation of child resources, setting owner references, and updating status—all driven by changes to a custom MyApp resource.

We further explored advanced topics, emphasizing the critical importance of idempotency, the graceful deletion facilitated by owner references and finalizers, and the architectural benefits of leveraging frameworks like Controller Runtime and Operator SDK. Moreover, we highlighted the necessity of robust error handling, comprehensive observability through logging and metrics, and strategic performance considerations for production readiness.

Ultimately, the power of CRDs and controllers lies in their ability to transform Kubernetes into a truly application-aware and domain-specific operating system. By encapsulating operational knowledge within code, operators automate complex tasks, enhance system resilience, and bridge the gap between abstract application requirements and concrete infrastructure configurations. This capability not only simplifies the deployment and management of cloud-native applications but also extends the Kubernetes control plane to interact seamlessly with external services, often facilitated by robust API gateways like APIPark.

As the cloud-native landscape continues to evolve, the demand for sophisticated automation and Kubernetes-native application management will only grow. Mastering the art of building controllers is not just a technical skill; it's an investment in the future of scalable, self-managing, and resilient systems. Empower yourself to extend Kubernetes, build custom abstractions, and become an architect of the next generation of cloud-native applications.

5 Frequently Asked Questions (FAQs)

1. What is the primary purpose of a Kubernetes controller watching for CRD changes? The primary purpose is to automate the management of application-specific resources and operational tasks within a Kubernetes cluster. By watching for changes in a Custom Resource (CR) (which is defined by a CRD), a controller can ensure that the desired state specified in the CR's manifest is continuously reconciled with the actual state of the cluster, creating, updating, or deleting underlying Kubernetes objects (like Deployments, Services, PersistentVolumes) or even interacting with external systems to achieve that desired state.

2. How do Informers and Workqueues contribute to an efficient controller? Informers significantly improve efficiency by abstracting away the complexities of the Kubernetes API Server's Watch API. They maintain a local, in-memory cache of resources, reducing the load on the API Server by allowing controllers to query the cache instead of making repeated API calls. Workqueues further enhance efficiency and reliability by decoupling event reception from processing. They provide a rate-limited, fault-tolerant mechanism to process events, ensuring that items are processed exactly once, handling retries with exponential back-off for transient errors, and preventing the controller from overwhelming the API Server during high churn.

3. What is an API gateway, and why might a Kubernetes controller need one? An API gateway acts as a single entry point for external API calls, routing them to appropriate backend services while handling cross-cutting concerns like authentication, rate limiting, and traffic management. A Kubernetes controller might need an API gateway (like APIPark) when its reconciliation logic requires interaction with various external REST services, AI models, or cloud provider APIs. An API gateway simplifies these external integrations by providing a unified interface, standardizing API formats, and managing the lifecycle of these external API interactions, allowing the controller to focus solely on its core reconciliation logic.

4. What are Owner References and Finalizers, and why are they important for controllers? Owner References establish a parent-child relationship between Kubernetes objects. They are crucial for automatic garbage collection, ensuring that when a parent resource (e.g., a Custom Resource) is deleted, all its dependent child resources (e.g., Deployments, Services) are also automatically cleaned up by Kubernetes. Finalizers, on the other hand, allow a controller to perform specific pre-deletion cleanup logic (e.g., de-provisioning external cloud resources) before a Custom Resource is fully removed from the cluster. They prevent an object from being deleted until the associated controller has completed its necessary cleanup operations.

5. When should I use Controller Runtime or Operator SDK instead of raw client-go for controller development? While raw client-go provides maximum control and a deep understanding of Kubernetes internals, it involves significant boilerplate for common controller patterns like informer setup, workqueue management, leader election, and metrics. Controller Runtime and Operator SDK (which is built on Controller Runtime) are higher-level frameworks that abstract away much of this complexity. You should use them to accelerate development, reduce boilerplate, and leverage well-established patterns for building robust, production-ready operators. They provide unified client access, simplified reconciliation loops, and tools for scaffolding, testing, and deployment, making them the preferred choice for most new controller projects.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh
APIPark Command Installation Process

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

APIPark System Interface 01

Step 2: Call the OpenAI API.

APIPark System Interface 02
Install APIPark – it’s free
Previous

Master MCP Desktop: Tips for Peak Gaming Performance

 Next

Mastering AI Gateway Kong: Setup & Optimization