How to Build a Kubernetes Controller to Watch CRD Changes

The modern cloud-native landscape is a vibrant ecosystem where applications are increasingly designed to be resilient, scalable, and self-healing. At the heart of this revolution lies Kubernetes, an open-source container orchestration system that has fundamentally reshaped how enterprises deploy, manage, and scale their workloads. Often described as the "operating system for the cloud," Kubernetes provides a declarative api for defining application states and an intelligent control plane to ensure those states are maintained. However, the true power of Kubernetes is not just in its out-of-the-box capabilities, but in its extensible architecture, which allows users to tailor its behavior to specific domain needs. This extensibility is primarily realized through Custom Resources (CRs) and Custom Resource Definitions (CRDs), coupled with the development of custom controllers.

Imagine a scenario where your application requires a new type of resource that isn't natively supported by Kubernetes – perhaps a "Website" resource that encapsulates all the necessary components for deploying a web application, including its deployment, service, ingress, and even SSL certificates. Or perhaps a "DatabaseInstance" that provisions and manages a database in an external cloud provider. Kubernetes doesn't natively understand these concepts, but it provides the mechanism to teach it. Custom Resource Definitions allow you to define these new api objects, making them first-class citizens in the Kubernetes api machinery, just like Pods or Deployments.

Once these custom resources are defined, there needs to be a mechanism to actually do something when they are created, updated, or deleted. This is where Kubernetes controllers come into play. A controller is essentially a control loop that continuously watches a specific set of resources (in our case, custom resources), compares their actual state with their desired state (as defined in the CR), and takes corrective actions to bring the actual state in line with the desired state. This reconciliation process is the fundamental pattern that drives Kubernetes' automation capabilities.

Building a Kubernetes controller to watch CRD changes is not merely an academic exercise; it is a critical skill for anyone looking to build powerful, domain-specific automation on top of Kubernetes. It enables the creation of "operators," which are application-specific controllers that extend the Kubernetes api to create, configure, and manage instances of complex applications. From provisioning databases to managing serverless functions or integrating with complex external api services, controllers unlock a new dimension of automation within your cluster. This comprehensive guide will walk you through the intricate process of designing, developing, and deploying a Kubernetes controller. We will start by demystifying the core concepts of CRDs and controllers, then delve into setting up your development environment, crafting your custom resource definition, and meticulously implementing the controller's logic using the client-go library and controller-runtime framework. Each step will be detailed, providing a robust understanding that goes beyond simple examples, enabling you to tackle complex scenarios and build truly intelligent automation for your cloud-native applications.

Understanding Kubernetes Extension Mechanisms

The core philosophy of Kubernetes is declarative state management. You declare what you want, and Kubernetes works tirelessly to make it so. While its built-in resources cover a vast array of common infrastructure needs, real-world applications often demand more specialized management logic or a higher-level abstraction. Kubernetes was designed with this in mind, offering powerful extension points to augment its capabilities without modifying its core code.

Custom Resources and CRDs: Extending the Kubernetes API

At the foundation of Kubernetes extensibility lies the concept of Custom Resources (CRs) and their definitions, Custom Resource Definitions (CRDs). A CRD is a powerful mechanism that allows you to define new, unique api objects within your Kubernetes cluster. Think of it as teaching Kubernetes a new vocabulary. Before a CRD is applied, Kubernetes has no inherent understanding of your custom resource type. Once defined, these custom resources become first-class citizens, behaving much like native resources such as Pods, Deployments, or Services.

What is a CRD? A CRD is a schema for a new api resource. It specifies the name of your new resource, its api group, version, scope (namespaced or cluster-wide), and most importantly, its openAPIV3Schema which validates the structure and types of the fields within your custom resource. This schema validation is crucial for ensuring that custom resources created by users adhere to a predefined structure, preventing malformed objects from being accepted by the api server. For instance, if you define a "Website" custom resource, its CRD would specify that a domain field must be a string and a contentSource field must also be a string, and perhaps that a status field for ready must be a boolean.

Why Not Just Use Native Resources? You might wonder why you wouldn't just combine existing Kubernetes resources (like Deployments, Services, and Ingresses) to achieve your desired application state. While this approach is perfectly valid and common, it often leads to several challenges: 1. Complexity: Managing dozens or hundreds of individual Kubernetes resources for each application instance can become unwieldy, especially when dealing with complex multi-service applications. 2. Lack of Abstraction: Native resources operate at a lower infrastructure level. They don't inherently understand the business logic or higher-level application concepts specific to your domain. A "Website" resource, for example, conveys a single, cohesive concept that might be composed of multiple native resources. 3. Inconsistent Management: Without a higher-level abstraction, ensuring consistent configuration, lifecycle management, and policy application across many instances of a similar application type becomes difficult and error-prone. 4. No Single Source of Truth: Changes might need to be applied across multiple YAML files, increasing the risk of configuration drift. A custom resource serves as a single, declarative source of truth for your application's desired state.

By defining a CRD, you encapsulate this complexity into a single, domain-specific api object. This not only simplifies the user experience for developers who interact with your system but also provides a clear, machine-readable definition for automation tools, which brings us to controllers.

Controllers: Bringing Desired State to Life

Once you have defined your custom resource, the Kubernetes api server will accept and store instances of it. However, merely storing the resource does not automatically bring it to life. This is where controllers come into play. A Kubernetes controller is a control loop that continuously monitors the state of specific resources within the cluster and takes actions to reconcile the current state with the desired state. In essence, it's a dedicated program running inside your cluster whose sole purpose is to observe and act.

The Reconciliation Loop: The core concept behind any Kubernetes controller is the "reconciliation loop." This loop runs continuously, typically performing the following steps: 1. Observe: The controller watches for changes to the custom resources it manages (e.g., a "Website" CR is created, updated, or deleted). It also watches for changes to any dependent native resources it creates (e.g., a Deployment for the website). 2. Diff: When a change is detected, or on a periodic re-sync, the controller fetches the desired state (from the CR) and compares it with the actual state of the corresponding Kubernetes resources (e.g., checking if the Nginx Deployment for the website exists and has the correct image). 3. Act: If a discrepancy is found, the controller takes corrective action. This might involve creating a missing Deployment, updating an existing Service, deleting an old Ingress, or provisioning an external database via an external api. 4. Update Status: After taking action, the controller updates the status field of the custom resource to reflect the current actual state (e.g., setting status.ready to true and status.url to the website's accessible URL). This feedback loop is crucial for users to understand the operational state of their custom resource.

Key Components of a Controller: Although the full implementation details vary, most controllers leverage several core client-go components to achieve their goals: * Informers: Efficiently watch for resource changes and maintain an in-memory cache of resources. * Listers: Provide read-only access to the informer's cache, preventing direct api server calls for reads. * Workqueues: Decouple event handling from the reconciliation logic, ensuring events are processed reliably and with rate limiting. * Reconcile Function: The main logic that fetches desired and actual states and performs the reconciliation.

Controllers vs. Operators: It's important to clarify the relationship between controllers and operators. A controller is the fundamental building block – a control loop for a specific resource. An "operator" is a specific type of controller that manages the lifecycle of a complex application on Kubernetes. Operators often encapsulate significant domain-specific knowledge, automating tasks such as application deployment, scaling, backup, restore, and upgrades. All operators are controllers, but not all controllers are operators (e.g., a simple controller might only manage a single custom resource and its immediate dependent deployments). This distinction highlights the flexibility and power controllers bring to the Kubernetes ecosystem, enabling sophisticated automation that was once only possible with complex external scripts or manual interventions.

By combining CRDs to define new api objects and controllers to manage their lifecycle, Kubernetes transforms from a mere container orchestrator into a powerful application management platform, capable of understanding and automating your most complex infrastructure and application requirements.

Setting Up Your Development Environment

Before diving into the intricacies of controller logic, a properly configured development environment is paramount. Developing Kubernetes controllers, especially with Go, requires specific tools and a structured approach. This section will guide you through setting up the necessary prerequisites and introducing Kubebuilder, a powerful framework that significantly streamlines the controller development process.

Prerequisites: The Foundation

To begin developing Kubernetes controllers, you'll need the following installed and configured on your local machine:

1. Go Language (v1.20+): Kubernetes controllers are predominantly written in Go. Ensure you have a recent version installed. You can download it from the official Go website. Verify the installation by running go version.
2. Docker (or a compatible container runtime): You'll need Docker to build container images for your controller and to run a local Kubernetes cluster using tools like Kind.
3. kubectl: The Kubernetes command-line tool. This is essential for interacting with your Kubernetes cluster, applying CRDs, creating custom resources, and debugging.
4. Local Kubernetes Cluster (Kind or Minikube): While you can theoretically develop against a remote cluster, having a local cluster is invaluable for rapid iteration and testing.
   • Kind (Kubernetes in Docker): A popular choice for local development, Kind allows you to run a multi-node Kubernetes cluster inside Docker containers. It's lightweight and excellent for testing controller deployments.
   • Minikube: Another excellent option, Minikube runs a single-node Kubernetes cluster inside a VM or directly on your machine.
   • Choose one and follow its installation instructions. Ensure kubectl config use-context kind-kind (for Kind) or minikube start (for Minikube) allows you to interact with your cluster.
5. Git: Essential for version control and cloning repositories.

Go Modules for Dependency Management

Go modules are the standard way to manage dependencies in Go projects. When you initialize a new Go project, you'll create a go.mod file that lists your project's dependencies and their versions. This ensures reproducible builds and simplifies dependency management. Kubebuilder projects automatically leverage Go modules.

Kubebuilder: Your Controller Development Companion

While it's possible to build a Kubernetes controller from scratch using only the client-go library, it involves a substantial amount of boilerplate code for setting up informers, listers, workqueues, and the overall manager structure. This is where Kubebuilder comes in.

What is Kubebuilder? Kubebuilder is an open-source framework built on top of controller-runtime (a library that simplifies controller development using client-go). It provides command-line tools to: * Scaffold projects: Generate the basic directory structure, Go module files, and boilerplate code for your api and controller. * Generate CRDs: Automatically create the YAML definitions for your custom resources based on your Go structs. * Generate api types: Create the Go structs for your custom resources, including Spec and Status fields, and DeepCopy methods. * Simplify controller setup: Handle the initialization of controller-runtime managers, informers, and workqueues. * Provide testing utilities: Aid in writing unit and integration tests for your controller. * Generate Dockerfiles and deployment manifests: Provide templates for containerizing and deploying your controller.

By using Kubebuilder, you can focus on writing the core reconciliation logic that defines your controller's behavior, rather than spending time on repetitive setup tasks.

Installation Steps for Kubebuilder:

1. Install go-get dependencies (for older Go versions or if needed for specific tools): bash go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest go install sigs.k8s.io/kubebuilder/cmd/kubebuilder@latest Note: For modern Go versions (1.17+), go install automatically handles modules. The @latest ensures you get the most recent stable version.
2. Verify installation: bash kubebuilder version This should output the version of Kubebuilder installed.

Creating a New Project with Kubebuilder

Once Kubebuilder is installed, you can create your first controller project. Let's create a project called website-controller that will manage "Website" custom resources.

1. Initialize the project: bash mkdir website-controller cd website-controller kubebuilder init --domain example.com --repo example.com/website-controllerThis command generates a basic project structure including: * main.go: The entry point for your controller. * Dockerfile: For building your controller's image. * PROJECT: A Kubebuilder marker file. * go.mod, go.sum: Go module files. * config/: Directory for deployment manifests (CRDs, RBAC, manager).
   • --domain example.com: This sets the api group domain for your custom resources (e.g., v1.website.example.com). Choose a domain that you control or is unique within your organization.
   • --repo example.com/website-controller: This sets the Go module path for your project.

By following these steps, you've established a robust development environment, complete with the necessary tools and a foundational project structure provided by Kubebuilder. You are now ready to design your custom resource and define its api schema, which is the next crucial step in building your Kubernetes controller.

Designing Your Custom Resource Definition (CRD)

The design of your Custom Resource Definition is foundational to your controller's functionality. It dictates what information your controller will receive and what state it's expected to manage. A well-designed CRD is intuitive, complete, and provides clear schema validation to prevent invalid configurations. For our example, we will design a Website custom resource.

Defining the CRD Spec: What Properties Should Your Custom Resource Have?

When designing a custom resource, you need to think about the declarative inputs a user would provide to define an instance of that resource. For our Website CRD, we envision a resource that allows users to specify:

• domain (string): The domain name under which the website will be accessible (e.g., my-app.example.com).
• contentSource (string): A reference to where the website's content comes from. This could be a Docker image name (e.g., nginx:latest, my-custom-website-image:v1), or perhaps a Git repository URL (e.g., https://github.com/myorg/my-website.git). For simplicity in this guide, we'll assume it's a Docker image that serves static content.

Beyond these input fields, every well-designed custom resource should include a status field. The status field is where the controller reports the actual state of the resource back to the user. It should reflect what the controller has accomplished and any relevant operational details. For our Website CR, we might include:

• ready (boolean): Indicates whether the website is successfully deployed and serving traffic.
• url (string): The actual URL where the website is accessible, which might be derived by the controller (e.g., from an Ingress status).
• observedGeneration (integer): A common field to track which generation of the CR the controller last reconciled. This helps avoid unnecessary reconciliation loops if the spec hasn't truly changed.

Schema Validation: Ensuring Data Integrity

The openAPIV3Schema field within a CRD is critical for api server-side validation. It enforces that any custom resource created or updated in the cluster conforms to the specified structure and data types. This prevents users from providing malformed input, making your controller more robust. You can define types, required fields, minimum/maximum values, string patterns (regex), and more.

API Versions and Versioning Strategy

Just like native Kubernetes resources, custom resources can have multiple api versions (e.g., v1alpha1, v1beta1, v1). This allows for backward-compatible evolution of your api. You specify which versions are served (available via the api) and which one is storage (the version in which the api server stores the resource). Typically, you start with v1alpha1 for early development, move to v1beta1 for more stability, and finally v1 for stable, production-ready apis. For this guide, we'll start with v1.

API Group and Kind

Every custom resource belongs to an api group and has a Kind. The api group helps organize custom resources and prevents naming collisions. For our project, we used --domain example.com, so our api group will be example.com. The Kind is the name of your resource (e.g., Website). These combine to form the apiVersion used when referencing the resource (e.g., apiVersion: example.com/v1, kind: Website).

Example CRD (website.yaml)

Now, let's use Kubebuilder to generate the initial CRD boilerplate. This is done by creating the api types in Go.

1. Generate API and Controller: bash kubebuilder create api --group example --version v1 --kind Website This command does several things:
   • It creates api/v1/website_types.go, defining the Go structs for your Website CR's Spec and Status.
   • It creates controllers/website_controller.go, which is where your main reconciliation logic will reside.
   • It updates main.go to register your new api and controller.
   • It generates initial CRD YAML in config/crd/bases/example.com_websites.yaml after running make manifests.
2. Generate and Apply the CRD: After modifying website_types.go, run the following commands from your project root: bash make manifests kubectl apply -f config/crd/bases/example.com_websites.yamlYou can verify the CRD is installed by running: bash kubectl get crd websites.example.com This output should show your websites.example.com CRD.
   • make manifests: This command executes controller-gen to generate (or update) the CRD YAML based on your Go structs and kubebuilder markers.
   • kubectl apply: This registers your new Website CRD with your Kubernetes cluster.

Modify api/v1/website_types.go: Open this file and define the Spec and Status fields as discussed.```go package v1import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" )// WebsiteSpec defines the desired state of Website type WebsiteSpec struct { // +kubebuilder:validation:MinLength=1 // +kubebuilder:validation:Pattern=^([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])(\.([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]{0,61}[a-zA-Z0-9]))*$ // Domain is the hostname for the website (e.g., my-app.example.com) Domain string json:"domain"

// +kubebuilder:validation:MinLength=1
// ContentSource is the Docker image to serve the website content (e.g., nginx:latest, my-custom-website-image:v1)
ContentSource string `json:"contentSource"`

}// WebsiteStatus defines the observed state of Website type WebsiteStatus struct { // +operator-sdk:builder:validation:Optional // Ready indicates if the website is successfully deployed and serving traffic Ready bool json:"ready"

// +operator-sdk:builder:validation:Optional
// URL is the accessible URL of the website
URL string `json:"url,omitempty"`

// +operator-sdk:builder:validation:Optional
// ObservedGeneration is the latest generation observed by the controller.
ObservedGeneration int64 `json:"observedGeneration,omitempty"`

}// +kubebuilder:object:root=true // +kubebuilder:subresource:status // +kubebuilder:resource:path=websites,scope=Namespaced,singular=website // +kubebuilder:printcolumn:name="Domain",type="string",JSONPath=".spec.domain",description="The domain of the website" // +kubebuilder:printcolumn:name="Ready",type="boolean",JSONPath=".status.ready",description="Whether the website is ready" // +kubebuilder:printcolumn:name="URL",type="string",JSONPath=".status.url",description="The URL of the website" // +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"/techblog/en// Website is the Schema for the websites API type Website struct { metav1.TypeMeta json:",inline" metav1.ObjectMeta json:"metadata,omitempty"

Spec   WebsiteSpec   `json:"spec,omitempty"`
Status WebsiteStatus `json:"status,omitempty"`

}// +kubebuilder:object:root=true// WebsiteList contains a list of Website type WebsiteList struct { metav1.TypeMeta json:",inline" metav1.ListMeta json:"metadata,omitempty" Items []Website json:"items" }func init() { SchemeBuilder.Register(&Website{}, &WebsiteList{}) } `` * **Go Struct Tags:** Notice thejson:"..."tags. These are crucial for marshaling/unmarshaling the Go struct to/from JSON (which Kubernetesapiuses). * **// +kubebuilder:markers:** These special comments are used bycontroller-gen(a tool invoked bymake manifests) to automatically generate the CRD YAML,rbac.yaml, and other Kubernetes manifests. *+kubebuilder:validation:MinLength,+kubebuilder:validation:Pattern: These generateopenAPIV3Schemavalidation rules. *+kubebuilder:object:root=true: Marks this as a root Kubernetes object. *+kubebuilder:subresource:status: Indicates that thestatussubresource should be enabled, allowing controllers to update status without needing to update thespec. *+kubebuilder:resource:path=websites,scope=Namespaced,singular=website: Defines theapipath, scope, and singular name. *+kubebuilder:printcolumn: Configureskubectl get website` to display these columns.

With your CRD successfully defined and installed, Kubernetes now understands the Website resource. The next step is to build the actual controller that watches for changes to these Website resources and orchestrates the necessary underlying Kubernetes objects to bring them to life.

The Core Components of a Kubernetes Controller

Building a robust Kubernetes controller requires understanding and effectively utilizing several fundamental components, primarily provided by the client-go library and orchestrated by the controller-runtime framework. These components work in concert to ensure efficient and reliable observation, caching, queuing, and reconciliation of resources within your cluster.

Client-go Library: The Fundamental API Client

client-go is the official Go client library for interacting with the Kubernetes api server. It provides the necessary building blocks for api requests, object serialization, authentication, and more. While you can use client-go directly, controller-runtime (which Kubebuilder utilizes) provides a higher-level abstraction that makes controller development significantly easier by handling much of the client-go boilerplate. Nevertheless, understanding its underlying principles is beneficial.

Informers: Efficiently Watching for Resource Changes

Controllers need to know when a resource they care about changes. Continuously polling the api server is inefficient and places undue load on the control plane. This is where Informers come in.

• Role: Informers are the most efficient way for controllers to receive notifications about resource changes (additions, updates, deletions) and to maintain a local, in-memory cache of these resources.
• Mechanics: List-and-Watch: An informer works by performing an initial "List" operation to get all existing resources of a certain type. It then establishes a "Watch" connection with the api server. Any subsequent changes (Add, Update, Delete) are streamed to the informer.
• Local Cache: The informer maintains an internal store (often an Indexer from client-go) that is kept synchronized with the api server's state. This cache is crucial because it allows the controller to read resource states locally, significantly reducing the number of direct api server calls, thereby improving performance and reducing api server load.
• Event Handlers: Informers provide event handlers (OnAdd, OnUpdate, OnDelete) that allow your controller to react to specific events. When an event occurs, these handlers typically enqueue the relevant resource's key (e.g., namespace/name) into a workqueue for processing.
• SharedInformers: In a complex controller or when running multiple controllers within the same process, SharedInformers are used. They ensure that only one list-and-watch call is made to the api server for a given resource type, and all interested controllers share the same local cache. controller-runtime automatically uses SharedInformers through its Manager.

Listers: Accessing Cached Objects

While informers populate the local cache, Listers are the components that provide convenient, thread-safe read-only access to that cache.

• Role: Listers allow your controller to retrieve objects from the informer's local cache without making direct api calls to the Kubernetes api server. This is vital for performance and consistency within your reconciliation loop.
• Avoiding Direct API Server Calls for Reads: By relying on the informer's cache via listers, your controller avoids potential network latency, api server rate limiting, and the overhead of deserializing api responses for every read operation. This makes the reconciliation loop much faster and more resilient.
• Consistency: The cache provided by informers and accessed by listers is eventually consistent. While it might be slightly stale compared to the absolute latest state in the api server, for most controller operations, this level of consistency is sufficient and provides significant performance benefits.

Workqueues: Decoupling Event Handling from Reconciliation Logic

The direct processing of OnAdd, OnUpdate, OnDelete events can lead to various issues, such as processing events too quickly, duplicate processing, or issues with rate limiting. This is where Workqueues (specifically workqueue.RateLimitingInterface) are indispensable.

• Role: A workqueue acts as a buffer and a mechanism to decouple the event handling logic (which simply adds an item to the queue) from the actual reconciliation logic (which processes items from the queue). It ensures that items are processed reliably, potentially with rate limiting and retry mechanisms.
• How They Prevent Processing Issues:
  • Debouncing/Deduplication: If multiple updates occur to the same resource in quick succession, the workqueue can often consolidate these into a single item or ensure that only one reconciliation for that resource is in flight at a time.
  • Rate Limiting: If a reconciliation fails due to a transient error (e.g., a temporary network issue or an external api timeout), the workqueue can re-add the item to the queue with an exponential backoff, preventing the controller from flooding the api server or external services with immediate retries.
  • Concurrency Control: The number of worker goroutines processing items from the queue can be controlled, preventing the controller from overwhelming itself or the cluster.
• Adding and Processing Items: When an event occurs (e.g., OnAdd for a Website CR), the event handler adds the namespace/name key of the resource to the workqueue. Worker goroutines then continuously pull items from the workqueue, execute the reconciliation logic, and mark the item as done. If a reconciliation fails, the item might be re-queued with a delay.

Reconciliation Loop (Reconcile function): The Heart of the Controller

The Reconcile function is the core business logic of your controller. It embodies the control loop pattern that drives Kubernetes automation.

• What it Does: When an item (typically a resource key) is pulled from the workqueue, the Reconcile function is invoked. Its primary responsibilities are:
  1. Fetch Desired State: Retrieve the current state of the custom resource (e.g., Website CR) from the informer's cache using a lister. This is the "desired state."
  2. Fetch Actual State: Retrieve the current state of any dependent Kubernetes resources (e.g., Deployment, Service, Ingress) that the controller is responsible for managing, also typically from the cache. This is the "actual state."
  3. Compare and Act: Compare the desired state with the actual state. If there are discrepancies, take the necessary actions:
     • Create missing resources (e.g., if a Deployment for the website doesn't exist).
     • Update existing resources (e.g., if the ContentSource in the Website CR changes, update the Deployment's image).
     • Delete extraneous resources (e.g., if a previous Deployment is no longer needed).
     • Interact with external services via their apis (e.g., provision a DNS record, configure a cloud load balancer).
  4. Idempotency: The Reconcile function must be idempotent. This means that applying the same desired state multiple times should always result in the same actual state without unintended side effects. For example, if a Deployment already exists with the correct image, the controller should not attempt to re-create it. This often involves checking for existence before creation and performing updates rather than wholesale recreation.
  5. Error Handling and Re-queuing: If an error occurs during reconciliation (e.g., api server timeout, invalid configuration, external api call failure), the Reconcile function should return an error. controller-runtime will then typically re-queue the item with a delay, allowing the controller to retry the operation later. If no error occurs, the item is considered successfully reconciled and removed from the workqueue.
  6. Updating Resource Status: After all actions are taken, the controller should update the status field of the custom resource to reflect the actual state. This provides crucial feedback to users. For instance, if the Website is successfully deployed, status.ready might be set to true and status.url to its public URL.

Manager (from controller-runtime): The Orchestrator

The controller-runtime Manager is a powerful component that orchestrates all the individual parts of your controller. It simplifies controller setup significantly by:

• Handling Shared Informers: It manages a single set of shared informers for all registered controllers, optimizing api server interactions.
• Setting up Caches: It sets up and starts the caches for all watched resources.
• Running Controllers: It starts and manages the lifecycle of multiple Reconcile loops.
• Leader Election: For high availability, it can facilitate leader election, ensuring that only one instance of your controller is actively reconciling resources at any given time in a multi-replica deployment.
• Health Checks and Metrics: It provides endpoints for health checks and exposes Prometheus metrics, which are essential for monitoring your controller's operation.

By leveraging the Manager, developers can focus almost entirely on the custom api types and the Reconcile function, with controller-runtime handling the complex interactions with client-go and the Kubernetes api machinery. This modular design greatly simplifies the development and maintenance of Kubernetes controllers.

Table: Core Components of a Kubernetes Controller

Component	Primary Role	Key Functionality	Interaction
Informer	Efficiently watches for resource changes and maintains a local cache.	Performs initial List, maintains Watch connection, stores resources in cache, triggers event handlers.	Communicates directly with Kubernetes api server (List & Watch).
Lister	Provides read-only, thread-safe access to the informer's local cache.	Retrieves specific objects (by name/namespace) or lists all objects of a type from cache. Avoids direct api server calls.	Reads from the cache maintained by Informer.
Workqueue	Decouples event handling from reconciliation, ensures reliable and rate-limited processing of events.	Stores resource keys to be reconciled, handles retries with backoff, debounces multiple events for the same object, controls concurrency.	Receives events from Informer's handlers, feeds items to the Reconcile function.
Reconcile Loop	The core business logic; brings the actual state of resources in line with the desired state specified in the Custom Resource.	Fetches desired state (CR), fetches actual state (dependent K8s resources), compares, creates/updates/deletes resources, handles errors, updates CR status.	Interacts with Lister (to get cached objects), Client (to create/update/delete actual resources), and Workqueue (to re-queue items).
Manager	Orchestrates and manages the lifecycle of multiple controllers.	Initializes shared informers and caches, starts controllers, handles leader election, provides health checks and metrics.	Sets up and runs Informers, Listers, Workqueues, and Reconcile functions for all managed controllers.

Understanding how these components interoperate is fundamental to writing effective and efficient Kubernetes controllers. With this knowledge, we can now proceed to implement the actual controller logic for our Website custom resource.

Implementing Your Controller Logic

Having defined our Website CRD and understood the theoretical underpinnings of Kubernetes controllers, it's time to translate that knowledge into executable code. Kubebuilder has already provided a significant head start by scaffolding much of the boilerplate. Our primary task now is to fill in the core reconciliation logic within the Reconcile function in controllers/website_controller.go.

Scaffolding with Kubebuilder: A Quick Review

As we discussed earlier, the command kubebuilder create api --group example --version v1 --kind Website generated two key files:

• api/v1/website_types.go: Contains the Go struct definitions for Website (spec, status) and WebsiteList. We already modified this to include our Domain and ContentSource fields, as well as Ready, URL, and ObservedGeneration for the status.
• controllers/website_controller.go: This file contains the Reconcile function and the SetupWithManager method, which configures how the controller watches resources.

Understanding the Generated Code

Let's briefly look at the structure of controllers/website_controller.go:

package controllers

import (
    "context"

    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    examplev1 "example.com/website-controller/api/v1" // Our CRD API
)

// WebsiteReconciler reconciles a Website object
type WebsiteReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=example.com,resources=websites,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=example.com,resources=websites/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="networking.k8s.io",resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify Reconcile to perform your controller's desired logic here.
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.16.3/pkg/reconcile
func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // your logic here

    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *WebsiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.Website{}).
        Complete(r)
}

• WebsiteReconciler struct: Holds references to the client.Client (for api interactions) and runtime.Scheme (for type information).
• +kubebuilder:rbac: markers: These are crucial. They automatically generate the ClusterRole and RoleBinding YAML for your controller, granting it the necessary permissions to get, list, watch, create, update, patch, and delete the specified resources (our Website CR, its status, Deployments, Services, and Ingresses). Always ensure your controller has the least privilege necessary.
• Reconcile function signature: (ctx context.Context, req ctrl.Request) (ctrl.Result, error). ctx is for context propagation, req contains the NamespacedName of the resource that triggered the reconciliation. It returns ctrl.Result (e.g., to re-queue with a delay) and an error.
• SetupWithManager: This is where you configure the controller to watch specific resources. For(&examplev1.Website{}) tells the controller to watch for changes to Website custom resources.

Modifying the Reconcile Function: The Core Logic

Now, let's implement the Reconcile function to manage the lifecycle of our Website CR. The controller will create an Nginx Deployment, a Service, and an Ingress for each Website CR.

package controllers

import (
    "context"
    "fmt"
    "reflect"
    "time"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    networkingv1 "k8s.io/api/networking/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/intstr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    examplev1 "example.com/website-controller/api/v1"
)

// WebsiteReconciler reconciles a Website object
type WebsiteReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=example.com,resources=websites,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=example.com,resources=websites/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="networking.k8s.io",resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.16.3/pkg/reconcile
func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Fetch the Website instance
    website := &examplev1.Website{}
    err := r.Get(ctx, req.NamespacedName, website)
    if err != nil {
        if errors.IsNotFound(err) {
            // Request object not found, could have been deleted after reconcile request.
            // Return empty result and do not re-queue.
            log.Info("Website resource not found. Ignoring since object must be deleted")
            return ctrl.Result{}, nil
        }
        // Error reading the object - re-queue the request.
        log.Error(err, "Failed to get Website")
        return ctrl.Result{}, err
    }

    // 2. Handle finalization for deletion
    // Define the finalizer name
    websiteFinalizer := "website.example.com/finalizer"
    if website.ObjectMeta.DeletionTimestamp.IsZero() {
        // The object is not being deleted, so if it does not have our finalizer,
        // then lets add it. This is equivalent to registering our finalizer.
        if !controllerutil.ContainsFinalizer(website, websiteFinalizer) {
            log.Info("Adding Finalizer for Website")
            controllerutil.AddFinalizer(website, websiteFinalizer)
            if err := r.Update(ctx, website); err != nil {
                return ctrl.Result{}, err
            }
        }
    } else {
        // The object is being deleted
        if controllerutil.ContainsFinalizer(website, websiteFinalizer) {
            // Our finalizer is present, so lets handle any external dependency
            log.Info("Performing Finalizer Operations for Website before deletion")
            // TODO(user): Add here the cleanup logic to remove any external resources that were created by the controller
            // In our case, for this simple example, there are no external resources,
            // but if you provisioned a DNS entry or cloud load balancer,
            // this is where you'd clean them up.
            log.Info("Finished Finalizer Operations for Website")

            // Remove our finalizer from the list and update it.
            controllerutil.RemoveFinalizer(website, websiteFinalizer)
            if err := r.Update(ctx, website); err != nil {
                return ctrl.Result{}, err
            }
        }

        // Stop reconciliation as the item is being deleted
        return ctrl.Result{}, nil
    }

    // 3. Reconcile Deployment
    deployment := &appsv1.Deployment{}
    expectedDeployment := r.desiredDeployment(website)

    // Set Website instance as the owner and controller
    // This ensures that when the Website is deleted, the Deployment is also garbage collected
    if err := ctrl.SetControllerReference(website, expectedDeployment, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for Deployment", "Deployment.Name", expectedDeployment.Name)
        return ctrl.Result{}, err
    }

    foundDeployment := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: expectedDeployment.Name, Namespace: expectedDeployment.Namespace}, foundDeployment)
    if err != nil && errors.IsNotFound(err) {
        log.Info("Creating a new Deployment", "Deployment.Namespace", expectedDeployment.Namespace, "Deployment.Name", expectedDeployment.Name)
        err = r.Create(ctx, expectedDeployment)
        if err != nil {
            log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", expectedDeployment.Namespace, "Deployment.Name", expectedDeployment.Name)
            return ctrl.Result{}, err
        }
        // Deployment created successfully - don't re-queue, status update will trigger next reconcile
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        return ctrl.Result{}, err
    } else {
        // Check if the Deployment needs to be updated
        if !reflect.DeepEqual(expectedDeployment.Spec, foundDeployment.Spec) {
            log.Info("Updating Deployment", "Deployment.Namespace", foundDeployment.Namespace, "Deployment.Name", foundDeployment.Name)
            foundDeployment.Spec = expectedDeployment.Spec // Update the spec
            if err := r.Update(ctx, foundDeployment); err != nil {
                log.Error(err, "Failed to update Deployment", "Deployment.Namespace", foundDeployment.Namespace, "Deployment.Name", foundDeployment.Name)
                return ctrl.Result{}, err
            }
        }
    }

    // 4. Reconcile Service
    service := &corev1.Service{}
    expectedService := r.desiredService(website)

    if err := ctrl.SetControllerReference(website, expectedService, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for Service", "Service.Name", expectedService.Name)
        return ctrl.Result{}, err
    }

    foundService := &corev1.Service{}
    err = r.Get(ctx, types.NamespacedName{Name: expectedService.Name, Namespace: expectedService.Namespace}, foundService)
    if err != nil && errors.IsNotFound(err) {
        log.Info("Creating a new Service", "Service.Namespace", expectedService.Namespace, "Service.Name", expectedService.Name)
        err = r.Create(ctx, expectedService)
        if err != nil {
            log.Error(err, "Failed to create new Service", "Service.Namespace", expectedService.Namespace, "Service.Name", expectedService.Name)
            return ctrl.Result{}, err
        }
    } else if err != nil {
        log.Error(err, "Failed to get Service")
        return ctrl.Result{}, err
    } else {
        // Check if the Service needs to be updated
        // (e.g., if target port or labels change)
        if !reflect.DeepEqual(expectedService.Spec.Ports, foundService.Spec.Ports) ||
            !reflect.DeepEqual(expectedService.Spec.Selector, foundService.Spec.Selector) {
            log.Info("Updating Service", "Service.Namespace", foundService.Namespace, "Service.Name", foundService.Name)
            foundService.Spec.Ports = expectedService.Spec.Ports
            foundService.Spec.Selector = expectedService.Spec.Selector
            if err := r.Update(ctx, foundService); err != nil {
                log.Error(err, "Failed to update Service", "Service.Namespace", foundService.Namespace, "Service.Name", foundService.Name)
                return ctrl.Result{}, err
            }
        }
    }

    // 5. Reconcile Ingress
    ingress := &networkingv1.Ingress{}
    expectedIngress := r.desiredIngress(website)

    if err := ctrl.SetControllerReference(website, expectedIngress, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for Ingress", "Ingress.Name", expectedIngress.Name)
        return ctrl.Result{}, err
    }

    foundIngress := &networkingv1.Ingress{}
    err = r.Get(ctx, types.NamespacedName{Name: expectedIngress.Name, Namespace: expectedIngress.Namespace}, foundIngress)
    if err != nil && errors.IsNotFound(err) {
        log.Info("Creating a new Ingress", "Ingress.Namespace", expectedIngress.Namespace, "Ingress.Name", expectedIngress.Name)
        err = r.Create(ctx, expectedIngress)
        if err != nil {
            log.Error(err, "Failed to create new Ingress", "Ingress.Namespace", expectedIngress.Namespace, "Ingress.Name", expectedIngress.Name)
            return ctrl.Result{}, err
        }
    } else if err != nil {
        log.Error(err, "Failed to get Ingress")
        return ctrl.Result{}, err
    } else {
        // Check if Ingress needs update (e.g., rules/host/backend changes)
        if !reflect.DeepEqual(expectedIngress.Spec, foundIngress.Spec) {
            log.Info("Updating Ingress", "Ingress.Namespace", foundIngress.Namespace, "Ingress.Name", foundIngress.Name)
            foundIngress.Spec = expectedIngress.Spec
            if err := r.Update(ctx, foundIngress); err != nil {
                log.Error(err, "Failed to update Ingress", "Ingress.Namespace", foundIngress.Namespace, "Ingress.Name", foundIngress.Name)
                return ctrl.Result{}, err
            }
        }
    }

    // 6. Update Website status
    updatedWebsite := website.DeepCopy() // Work with a copy to prevent race conditions during updates
    statusUpdated := false

    // Update Ready status based on Deployment
    if foundDeployment.Status.AvailableReplicas > 0 {
        if !updatedWebsite.Status.Ready {
            updatedWebsite.Status.Ready = true
            log.Info("Website is ready", "Website.Name", updatedWebsite.Name)
            statusUpdated = true
        }
    } else {
        if updatedWebsite.Status.Ready {
            updatedWebsite.Status.Ready = false
            log.Info("Website is not ready", "Website.Name", updatedWebsite.Name)
            statusUpdated = true
        }
    }

    // Update URL status based on Ingress
    websiteURL := ""
    if len(foundIngress.Spec.Rules) > 0 && len(foundIngress.Spec.Rules[0].Host) > 0 {
        websiteURL = fmt.Sprintf("http://%s", foundIngress.Spec.Rules[0].Host) // Assuming HTTP for simplicity
    }
    if websiteURL != updatedWebsite.Status.URL {
        updatedWebsite.Status.URL = websiteURL
        log.Info("Updating Website URL", "Website.Name", updatedWebsite.Name, "URL", websiteURL)
        statusUpdated = true
    }

    // Update ObservedGeneration
    if website.Generation != website.Status.ObservedGeneration {
        updatedWebsite.Status.ObservedGeneration = website.Generation
        statusUpdated = true
    }

    if statusUpdated {
        if err := r.Status().Update(ctx, updatedWebsite); err != nil {
            log.Error(err, "Failed to update Website status")
            return ctrl.Result{}, err
        }
        log.Info("Website status updated", "Website.Name", updatedWebsite.Name)
        return ctrl.Result{Requeue: true}, nil // Re-queue to ensure the status change is processed
    }

    // Re-queue after a delay to periodically check the deployment status and update Website status
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

// Helper functions to construct desired Kubernetes resources
func (r *WebsiteReconciler) desiredDeployment(website *examplev1.Website) *appsv1.Deployment {
    labels := labelsForWebsite(website.Name)
    replicas := int32(1) // Always 1 replica for simplicity
    return &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      website.Name + "-deployment",
            Namespace: website.Namespace,
            Labels:    labels,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: labels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: labels,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "webserver",
                        Image: website.Spec.ContentSource, // Use the image from CRD spec
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 80, // Assuming HTTP on port 80
                        }},
                    }},
                },
            },
        },
    }
}

func (r *WebsiteReconciler) desiredService(website *examplev1.Website) *corev1.Service {
    labels := labelsForWebsite(website.Name)
    return &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      website.Name + "-service",
            Namespace: website.Namespace,
            Labels:    labels,
        },
        Spec: corev1.ServiceSpec{
            Selector: labels,
            Ports: []corev1.ServicePort{{
                Protocol:   corev1.ProtocolTCP,
                Port:       80,
                TargetPort: intstr.FromInt(80),
            }},
            Type: corev1.ServiceTypeClusterIP, // Internal service
        },
    }
}

func (r *WebsiteReconciler) desiredIngress(website *examplev1.Website) *networkingv1.Ingress {
    labels := labelsForWebsite(website.Name)
    pathType := networkingv1.PathTypePrefix // Or PathTypeExact
    return &networkingv1.Ingress{
        ObjectMeta: metav1.ObjectMeta{
            Name:      website.Name + "-ingress",
            Namespace: website.Namespace,
            Labels:    labels,
            Annotations: map[string]string{
                // Add common Ingress controller annotations if needed, e.g., for Nginx
                // "nginx.ingress.kubernetes.io/ssl-redirect": "false",
                // "nginx.ingress.kubernetes.io/force-ssl-redirect": "false",
            },
        },
        Spec: networkingv1.IngressSpec{
            Rules: []networkingv1.IngressRule{{
                Host: website.Spec.Domain, // Use the domain from CRD spec
                IngressRuleValue: networkingv1.IngressRuleValue{
                    HTTP: &networkingv1.HTTPIngressRuleValue{
                        Paths: []networkingv1.HTTPIngressPath{{
                            Path:     "/techblog/en/",
                            PathType: &pathType,
                            Backend: networkingv1.IngressBackend{
                                Service: &networkingv1.IngressServiceBackend{
                                    Name: website.Name + "-service",
                                    Port: networkingv1.ServiceBackendPort{
                                        Number: 80,
                                    },
                                },
                            },
                        }},
                    },
                },
            }},
        },
    }
}

// labelsForWebsite returns the labels for selecting the resources
// belonging to the given Website CR name.
func labelsForWebsite(name string) map[string]string {
    return map[string]string{
        "app":       "website",
        "website_cr": name,
    }
}

// SetupWithManager sets up the controller with the Manager.
func (r *WebsiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.Website{}).                                // Watches Website CRs
        Owns(&appsv1.Deployment{}).                              // Watches Deployments owned by Website
        Owns(&corev1.Service{}).                                 // Watches Services owned by Website
        Owns(&networkingv1.Ingress{}).                           // Watches Ingresses owned by Website
        Complete(r)
}

Detailed Breakdown of the Reconcile Function

1. Fetch the Website instance:
   • r.Get(ctx, req.NamespacedName, website): This is the first action. It attempts to fetch the Website custom resource that triggered this reconciliation from the api server (or more accurately, from the informer's cache managed by controller-runtime).
   • errors.IsNotFound(err): If the resource is not found, it likely means it was deleted. In this case, we simply log and return, as there's nothing left to reconcile.
   • Any other error implies a transient issue, so we log the error and return, allowing controller-runtime to re-queue the request with an exponential backoff.
2. Handle Finalization for Deletion:
   • Finalizers: These are crucial for performing cleanup operations before a resource is fully deleted. If a resource has finalizers, its deletion is blocked until all finalizers are removed. This gives your controller a chance to clean up any external resources (e.g., cloud load balancers, DNS entries, API registrations via external api calls) or dependent Kubernetes resources that wouldn't be garbage collected automatically.
   • We add a finalizer (website.example.com/finalizer) when the Website CR is created or updated and is not marked for deletion.
   • When DeletionTimestamp is set (meaning the user ran kubectl delete website <name>), the controller checks for its finalizer. If present, it performs cleanup logic (in a real-world scenario, this is where you'd call out to an external api to de-provision resources). After cleanup, the controller removes its finalizer, allowing the Kubernetes api server to complete the deletion.
3. Reconcile Deployment, Service, and Ingress:
   • For each dependent resource (Deployment, Service, Ingress), the pattern is similar:
     • r.desiredDeployment(website): Helper functions (desiredDeployment, desiredService, desiredIngress) are used to construct the desired state of the Kubernetes native resource based on the Website CR's Spec.
     • ctrl.SetControllerReference: This is a vital function. It sets the Website CR as the owner of the Deployment, Service, and Ingress. This enables Kubernetes' garbage collection, meaning if the Website CR is deleted, its owned resources will automatically be deleted by the garbage collector.
     • Fetch and Compare: The controller attempts to Get the existing dependent resource (foundDeployment, foundService, foundIngress).
     • Create if Not Found: If errors.IsNotFound(err) for the dependent resource, the controller Creates it.
     • Update if Different: If the resource exists, the controller uses reflect.DeepEqual (or a more efficient comparison for specific fields) to check if the Spec of the found resource matches the expected resource. If they differ, the found resource's Spec is updated, and r.Update(ctx, found...) is called.
     • Error Handling: Any errors during Get, Create, or Update are logged, and the reconciliation is re-queued.
4. Update Website Status:
   • After attempting to reconcile all dependent resources, the controller's responsibility is to update the Website CR's Status field to reflect the current operational state.
   • It checks the Status of the Deployment (e.g., AvailableReplicas > 0) to determine if the Website.Status.Ready flag should be true or false.
   • It extracts the URL from the Ingress object (assuming a simple HTTP setup here, more complex Ingress rules might require more sophisticated URL derivation).
   • It updates Website.Status.ObservedGeneration to match website.Generation. This helps prevent unnecessary reconciliation if only metadata changes.
   • r.Status().Update(ctx, updatedWebsite): It's important to use r.Status().Update specifically for updating the status subresource. This method requires less permissions and avoids conflicts with concurrent updates to the spec by users.
   • If the status is updated, we Requeue: true to trigger another reconciliation immediately, allowing the controller to react to the status change itself.
5. Re-queue for Periodic Check:
   • return ctrl.Result{RequeueAfter: 30 * time.Second}, nil: Even if no changes were detected, it's good practice to periodically re-queue the resource for reconciliation. This acts as a self-healing mechanism, ensuring that if something external modifies a dependent resource (e.g., someone manually deletes the Deployment), the controller will eventually detect the drift and correct it.

Watching Dependent Resources

The SetupWithManager function is where you configure what resources your controller watches.

func (r *WebsiteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&examplev1.Website{}).     // Main resource to watch
        Owns(&appsv1.Deployment{}).    // Also watch Deployments owned by Website
        Owns(&corev1.Service{}).       // Also watch Services owned by Website
        Owns(&networkingv1.Ingress{}). // Also watch Ingresses owned by Website
        Complete(r)
}

• For(&examplev1.Website{}): This tells the manager that Website resources are the primary type this controller manages. Any Add, Update, or Delete event for a Website CR will trigger a reconciliation.
• Owns(&appsv1.Deployment{}), Owns(&corev1.Service{}), Owns(&networkingv1.Ingress{}): These lines are critical. They instruct the controller to also watch for changes to Deployment, Service, and Ingress resources that are owned by a Website CR. If, for example, a Deployment created by our controller is accidentally deleted by a user, this Owns declaration will cause an event to be sent, triggering a reconciliation for the owning Website CR, which will then re-create the missing Deployment. This is how the controller achieves self-healing.

With this logic, our controller is now capable of observing changes to Website custom resources, creating and managing the necessary underlying Kubernetes primitives, and reporting back the operational status. The next phase involves building and deploying this controller to a Kubernetes cluster and understanding how to manage it effectively.

Advanced Topics and Best Practices

Building a basic controller is a great start, but creating production-ready, resilient, and observable controllers requires delving into more advanced concepts and adhering to best practices. These considerations ensure your controller is robust, secure, and maintainable in the long run.

Finalizers: Ensuring Proper Cleanup

We briefly touched upon finalizers in the Reconcile function. Finalizers are strings defined in a resource's metadata.finalizers field. When a resource with finalizers is requested to be deleted, Kubernetes doesn't immediately remove it from the api server. Instead, it sets the metadata.deletionTimestamp and continues to show the resource as "terminating." It's then the responsibility of the controller (or controllers) owning those finalizers to perform any necessary cleanup operations.

Use Cases for Finalizers: * External Resource De-provisioning: If your controller provisioned resources outside the Kubernetes cluster (e.g., cloud load balancers, DNS records in an external api, database instances, APIPark configurations), finalizers ensure these external resources are properly cleaned up before the Kubernetes CR itself is removed. * Dependent Resource Cleanup (beyond owner references): While OwnerReferences handle most cascading deletions within Kubernetes, there might be cases where more complex, ordered cleanup is needed, or if a resource cannot be directly owned (e.g., cluster-scoped resources linked to namespaced CRs). * Preventing Premature Deletion: Finalizers act as a safeguard, preventing users from accidentally or prematurely deleting a resource before all associated dependencies are handled.

Implementation: The pattern involves checking website.ObjectMeta.DeletionTimestamp.IsZero(). If it's zero, the resource is active, and you ensure your finalizer is present. If it's non-zero, the resource is being deleted, and you execute your cleanup logic. After cleanup, you remove your finalizer from the list and update the resource, allowing Kubernetes to complete its deletion.

Webhooks (Validating & Mutating): Intercepting API Requests

Kubernetes webhooks allow you to intercept api requests to the api server before they are persisted. This provides a powerful mechanism to enforce custom policies and automate object modifications.

• Validating Webhooks:
  • Purpose: To validate resource creations, updates, or deletions against custom business logic that goes beyond schema validation defined in the CRD. If the validation fails, the api request is rejected.
  • Example: For our Website CR, a validating webhook could ensure that the spec.domain is globally unique within the cluster, or that the contentSource refers to an approved image registry. It could also prevent deletion of a Website if it's currently marked as critical.
• Mutating Webhooks:
  • Purpose: To modify (mutate) resource objects before they are persisted to the api server. This is commonly used for injecting default values, adding labels/annotations, or even injecting sidecar containers into pods.
  • Example: A mutating webhook could automatically add a https:// prefix to a URL if not specified, or inject specific default annotations onto the Ingress created by the Website controller.

Kubebuilder simplifies webhook creation through code generation and admission.Handler interfaces. Webhooks run as separate HTTP servers (often co-located with the controller) that receive AdmissionReview requests from the api server and return AdmissionReview responses.

Status Subresources: Separating Concerns

By adding +kubebuilder:subresource:status to your CRD's Go struct, you enable the status subresource. This has several benefits: * Reduced Conflicts: It allows a controller to update the status field of a resource without needing to know or update the spec field. This reduces potential conflicts if a user is simultaneously updating the spec. * Fine-Grained RBAC: You can grant specific RBAC permissions (update or patch on the status subresource) to controllers, while user roles might only have permissions to update the spec. This enforces a clear separation of duties. * Optimized api Calls: Updates to the status subresource can be more lightweight for the api server.

Always use r.Status().Update(ctx, yourResource) when updating the status field.

Testing: Ensuring Correctness and Reliability

Comprehensive testing is non-negotiable for controllers. * Unit Tests: Test individual functions and components in isolation (e.g., helper functions that construct Kubernetes objects). * Integration Tests: Test the controller's Reconcile function against a real (or mock) Kubernetes api server. Kubebuilder provides a envtest package that spins up a lightweight api server for this purpose, allowing you to test Create, Get, Update, Delete operations and their effects. * End-to-End (E2E) Tests: Deploy your controller and CRDs to a full Kubernetes cluster (e.g., Kind or Minikube) and verify its behavior from a user's perspective (e.g., create a Website CR, then kubectl get website to check its status, and kubectl get deployment to verify the underlying resources).

Logging and Metrics: Essential for Observability

Controllers are background processes, making good observability critical for debugging and monitoring. * Logging: Use structured logging (e.g., logr or zap provided by controller-runtime). Log key events, errors, and changes to resource states. Include relevant object identifiers (Namespace, Name, Kind) in log messages. * Metrics: controller-runtime automatically exposes Prometheus metrics (e.g., controller_runtime_reconcile_total, controller_runtime_reconcile_errors_total). These metrics provide insights into your controller's performance, reconciliation rates, and error rates. You can also add custom metrics for domain-specific events or states.

Performance Considerations: Efficiency is Key

• Efficient api Calls: Minimize direct api server calls. Leverage informers and listers to access cached data whenever possible.
• Avoid Busy Loops: Ensure your Reconcile function doesn't enter an infinite loop or repeatedly make expensive operations without necessary checks. Use ObservedGeneration to prevent reconciliation if the spec hasn't changed.
• Workqueue Rate Limiting: Configure workqueues with appropriate rate limits and exponential backoff to prevent overwhelming the api server or external services during retries.
• Selective Updates: When updating resources, only modify the fields that have actually changed to minimize api traffic and potential conflicts.

Security: Principle of Least Privilege

Always configure your controller's Role-Based Access Control (RBAC) with the principle of least privilege. Grant only the verbs (get, list, watch, create, update, patch, delete) and resources that your controller absolutely needs. Over-privileging a controller is a significant security risk. Kubebuilder's +kubebuilder:rbac: markers help generate granular RBAC rules.

The APIPark Mention: Managing External API Integrations

As controllers become more sophisticated, they often move beyond just orchestrating Kubernetes native resources. Many advanced controllers integrate with external services, cloud providers, or even other microservices through various APIs. For instance, our Website controller could be extended to: * Provision DNS records via a cloud provider's DNS api. * Order SSL certificates from a certificate authority's api. * Integrate with a content delivery network (CDN) api. * Push deployment notifications to a messaging api.

Each of these integrations introduces its own set of challenges: managing API keys, rate limiting, monitoring, versioning, and ensuring secure communication. Ensuring consistent authentication, rate limiting, and monitoring across diverse APIs, whether internal or external, becomes paramount. This is where platforms like APIPark offer significant value. APIPark, an open-source AI gateway and API management platform, simplifies the integration and deployment of AI and REST services. It provides a unified API format for AI invocation, end-to-end API lifecycle management, and powerful data analysis, making it an invaluable tool for enterprises dealing with a multitude of api interactions, including those orchestrated by Kubernetes controllers. By streamlining api governance, APIPark helps ensure the efficiency and security of the broader ecosystem your controllers interact with, allowing your controllers to reliably connect with external services without reinventing complex api management logic for each integration. It's a prime example of how specialized platforms can complement Kubernetes' extensibility by handling the complexities of api consumption and exposure.

These advanced topics and best practices are crucial for building production-grade Kubernetes controllers. They move beyond the basic creation of resources, focusing on resilience, maintainability, and operational excellence, ensuring your custom automation works reliably in complex, real-world environments.

Deployment of Your Controller

With the controller logic complete and best practices considered, the final step is to build its Docker image and deploy it to a Kubernetes cluster. Kubebuilder provides helpful scaffolding for this process.

1. Building the Docker Image for Your Controller

First, ensure you are in the root directory of your website-controller project.

# Build the controller binary
make manager

# Build the Docker image
# Replace <your-docker-registry> with your Docker Hub username or private registry.
# For example: docker.io/myusername/website-controller:v0.0.1
docker build -t <your-docker-registry>/website-controller:v0.0.1 .

# Push the Docker image to your registry
docker push <your-docker-registry>/website-controller:v0.0.1

• make manager: This command compiles your Go controller code into an executable binary.
• docker build: Uses the Dockerfile generated by Kubebuilder (located in your project root) to create a Docker image. The Dockerfile typically builds a minimal image using a multi-stage build, resulting in a small and secure final image.
• docker push: Uploads your image to a container registry. Ensure you are logged in to your Docker registry (docker login).

2. Creating the Deployment YAML for Your Controller

Kubebuilder generates deployment manifests in the config/ directory.

• config/rbac/role.yaml: Defines the ClusterRole with the permissions (verbs on resources) specified by your +kubebuilder:rbac: markers in controllers/website_controller.go.
• config/rbac/role_binding.yaml: Binds the ClusterRole to a ServiceAccount.
• config/rbac/service_account.yaml: Defines the ServiceAccount for your controller.
• config/manager/manager.yaml: Defines the Deployment for your controller, including the container image, resource limits, and environment variables.

Before applying, you need to update the manager.yaml to point to your newly pushed Docker image.

Open config/manager/manager.yaml and locate the image field under the controller container. Change controller:latest to your image:

# ... (rest of the file)
      containers:
      - command:
        - /manager
        args:
        - --leader-elect
        image: <your-docker-registry>/website-controller:v0.0.1 # <--- Update this line
        name: manager
# ... (rest of the file)

3. RBAC Setup: ServiceAccount, ClusterRole, ClusterRoleBinding

The make manifests command (which you would have run earlier to generate the CRD) also updates the RBAC manifests. It's crucial that your controller has the correct permissions to interact with Website CRs, Deployments, Services, and Ingresses.

The generated config/rbac directory contains the necessary YAML files.

4. Deploying to a Kubernetes Cluster

With the Docker image pushed and manager.yaml updated, you can now deploy your controller.

1. Deploy CRDs (if not already done): bash kubectl apply -f config/crd/bases/example.com_websites.yaml
2. Deploy RBAC and Manager (Controller): bash kubectl apply -f config/rbac/ kubectl apply -f config/manager/Alternatively, you can use make deploy after setting the IMG environment variable: bash export IMG=<your-docker-registry>/website-controller:v0.0.1 make deploy This command will apply all the necessary config manifests.

5. Monitoring Logs

Once deployed, you can check if your controller pod is running and monitor its logs:

# Get the name of your controller pod (it will be something like website-controller-manager-...)
kubectl get pods -n website-controller-system

# View logs
kubectl logs -f <your-controller-pod-name> -n website-controller-system

Note: Kubebuilder typically deploys controllers into a namespace named <project-name>-system, e.g., website-controller-system.

If everything is set up correctly, you should see logs indicating that the manager is starting, caches are syncing, and the controller is ready to reconcile Website resources.

6. Testing Your Deployed Controller

Now that your controller is running, create an instance of your Website custom resource:

# website-test.yaml
apiVersion: example.com/v1
kind: Website
metadata:
  name: my-first-website
  namespace: default # Or any namespace where your controller has permissions
spec:
  domain: my-first-website.local # Make sure this domain resolves in your environment or use a local hosts file mapping
  contentSource: nginx:latest # Or any other web server image

Apply this resource:

kubectl apply -f website-test.yaml

Then observe: * Controller Logs: Your controller logs should show it detecting the new Website CR, creating the Deployment, Service, and Ingress. * Kubernetes Resources: bash kubectl get website my-first-website kubectl get deployment my-first-website-deployment kubectl get service my-first-website-service kubectl get ingress my-first-website-ingress You should see these resources created and eventually the Website CR's STATUS.READY become True. * Update: Try modifying website-test.yaml (e.g., change contentSource to httpd:latest) and re-apply. Observe the controller updating the deployment. * Delete: Delete the Website CR: kubectl delete website my-first-website. Observe the controller cleaning up the dependent resources and then deleting the Website CR itself.

By following these deployment steps and diligently monitoring the logs, you can successfully deploy and verify the operation of your custom Kubernetes controller, bringing your declarative api extensions to life.

Conclusion

Our journey through building a Kubernetes controller to watch CRD changes has traversed the fundamental principles of Kubernetes extensibility, from the conceptual power of Custom Resource Definitions to the intricate implementation of a fully functional controller. We began by understanding how CRDs empower us to extend Kubernetes' native apis with domain-specific objects, transforming the platform into an even more versatile application management system. This laid the groundwork for appreciating the necessity of controllers—the vigilant, self-healing control loops that breathe life into these custom resources.

We then meticulously set up a robust development environment, highlighting the role of Go, Docker, and kubectl, and emphasizing the invaluable assistance provided by Kubebuilder in scaffolding our project. The design of our Website CRD showcased the importance of a well-defined spec for declarative input and a status field for operational feedback, all while ensuring data integrity through openAPIV3Schema validation.

The core of our exploration was the deep dive into the components that make a Kubernetes controller tick: the efficient event watching facilitated by Informers, the local data access provided by Listers, the reliable event processing managed by Workqueues, and the central orchestration capability of the controller-runtime Manager. We meticulously implemented the Reconcile function, demonstrating how to fetch desired states, compare them with actual states, and take corrective actions to create, update, or delete dependent Kubernetes resources like Deployments, Services, and Ingresses. The critical role of OwnerReferences for garbage collection and Finalizers for external cleanup was also highlighted.

Furthermore, we explored advanced topics that elevate a basic controller to a production-grade solution. Concepts such as webhooks for api interception, status subresources for fine-grained updates, and the paramount importance of thorough testing, logging, metrics, and adherence to the principle of least privilege for security were discussed. In discussing the broader api ecosystem controllers often interact with, we naturally integrated the mention of APIPark, an open-source AI gateway and API management platform that offers a streamlined approach to managing the diverse api integrations crucial for sophisticated controllers.

Finally, we covered the practical steps of containerizing our controller, preparing its Kubernetes deployment manifests, setting up RBAC permissions, and deploying it to a cluster. The iterative process of testing and monitoring logs reinforced the full lifecycle of controller development.

The ability to build custom Kubernetes controllers is a testament to the platform's open and extensible design. It empowers developers and operators to automate highly specialized workflows, create powerful, self-managing applications, and integrate complex external systems seamlessly within their cloud-native environments. As the demand for sophisticated, domain-aware automation continues to grow, mastering controller development will remain an indispensable skill, enabling you to truly harness the full potential of Kubernetes and shape its future capabilities. The cloud-native landscape is continuously evolving, and with the tools and understanding gained here, you are well-equipped to contribute to its next wave of innovation.

Frequently Asked Questions (FAQ)

1. What is the main difference between a Custom Resource (CR) and a Custom Resource Definition (CRD)? A CRD (Custom Resource Definition) is a schema definition that tells Kubernetes what your new custom api object will look like, including its name, group, version, and the structure of its spec and status fields. It's like a blueprint for a new type of resource. A CR (Custom Resource) is an actual instance of that custom api object, created by users according to the CRD's blueprint. So, you define a CRD once, and then you can create many CRs based on that definition.

2. Why do I need a controller if I have a CRD? A CRD only defines the schema for a new resource type; it doesn't add any operational logic. When you create a Custom Resource, the Kubernetes api server accepts and stores it, but nothing actually "happens" in your cluster as a result. A Kubernetes controller is a program that watches for changes to those Custom Resources and then takes specific actions (e.g., creating Deployments, Services, Ingresses, or interacting with external apis) to bring the actual state of the cluster in line with the desired state declared in the CR. Without a controller, your Custom Resources are just inert data objects.

3. What is the role of controller-runtime and kubebuilder in controller development? client-go is the low-level Go client library for interacting with the Kubernetes api. However, building a controller directly with client-go involves a lot of boilerplate code for informers, listers, and workqueues. controller-runtime is a higher-level library built on client-go that abstracts away much of this complexity, providing a structured framework (like the Manager and Reconciler interface) to simplify controller development. kubebuilder is a command-line tool and framework that sits on top of controller-runtime. It generates boilerplate code, CRD YAML, RBAC manifests, and provides utilities that streamline the entire development process, allowing developers to focus primarily on the core reconciliation logic.

4. How do I make my controller delete associated resources when the Custom Resource is deleted? The most common and recommended way to achieve this is by setting OwnerReferences on the dependent resources (e.g., Deployments, Services) that your controller creates. In controller-runtime, this is typically done using controllerutil.SetControllerReference(owner, dependent, scheme). When an owner resource (your Custom Resource) is deleted, Kubernetes' garbage collector will automatically delete all resources that have that Custom Resource as an owner. For more complex cleanup scenarios, especially involving external resources or ordered deletions, Finalizers are used in conjunction with OwnerReferences.

5. How can I ensure my controller is robust and doesn't overload the Kubernetes API server? Several best practices contribute to a robust and efficient controller: * Leverage Informers and Listers: Always read resource states from the informer's local cache via listers instead of making direct api server calls for reads, which significantly reduces api server load. * Workqueue Rate Limiting: Implement exponential backoff and rate limiting in your workqueue to prevent rapid, consecutive retries during transient errors. * Idempotency: Ensure your Reconcile logic is idempotent, meaning applying the same desired state multiple times yields the same result without side effects. Check for existence and compare specs before attempting to create or update resources. * Status Subresource: Use the status subresource for status updates, which often requires fewer permissions and can prevent conflicts with spec updates. * Observability: Implement comprehensive logging and metrics to quickly identify performance bottlenecks or error patterns. * Least Privilege RBAC: Grant your controller only the minimal api permissions it needs to perform its functions.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.