Kubernetes: Build a Controller to Watch for Changes to CRD
In the dynamic landscape of modern software development, where applications are increasingly distributed, resilient, and declarative, Kubernetes has emerged as the de facto operating system for the cloud. It provides a robust platform for automating the deployment, scaling, and management of containerized workloads. However, the power of Kubernetes extends far beyond its built-in functionalities; it offers unparalleled extensibility, allowing users to tailor the platform to their unique operational needs and application requirements. This extensibility is primarily realized through Custom Resource Definitions (CRDs) and the creation of custom controllers, often referred to as operators.
This comprehensive guide delves into the intricate process of building a Kubernetes controller specifically designed to observe and react to changes in Custom Resource Definitions. We will embark on a journey that begins with understanding the fundamental concepts of Kubernetes extensibility, moves through the core architecture of controllers, and culminates in a detailed, step-by-step exploration of how to implement such a system using Go and the client-go library. By the end of this article, you will possess a profound understanding of how to empower your Kubernetes clusters with custom logic, transforming them from generic container orchestrators into highly specialized, domain-aware platforms capable of managing virtually any kind of application or infrastructure.
The Foundation of Extensibility: Kubernetes API and Custom Resources
At its core, Kubernetes is an API-driven system. Every interaction, every operation, and every piece of information within a Kubernetes cluster flows through its central component: the kube-apiserver. This powerful component acts as the front-end for the Kubernetes control plane, exposing a RESTful API that allows users and other control plane components to query, create, update, and delete objects within the cluster. The strength of this API-centric design is its uniformity and extensibility.
Initially, Kubernetes provided a set of built-in resource types like Pods, Deployments, Services, and Namespaces. While these are sufficient for many common use cases, real-world applications often involve complex, application-specific infrastructure or business logic that doesn't fit neatly into these predefined categories. For instance, you might want to manage a database instance, an external CDN configuration, or a specialized machine learning model deployment directly through Kubernetes, treating these external concerns as first-class citizens of your cluster. This is precisely where Custom Resource Definitions (CRDs) come into play.
Unveiling Custom Resource Definitions (CRDs)
A Custom Resource Definition (CRD) is a powerful mechanism that allows you to define your own resource types within a Kubernetes cluster. It effectively extends the Kubernetes API schema, telling the kube-apiserver about a new kind of object it should recognize and persist. Once a CRD is created, you can then create instances of that custom resource, much like you would create a Pod or a Deployment. These instances are called Custom Resources (CRs).
Let's illustrate with a practical example. Imagine you want to manage database instances (e.g., PostgreSQL, MySQL) declaratively within your Kubernetes cluster. Instead of manually provisioning databases outside Kubernetes and then deploying applications that connect to them, you could define a Database CRD.
Here’s what a simplified Database CRD might look like:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
enum: ["PostgreSQL", "MySQL"]
description: The database engine to use.
version:
type: string
description: The version of the database engine.
size:
type: string
description: The desired storage size (e.g., 10Gi, 500Mi).
users:
type: array
items:
type: object
properties:
username: {type: string}
passwordSecret: {type: string}
description: A list of users for the database.
required: ["engine", "version", "size"]
status:
type: object
properties:
phase: {type: string}
connectionString: {type: string}
ready: {type: boolean}
scope: Namespaced # Or Cluster, depending on your needs
names:
plural: databases
singular: database
kind: Database
shortNames: ["db"]
Once this CRD is applied to your cluster (kubectl apply -f database-crd.yaml), the Kubernetes API server now understands a new resource type: Database within the stable.example.com group. You can then create instances of this custom resource, like so:
apiVersion: stable.example.com/v1
kind: Database
metadata:
name: my-app-db
spec:
engine: PostgreSQL
version: "14"
size: "50Gi"
users:
- username: admin
passwordSecret: my-db-admin-secret
- username: appuser
passwordSecret: my-db-appuser-secret
Creating this Database Custom Resource (CR) will register an object in the Kubernetes API server's persistent storage (etcd). However, at this point, nothing actually happens. Kubernetes merely stores the information. It doesn't automatically provision a PostgreSQL instance, create users, or generate connection strings. This is where controllers come into play.
CRDs empower developers and operators to model their infrastructure and applications directly within Kubernetes using a declarative syntax. They transform Kubernetes from a generic container orchestrator into a powerful, domain-specific platform. The consistency of the Kubernetes API for all resources, whether built-in or custom, significantly simplifies operations and automation.
The Brains Behind the Operations: Kubernetes Controllers
A Kubernetes controller is an active reconciliation loop that continuously monitors the state of resources within the cluster and makes changes to drive the current state towards a desired state. It's the "engine" that observes your Custom Resources and takes action based on their specifications. Think of it as an autonomous agent that tirelessly works to ensure the reality matches your declarations.
The Reconciliation Loop
The core principle behind every Kubernetes controller is the reconciliation loop. This loop involves three primary steps:
- Observe: The controller watches for changes to specific resource types (e.g., Pods, Deployments, or in our case, our custom
Databaseresources). This observation is not a constant polling of the API server, which would be inefficient. Instead, controllers leverage an efficient watch mechanism provided by the Kubernetes API server that pushes events (Add, Update, Delete) to the controller when relevant resources change. - Analyze: When a change is detected for a resource it manages, the controller analyzes the desired state (as specified in the resource's
specfield) and compares it against the actual state of the world (e.g., what external database instances actually exist, what Kubernetes native resources like Deployments and Services are currently running). - Act: If a discrepancy is found, the controller takes corrective actions to bring the actual state in line with the desired state. For our
Databaseexample, this might involve:- Provisioning a new PostgreSQL instance in an external cloud provider if a
DatabaseCR is created. - Scaling up storage for an existing database if the
sizefield is updated. - Creating Kubernetes Secrets for database credentials.
- Deleting the external database instance if the
DatabaseCR is deleted. - Updating the
statusfield of theDatabaseCR to reflect the current state (e.g.,phase: Provisioning,phase: Ready,connectionString: ...).
- Provisioning a new PostgreSQL instance in an external cloud provider if a
Why Build Custom Controllers (Operators)?
While Kubernetes provides controllers for its built-in resources (e.g., the Deployment controller manages Pods and ReplicaSets), custom controllers (often packaged as "Operators") extend this pattern to your custom resources. They automate the operational knowledge that site reliability engineers (SREs) or database administrators (DBAs) would typically manually perform.
The benefits are substantial:
- Automation: Automate complex, multi-step tasks.
- Self-Healing: Controllers can automatically detect and fix issues.
- Standardization: Enforce consistent patterns for deploying and managing applications.
- Declarative Management: Treat application-specific infrastructure as first-class Kubernetes objects, managed through the familiar
kubectlinterface. - Reduced Operational Burden: Offload repetitive tasks, allowing engineers to focus on higher-value work.
For instance, an operator for our Database CRD wouldn't just provision a database; it could also handle backups, upgrades, monitoring integration, and failover scenarios—all driven by the declarative specification within the Database CR.
The Architecture of a Kubernetes Controller
Building a controller from scratch involves interacting with the Kubernetes API using client libraries, managing a local cache of resources, queuing events for processing, and implementing the core reconciliation logic. In the Go ecosystem, the client-go library is the standard toolkit for this task.
Let's break down the essential components of a typical client-go-based controller.
1. client-go: The Kubernetes Client Library for Go
client-go provides a set of Go packages that allow your application to communicate with the Kubernetes API server. It offers various levels of abstraction:
- RESTClient: The lowest-level client, directly interacting with the Kubernetes API via HTTP.
- Clientset: A higher-level client generated for all standard Kubernetes resource types. It provides methods like
Pods().Create(),Deployments().Get(), etc. - DynamicClient: A client that can interact with any resource, including custom resources, without needing compile-time knowledge of their Go types. Useful for generic tools.
- Informer/Lister: The most crucial components for controllers, providing efficient caching and event-driven notifications.
For building a controller for a specific CRD, we typically generate a clientset for our custom resource, which provides a type-safe way to interact with it.
2. Informers: Efficiently Watching and Caching Resources
Directly watching the Kubernetes API server for changes to resources for every reconciliation loop would be highly inefficient and put undue stress on the API server. This is where the Informer pattern comes in.
An Informer (SharedInformer) handles the complexities of watching resources and maintaining a local, consistent cache of those resources. Here’s how it works:
- List-Watch Mechanism: The Informer first performs a full
LISToperation to retrieve all existing resources of a specific type. It then establishes aWATCHconnection to the Kubernetes API server. - Local Cache: All resources retrieved from the
LISToperation and subsequentWATCHevents are stored in an in-memory cache within the controller. This cache is typically implemented using anIndexer, which allows for efficient lookups by key (e.g., namespace/name). - Event Handlers: When the Informer receives an event (Add, Update, Delete) from the API server, it updates its local cache and then invokes user-defined event handler functions (
AddFunc,UpdateFunc,DeleteFunc). These functions are where the controller is notified of changes.
The benefits of Informers are significant:
- Reduced API Server Load: By caching resources locally, the controller rarely needs to make direct API calls for read operations.
- Event-Driven: Controllers react to changes in real-time rather than polling.
- Concurrency Safety:
client-go's Informers are designed with concurrency in mind.
3. Listers: Reading from the Cache
Associated with an Informer is a Lister (Lister). A Lister provides a convenient, read-only interface to query the Informer's local cache. Instead of making an API call every time the controller needs to retrieve a resource, it simply queries its Lister, which is much faster and less resource-intensive. For example, databaseLister.Databases("my-namespace").Get("my-app-db") would retrieve our Database CR from the local cache.
4. Workqueue: Decoupling Event Handling from Processing
When an Informer's event handler is triggered, it's generally a bad idea to perform the heavy reconciliation logic directly within the handler. This is because handlers run in the Informer's goroutine, and blocking them can prevent the Informer from processing further events and keeping its cache up to date.
To solve this, controllers use a Workqueue (specifically, a RateLimitingWorkqueue). When an event handler is invoked, instead of performing reconciliation, it simply adds the key (typically namespace/name) of the affected resource to the workqueue.
The Workqueue provides several important features:
- Decoupling: Events are quickly enqueued, allowing the Informer to continue processing.
- Batching: Multiple updates to the same resource can be collapsed into a single item in the queue, preventing redundant processing.
- Retries and Rate Limiting: If processing an item fails, the workqueue can automatically re-add the item with an exponential backoff, preventing tight loops on persistently failing resources and reducing the load on external systems.
- Ordered Processing: Items related to the same resource are processed in order.
5. Reconciliation Logic: The SyncHandler
The core business logic of the controller resides in what's typically called the syncHandler or Reconcile function. This function is responsible for:
- Retrieving the resource from the local cache using the Lister based on the key received from the workqueue.
- Comparing the desired state (from the resource's
spec) with the actual state of the world. - Taking necessary actions (e.g., creating/updating/deleting dependent Kubernetes resources like Deployments, Services, or interacting with external APIs).
- Updating the
statusfield of the custom resource to reflect the current state. - Handling errors gracefully and potentially requeueing the item if transient failures occur.
Step-by-Step Implementation Guide: Building a CRD Controller
Let’s now walk through building a controller for our Database CRD. We will focus on the fundamental structure and interactions.
Prerequisites
Before we begin, ensure you have the following installed:
- Go: Version 1.16 or later.
- Docker: For building container images.
- Kubernetes Cluster: A local cluster like Minikube or Kind is ideal for development.
- kubectl: The Kubernetes command-line tool.
- controller-gen: A tool for generating client-go code for CRDs. Install it:
bash go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest
1. Project Setup and CRD Definition
First, create a new Go module and define our Database CRD.
mkdir database-controller
cd database-controller
go mod init github.com/your-org/database-controller
Create a file crd/database.yaml for our Database CRD as shown previously. Apply it to your cluster:
kubectl apply -f crd/database.yaml
Now, create a file crd/sample-database.yaml:
apiVersion: stable.example.com/v1
kind: Database
metadata:
name: my-first-db
namespace: default
spec:
engine: PostgreSQL
version: "14"
size: "20Gi"
users:
- username: admin
passwordSecret: "" # Will be populated by controller
2. Define Go Types for the Custom Resource
We need Go structs that represent our Database custom resource. These structs will be used by client-go to unmarshal and marshal the resource data. We'll also add // +kubebuilder markers that controller-gen uses.
Create pkg/apis/stable/v1/types.go:
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// Database is the Schema for the databases API
type Database struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseSpec `json:"spec,omitempty"`
Status DatabaseStatus `json:"status,omitempty"`
}
// DatabaseSpec defines the desired state of Database
type DatabaseSpec struct {
Engine string `json:"engine"`
Version string `json:"version"`
Size string `json:"size"`
Users []User `json:"users,omitempty"`
}
// User defines a database user
type User struct {
Username string `json:"username"`
PasswordSecret string `json:"passwordSecret"` // Name of a secret to hold the password
}
// DatabaseStatus defines the observed state of Database
type DatabaseStatus struct {
Phase string `json:"phase,omitempty"`
ConnectionString string `json:"connectionString,omitempty"`
Ready bool `json:"ready,omitempty"`
Message string `json:"message,omitempty"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// DatabaseList contains a list of Database
type DatabaseList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Database `json:"items"`
}
Now, generate the client-go code:
controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./pkg/apis/..."
# You'll need to create hack/boilerplate.go.txt with a license header, e.g.:
# // +build !ignore_autogenerated
# // Code generated by controller-gen. DO NOT EDIT.
# //go:build !ignore_autogenerated
# // +build !ignore_autogenerated
#
# /*
# Copyright The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# */
This will generate pkg/apis/stable/v1/zz_generated.deepcopy.go and pkg/client directories with clientsets, informers, and listers for our Database resource. Run go mod tidy to update dependencies.
3. Implement the Controller Logic
We will organize our controller into a main.go file (for setup) and a controller.go file (for the core logic).
controller.go
This file will contain the Controller struct and its methods.
package controller
import (
"context"
"fmt"
"time"
"github.com/go-logr/logr"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/runtime"
"k8s.io/apimachinery/pkg/util/wait"
kubeinformers "k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/kubernetes/scheme"
typedcorev1 "k8s.io/client-go/kubernetes/typed/core/v1"
appslisters "k8s.io/client-go/listers/apps/v1"
corelisters "k8s.io/client-go/listers/core/v1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/record"
"k8s.io/client-go/util/workqueue"
clientset "github.com/your-org/database-controller/pkg/client/clientset/versioned"
databasescheme "github.com/your-org/database-controller/pkg/client/clientset/versioned/scheme"
informers "github.com/your-org/database-controller/pkg/client/informers/externalversions"
listers "github.com/your-org/database-controller/pkg/client/listers/stable/v1"
stablev1 "github.com/your-org/database-controller/pkg/apis/stable/v1" // Import our CRD type
)
const controllerAgentName = "database-controller"
// Controller is the controller for Database resources
type Controller struct {
kubeclientset kubernetes.Interface
databaseclientset clientset.Interface
deploymentsLister appslisters.DeploymentLister
deploymentsSynced cache.InformerSynced
databasesLister listers.DatabaseLister
databasesSynced cache.InformerSynced
secretsLister corelisters.SecretLister
secretsSynced cache.InformerSynced
workqueue workqueue.RateLimitingInterface
recorder record.EventRecorder
logger logr.Logger
}
// NewController returns a new Database controller
func NewController(
kubeclientset kubernetes.Interface,
databaseclientset clientset.Interface,
kubeInformerFactory kubeinformers.SharedInformerFactory,
databaseInformerFactory informers.SharedInformerFactory,
logger logr.Logger) *Controller {
logger.V(4).Info("Creating event broadcaster")
databasescheme.AddToScheme(scheme.Scheme)
eventBroadcaster := record.NewBroadcaster()
eventBroadcaster.StartStructuredLogging(0)
eventBroadcaster.StartRecordingToSink(&typedcorev1.EventSinkImpl{Interface: kubeclientset.CoreV1().Events("")})
recorder := eventBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{Component: controllerAgentName})
deploymentInformer := kubeInformerFactory.Apps().V1().Deployments()
secretInformer := kubeInformerFactory.Core().V1().Secrets()
databaseInformer := databaseInformerFactory.Stable().V1().Databases()
controller := &Controller{
kubeclientset: kubeclientset,
databaseclientset: databaseclientset,
deploymentsLister: deploymentInformer.Lister(),
deploymentsSynced: deploymentInformer.Informer().HasSynced,
secretsLister: secretInformer.Lister(),
secretsSynced: secretInformer.Informer().HasSynced,
databasesLister: databaseInformer.Lister(),
databasesSynced: databaseInformer.Informer().HasSynced,
workqueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "Databases"),
recorder: recorder,
logger: logger.WithName("database-controller"),
}
logger.Info("Setting up event handlers")
// Set up an event handler for when Database resources change
databaseInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.enqueueDatabase,
UpdateFunc: func(old, new interface{}) {
controller.enqueueDatabase(new)
},
DeleteFunc: controller.enqueueDatabase,
})
// Set up event handlers for resources created by Database controllers
deploymentInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
newDepl := new.(*appsv1.Deployment)
oldDepl := old.(*appsv1.Deployment)
if newDepl.ResourceVersion == oldDepl.ResourceVersion {
// Periodic resync will send update events for the object without changes.
// We don't want to reprocess if the object's resource version is unchanged.
return
}
controller.handleObject(new)
},
DeleteFunc: controller.handleObject,
})
secretInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
newSecret := new.(*corev1.Secret)
oldSecret := old.(*corev1.Secret)
if newSecret.ResourceVersion == oldSecret.ResourceVersion {
return
}
controller.handleObject(new)
},
DeleteFunc: controller.handleObject,
})
return controller
}
// Run will set up the event handlers for types we are interested in, as well
// as syncing informer caches and starting workers. It will block until stopCh
// is closed, at which point it will shutdown the workqueue and wait for
// workers to finish processing their current work items.
func (c *Controller) Run(workers int, stopCh <-chan struct{}) error {
defer runtime.HandleCrash()
defer c.workqueue.ShutDown()
// Start the informer factories to begin populating the informer caches
c.logger.Info("Starting Database controller")
// Wait for the caches to be synced before starting workers
c.logger.Info("Waiting for informer caches to sync")
if ok := cache.WaitForCacheSync(stopCh, c.deploymentsSynced, c.databasesSynced, c.secretsSynced); !ok {
return fmt.Errorf("failed to wait for caches to sync")
}
c.logger.Info("Starting workers")
for i := 0; i < workers; i++ {
go wait.Until(c.runWorker, time.Second, stopCh)
}
c.logger.Info("Started workers")
<-stopCh
c.logger.Info("Shutting down workers")
return nil
}
// runWorker is a long-running function that will continually call the
// processNextWorkItem function in order to read and process a message on the
// workqueue.
func (c *Controller) runWorker() {
for c.processNextWorkItem() {
}
}
// processNextWorkItem will read a single item from the workqueue and
// attempt to process it, by calling the syncHandler.
func (c *Controller) processNextWorkItem() bool {
obj, shutdown := c.workqueue.Get()
if shutdown {
return false
}
// We wrap this block in a func so we can defer c.workqueue.Done.
err := func(obj interface{}) error {
defer c.workqueue.Done(obj)
var key string
var ok bool
if key, ok = obj.(string); !ok {
// As the item in the workqueue is actually a string, we cannot cast it to the expected type.
runtime.HandleError(fmt.Errorf("expected string in workqueue but got %#v", obj))
c.workqueue.Forget(obj)
return nil
}
// Run the syncHandler, passing it the namespace/name string of the
// Foo resource to be synced.
if err := c.syncHandler(key); err != nil {
// Put the item back on the workqueue to handle any transient errors.
c.workqueue.AddRateLimited(key)
return fmt.Errorf("error syncing '%s': %s, requeuing", key, err.Error())
}
// If no error occurs we Forget this item so it won't be retried again.
c.workqueue.Forget(obj)
c.logger.V(4).Info("Successfully synced", "resourceName", key)
return nil
}(obj)
if err != nil {
runtime.HandleError(err)
return true
}
return true
}
// enqueueDatabase takes a Database resource and converts it into a namespace/name
// string which is then put onto the work queue. This method should *not* be
// passed object which may be mutated by the informer's cache.
func (c *Controller) enqueueDatabase(obj interface{}) {
var key string
var err error
if key, err = cache.MetaNamespaceKeyFunc(obj); err != nil {
runtime.HandleError(err)
return
}
c.workqueue.Add(key)
}
// handleObject will take any resource implementing metav1.Object and attempt
// to find the Database resource that owns it. It does this by looking at the
// objects metadata.ownerReferences. It then enqueues that Foo resource to be
// processed. If the object does not have an appropriate OwnerReference, it
// will simply be skipped.
func (c *Controller) handleObject(obj interface{}) {
var object metav1.Object
var ok bool
if object, ok = obj.(metav1.Object); !ok {
runtime.HandleError(fmt.Errorf("expected metav1.Object but got %#v", obj))
return
}
if ownerRef := metav1.Get Controller Component & Role |
|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ APIs.
c.kubeclientset.CoreV1().Secrets(database.Namespace).Create(context.TODO(), createPostgreSQLSecret(database), metav1.CreateOptions{})
```
### Table: Key Controller Components and Their Roles
To summarize the roles of the various components we've discussed, here's a detailed table:
| Controller Component | Description # This file implements the Database controller (from our Database example) with the kubebuilder-specific manifests.
# The controller follows the usual Kubernetes controller pattern, where it
# watches for changes to the resource, then verifies if the actual state
# matches the desired state, then makes the changes (in this example, creates
# a Deployment and a Service), and updates the status of the resource.
#
# This controller is a minimal example, and should be extended with
# error handling, retries, and more robust state management.
#
package controller
import (
"context"
"fmt"
"time"
"github.com/go-logr/logr"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/runtime"
"k8s.io/apimachinery/pkg/util/wait"
kubeinformers "k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/kubernetes/scheme"
typedcorev1 "k8s.io/client-go/kubernetes/typed/core/v1"
appslisters "k8s.io/client-go/listers/apps/v1"
corelisters "k8s.io/client-go/listers/core/v1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/tools/record"
"k8s.io/client-go/util/workqueue"
clientset "github.com/your-org/database-controller/pkg/client/clientset/versioned"
databasescheme "github.com/your-org/database-controller/pkg/client/clientset/versioned/scheme"
informers "github.com/your-org/database-controller/pkg/client/informers/externalversions"
listers "github.com/your-org/database-controller/pkg/client/listers/stable/v1"
stablev1 "github.com/your-org/database-controller/pkg/apis/stable/v1" // Import our CRD type
)
const controllerAgentName = "database-controller"
// Controller is the controller for Database resources
type Controller struct {
kubeclientset kubernetes.Interface
databaseclientset clientset.Interface
deploymentsLister appslisters.DeploymentLister
deploymentsSynced cache.InformerSynced
databasesLister listers.DatabaseLister
databasesSynced cache.InformerSynced
secretsLister corelisters.SecretLister
secretsSynced cache.InformerSynced
workqueue workqueue.RateLimitingInterface
recorder record.EventRecorder
logger logr.Logger
}
// NewController returns a new Database controller
func NewController(
kubeclientset kubernetes.Interface,
databaseclientset clientset.Interface,
kubeInformerFactory kubeinformers.SharedInformerFactory,
databaseInformerFactory informers.SharedInformerFactory,
logger logr.Logger) *Controller {
logger.V(4).Info("Creating event broadcaster")
databasescheme.AddToScheme(scheme.Scheme)
eventBroadcaster := record.NewBroadcaster()
eventBroadcaster.StartStructuredLogging(0)
eventBroadcaster.StartRecordingToSink(&typedcorev1.EventSinkImpl{Interface: kubeclientset.CoreV1().Events("")})
recorder := eventBroadcaster.NewRecorder(scheme.Scheme, corev1.EventSource{Component: controllerAgentName})
deploymentInformer := kubeInformerFactory.Apps().V1().Deployments()
secretInformer := kubeInformerFactory.Core().V1().Secrets()
databaseInformer := databaseInformerFactory.Stable().V1().Databases()
controller := &Controller{
kubeclientset: kubeclientset,
databaseclientset: databaseclientset,
deploymentsLister: deploymentInformer.Lister(),
deploymentsSynced: deploymentInformer.Informer().HasSynced,
secretsLister: secretInformer.Lister(),
secretsSynced: secretInformer.Informer().HasSynced,
databasesLister: databaseInformer.Lister(),
databasesSynced: databaseInformer.Informer().HasSynced,
workqueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "Databases"),
recorder: recorder,
logger: logger.WithName("database-controller"),
}
logger.Info("Setting up event handlers")
// Set up an event handler for when Database resources change
databaseInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.enqueueDatabase,
UpdateFunc: func(old, new interface{}) {
controller.enqueueDatabase(new)
},
DeleteFunc: controller.enqueueDatabase,
})
// Set up event handlers for resources created by Database controllers
deploymentInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
newDepl := new.(*appsv1.Deployment)
oldDepl := old.(*appsv1.Deployment)
if newDepl.ResourceVersion == oldDepl.ResourceVersion {
// Periodic resync will send update events for the object without changes.
// We don't want to reprocess if the object's resource version is unchanged.
return
}
controller.handleObject(new)
},
DeleteFunc: controller.handleObject,
})
secretInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
newSecret := new.(*corev1.Secret)
oldSecret := old.(*corev1.Secret)
if newSecret.ResourceVersion == oldSecret.ResourceVersion {
return
}
controller.handleObject(new)
},
DeleteFunc: controller.handleObject,
})
return controller
}
// Run will set up the event handlers for types we are interested in, as well
// as syncing informer caches and starting workers. It will block until stopCh
// is closed, at which point it will shutdown the workqueue and wait for
// workers to finish processing their current work items.
func (c *Controller) Run(workers int, stopCh <-chan struct{}) error {
defer runtime.HandleCrash()
defer c.workqueue.ShutDown()
// Start the informer factories to begin populating the informer caches
c.logger.Info("Starting Database controller")
// Wait for the caches to be synced before starting workers
c.logger.Info("Waiting for informer caches to sync")
if ok := cache.WaitForCacheSync(stopCh, c.deploymentsSynced, c.databasesSynced, c.secretsSynced); !ok {
return fmt.Errorf("failed to wait for caches to sync")
}
c.logger.Info("Starting workers")
for i := 0; i < workers; i++ {
go wait.Until(c.runWorker, time.Second, stopCh)
}
c.logger.Info("Started workers")
<-stopCh
c.logger.Info("Shutting down workers")
return nil
}
// runWorker is a long-running function that will continually call the
// processNextWorkItem function in order to read and process a message on the
// workqueue.
func (c *Controller) runWorker() {
for c.processNextWorkItem() {
}
}
// processNextWorkItem will read a single item from the workqueue and
// attempt to process it, by calling the syncHandler.
func (c *Controller) processNextWorkItem() bool {
obj, shutdown := c.workqueue.Get()
if shutdown {
return false
}
// We wrap this block in a func so we can defer c.workqueue.Done.
err := func(obj interface{}) error {
defer c.workqueue.Done(obj)
var key string
var ok bool
if key, ok = obj.(string); !ok {
// As the item in the workqueue is actually a string, we cannot cast it to the expected type.
runtime.HandleError(fmt.Errorf("expected string in workqueue but got %#v", obj))
c.workqueue.Forget(obj)
return nil
}
// Run the syncHandler, passing it the namespace/name string of the
// Foo resource to be synced.
if err := c.syncHandler(key); err != nil {
// Put the item back on the workqueue to handle any transient errors.
c.workqueue.AddRateLimited(key)
return fmt.Errorf("error syncing '%s': %s, requeuing", key, err.Error())
}
// If no error occurs we Forget this item so it won't be retried again.
c.workqueue.Forget(obj)
c.logger.V(4).Info("Successfully synced", "resourceName", key)
return nil
}(obj)
if err != nil {
runtime.HandleError(err)
return true
}
return true
}
// enqueueDatabase takes a Database resource and converts it into a namespace/name
// string which is then put onto the work queue. This method should *not* be
// passed object which may be mutated by the informer's cache.
func (c *Controller) enqueueDatabase(obj interface{}) {
var key string
var err error
if key, err = cache.MetaNamespaceKeyFunc(obj); err != nil {
runtime.HandleError(err)
return
}
c.workqueue.Add(key)
}
// handleObject will take any resource implementing metav1.Object and attempt
// to find the Database resource that owns it. It does this by looking at the
// objects metadata.ownerReferences. It then enqueues that Foo resource to be
// processed. If the object does not have an appropriate OwnerReference, it
// will simply be skipped.
func (c *Controller) handleObject(obj interface{}) {
var object metav1.Object
var ok bool
if object, ok = obj.(metav1.Object); !ok {
runtime.HandleError(fmt.Errorf("expected metav1.Object but got %#v", obj))
return
}
if ownerRef := metav1.GetControllerOf(object); ownerRef != nil {
// We only care about objects that are owned by a Database
if ownerRef.Kind != "Database" {
return
}
database, err := c.databasesLister.Databases(object.GetNamespace()).Get(ownerRef.Name)
if err != nil {
c.logger.V(4).Info("ignoring orphaned object '%s/%s' of foo '%s'", object.GetNamespace(), object.GetName(), ownerRef.Name)
return
}
c.enqueueDatabase(database)
return
}
}
// syncHandler compares the actual state with the desired, and attempts to
// converge the two. It is the main reconciliation logic.
func (c *Controller) syncHandler(key string) error {
// Convert the namespace/name string into a distinct namespace and name
namespace, name, err := cache.SplitMetaNamespaceKey(key)
if err != nil {
runtime.HandleError(fmt.Errorf("invalid resource key: %s", key))
return nil
}
// Get the Database resource with the namespace/name from informer cache
database, err := c.databasesLister.Databases(namespace).Get(name)
if err != nil {
// The Database resource may no longer exist, in which case we stop
// processing.
if errors.IsNotFound(err) {
c.logger.V(4).Info("Database '%s' in work queue no longer exists", key)
// TODO: Clean up external resources here, like the actual PostgreSQL instance.
return nil
}
return err
}
// --- Core Reconciliation Logic ---
// 1. Ensure a Secret exists for the database user passwords
// This would typically generate strong passwords and store them securely.
secretName := fmt.Sprintf("%s-secrets", database.Name)
secret, err := c.secretsLister.Secrets(database.Namespace).Get(secretName)
if errors.IsNotFound(err) {
c.logger.Info("Creating Secret for Database", "database", key, "secret", secretName)
secret, err = c.kubeclientset.CoreV1().Secrets(database.Namespace).Create(context.TODO(), createPostgreSQLSecret(database), metav1.CreateOptions{})
if err != nil {
return fmt.Errorf("failed to create secret '%s': %w", secretName, err)
}
c.recorder.Event(database, corev1.EventTypeNormal, "CreatedSecret", fmt.Sprintf("Created secret %q", secret.Name))
} else if err != nil {
return err
} else {
// Check if the secret needs updating (e.g., new users added, rotate passwords)
// For simplicity, we assume secret content is static once created here.
}
// 2. Provision / Manage the external database (mocked for this example)
// In a real-world scenario, this is where you'd interact with a cloud provider API,
// or an on-premise database management system.
// For example, using a cloud provider SDK to create a PostgreSQL instance.
c.logger.Info("Simulating external database provisioning for", "database", key)
// Placeholder for external API calls
externalDBConnectionString := fmt.Sprintf("jdbc:postgresql://%s.external.example.com:5432/%s", database.Name, database.Name)
// 3. Update the Database CR status
// Reflect the current state of the external resource and its availability.
if database.Status.Phase == "" || database.Status.Phase == "Provisioning" {
c.recorder.Event(database, corev1.EventTypeNormal, "Provisioning", "Initiated external database provisioning")
database.Status.Phase = "Provisioning"
database.Status.Ready = false
database.Status.Message = "External database is being provisioned."
} else if database.Status.Phase == "Provisioning" {
// Simulate provisioning time
// In a real controller, you would poll the external API for completion.
// For demonstration, we'll "complete" it after a few syncs.
if time.Now().Minute()%2 == 0 { // Simple condition for demo purposes
c.logger.Info("Simulating external database ready for", "database", key)
database.Status.Phase = "Ready"
database.Status.Ready = true
database.Status.ConnectionString = externalDBConnectionString
database.Status.Message = "External database is ready."
c.recorder.Event(database, corev1.EventTypeNormal, "Provisioned", "External database provisioned successfully")
} else {
// Requeue to check status again later
return fmt.Errorf("external database still provisioning, requeuing for status check")
}
}
// Update the CR status
_, err = c.databaseclientset.StableV1().Databases(database.Namespace).UpdateStatus(context.TODO(), database, metav1.UpdateOptions{})
if err != nil {
return fmt.Errorf("failed to update status of Database '%s': %w", key, err)
}
c.recorder.Event(database, corev1.EventTypeNormal, "Synced", "Database synced successfully")
return nil
}
// createPostgreSQLSecret creates a dummy secret for the database.
// In a real scenario, this would generate robust passwords.
func createPostgreSQLSecret(database *stablev1.Database) *corev1.Secret {
// Generate dummy passwords for now
data := make(map[string][]byte)
for _, user := range database.Spec.Users {
data[user.Username] = []byte("superSecurePassword123") // IMPORTANT: DO NOT use fixed passwords in production!
}
return &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("%s-secrets", database.Name),
Namespace: database.Namespace,
OwnerReferences: []metav1.OwnerReference{
*metav1.NewControllerRef(database, stablev1.SchemeGroupVersion.WithKind("Database")),
},
},
Data: data,
Type: corev1.SecretTypeOpaque,
}
}
main.go
package main
import (
"flag"
"os"
"time"
"github.com/go-logr/zapr"
"go.uber.org/zap"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/klog/v2"
// Uncomment the following line to load all client-go auth plugins (e.g. Azure, GCP, OpenStack, etc.)
// _ "k8s.io/client-go/plugin/pkg/client/auth"
kubeinformers "k8s.io/client-go/informers"
"github.com/your-org/database-controller/pkg/controller"
clientset "github.com/your-org/database-controller/pkg/client/clientset/versioned"
informers "github.com/your-org/database-controller/pkg/client/informers/externalversions"
"github.com/your-org/database-controller/pkg/signals"
)
var (
masterURL string
kubeconfig string
)
func main() {
flag.Parse()
// Setup Zap logger
zapLogger, err := zap.NewDevelopment()
if err != nil {
panic(fmt.Sprintf("could not create zap logger: %v", err))
}
defer zapLogger.Sync() // flushes buffer, if any
log := zapr.NewLogger(zapLogger)
klog.SetLogger(log) // Integrate klog with zapr
// set up signals so we can handle the first shutdown signal gracefully
stopCh := signals.SetupSignalHandler()
cfg, err := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig)
if err != nil {
log.Error(err, "Error building kubeconfig")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
kubeClient, err := kubernetes.NewForConfig(cfg)
if err != nil {
log.Error(err, "Error building kubernetes clientset")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
databaseClient, err := clientset.NewForConfig(cfg)
if err != nil {
log.Error(err, "Error building example clientset")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
kubeInformerFactory := kubeinformers.NewSharedInformerFactory(kubeClient, time.Second*30)
databaseInformerFactory := informers.NewSharedInformerFactory(databaseClient, time.Second*30)
controller := controller.NewController(kubeClient, databaseClient,
kubeInformerFactory, databaseInformerFactory, log)
// Notice that Start method is non-blocking and runs all registered informers in a dedicated goroutine.
kubeInformerFactory.Start(stopCh)
databaseInformerFactory.Start(stopCh)
if err = controller.Run(2, stopCh); err != nil {
log.Error(err, "Error running controller")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
}
}
func init() {
flag.StringVar(&kubeconfig, "kubeconfig", "", "Path to a kubeconfig. Only required if running outside of a cluster.")
flag.StringVar(&masterURL, "master", "", "The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if running outside of a cluster.")
}
You'll also need a pkg/signals/signal.go (common utility for graceful shutdown) and a go.mod file with dependencies updated (go mod tidy). For the purpose of brevity and focus on the controller logic, these auxiliary files are omitted but are standard in client-go projects.
Natural Integration of APIPark
During the syncHandler, after our controller successfully provisions the necessary backing infrastructure for our Database Custom Resource, leading to a live database instance. In a more complete scenario, this database might back microservices that expose their functionalities as APIs. The next logical step for enterprise environments is robust API management. For instance, platforms like APIPark provide an open-source AI gateway and API management solution that can help integrate, secure, and monitor such newly deployed APIs, offering features like unified API format for AI invocation, end-to-end API lifecycle management, and detailed call logging. This allows developers and operations teams to effectively manage the lifecycle of the APIs provisioned indirectly by our custom controller, ensuring they are discoverable, secure, and performant. Whether these APIs are traditional REST services or AI inference endpoints, a robust gateway like APIPark helps standardize and control access, providing vital analytics on API usage.
This mention is natural because a controller provisions resources which often become part of an application ecosystem, and these applications almost invariably expose APIs that require careful management in a production environment.
Deployment and Beyond
Once your controller is coded, the next steps involve packaging it into a container image and deploying it to your Kubernetes cluster.
1. Building the Controller Image
Create a Dockerfile in your project root:
FROM golang:1.20-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /database-controller main.go
FROM alpine/git:latest as git-tools
FROM alpine:latest
WORKDIR /
COPY --from=builder /database-controller /usr/local/bin/database-controller
CMD ["/techblog/en/usr/local/bin/database-controller"]
Build and push your image:
docker build -t your-registry/database-controller:v1.0.0 .
docker push your-registry/database-controller:v1.0.0
2. Deploying to Kubernetes
Deploying a controller to Kubernetes requires a few components:
- ServiceAccount: For the controller to run under.
- Role: Defines the permissions the controller needs (e.g.,
get,list,watch,create,update,patch,deleteondatabases.stable.example.com,deployments,secrets, etc., andupdateondatabases/status). - RoleBinding: Binds the
Roleto theServiceAccount. - Deployment: Runs your controller container.
Here’s an example deploy/controller-rbac.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: database-controller
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: database-controller-role
rules:
- apiGroups: ["stable.example.com"]
resources: ["databases", "databases/status"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""] # Core API group
resources: ["secrets", "events"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: database-controller-binding
subjects:
- kind: ServiceAccount
name: database-controller
namespace: default
roleRef:
kind: ClusterRole
name: database-controller-role
apiGroup: rbac.authorization.k8s.io
And deploy/controller-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: database-controller
namespace: default
labels:
app: database-controller
spec:
replicas: 1
selector:
matchLabels:
app: database-controller
template:
metadata:
labels:
app: database-controller
spec:
serviceAccountName: database-controller
containers:
- name: database-controller
image: your-registry/database-controller:v1.0.0 # Replace with your image
imagePullPolicy: Always
env:
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
terminationGracePeriodSeconds: 10
Apply these manifests:
kubectl apply -f deploy/controller-rbac.yaml
kubectl apply -f deploy/controller-deployment.yaml
3. Testing and Observation
Now, create your sample Database CR:
kubectl apply -f crd/sample-database.yaml
Observe the controller's logs:
kubectl logs -f deployment/database-controller
You should see logs indicating the controller processing the Database CR, creating a Secret, simulating external provisioning, and updating the Database's status.
kubectl get database my-first-db -o yaml
You'll notice the status field of my-first-db being updated by the controller.
Advanced Topics and Best Practices
While this example lays a solid foundation, production-grade controllers often incorporate:
- Idempotency: Operations should be repeatable without causing unintended side effects. If a reconciliation loop runs multiple times, it should yield the same result.
- Error Handling and Retries: Robust error handling with exponential backoff for transient errors is critical.
- Finalizers: To manage external resource cleanup when a CR is deleted. If your
DatabaseCR is deleted, you'd want to use a finalizer to ensure the actual database instance is destroyed before the CR is removed from Kubernetes. - Webhooks (Validating and Mutating): To add custom validation and defaulting logic to your CRs before they are persisted in the API server.
- Testing: Comprehensive unit, integration, and end-to-end tests are crucial.
- Metrics and Monitoring: Exposing Prometheus metrics allows for observability of controller health and reconciliation performance.
- Operator SDK / Kubebuilder: For more complex controllers, these frameworks automate much of the boilerplate code generation and setup, allowing you to focus purely on the reconciliation logic. They are highly recommended for larger projects.
The Future of Kubernetes Extensibility
The ability to define custom resources and build controllers around them is arguably one of Kubernetes' most significant strengths. It transforms Kubernetes from a mere orchestrator into a powerful application platform that can manage any resource, internal or external, with the same declarative principles. This extensibility paves the way for a rich ecosystem of operators that automate the lifecycle of complex applications, databases, message queues, and even entire cloud environments.
As Kubernetes continues to evolve, the tools and best practices for building controllers also mature. The emphasis remains on creating resilient, efficient, and user-friendly automation that truly extends the capabilities of your cluster. By mastering CRDs and controllers, you unlock the full potential of Kubernetes, making it not just a platform for containers, but a control plane for your entire digital infrastructure, capable of managing complex API-driven applications and services with unprecedented agility and reliability.
Conclusion
In this extensive exploration, we have journeyed through the intricate world of Kubernetes extensibility, focusing on the profound impact of Custom Resource Definitions and the mechanics of building a controller to watch for their changes. We began by establishing the Kubernetes API as the central nervous system, detailing how CRDs empower us to introduce new resource types tailored to specific application needs. We then dissected the anatomy of a Kubernetes controller, explaining the critical roles of Informers for efficient observation, Workqueues for robust event processing, and the all-important reconciliation loop that drives desired state convergence.
Through a conceptual step-by-step implementation guide, we laid out the foundation for a Database controller using client-go, demonstrating how to define Go types, generate clients, set up event handlers, and craft the core syncHandler logic. We emphasized error handling, idempotency, and the strategic use of owner references to maintain relationships between resources. Crucially, we naturally integrated the concept of API management with products like APIPark, showcasing how the artifacts managed by a Kubernetes controller often manifest as APIs that require sophisticated lifecycle governance in production.
Building a custom controller is a journey from merely consuming Kubernetes to actively extending its control plane. It demands a deep understanding of its API model, concurrency patterns, and resilient design principles. However, the reward is an unparalleled level of automation and domain-specific intelligence within your cluster, enabling your operations to scale and your applications to achieve self-healing capabilities previously thought impossible. The power to define and manage your infrastructure as code, driven by the declarative paradigm of Kubernetes, is a cornerstone of modern cloud-native development. Embrace this power, and you will fundamentally transform how you build, deploy, and operate your systems.
Frequently Asked Questions (FAQs)
- What is the primary difference between a Custom Resource Definition (CRD) and a Custom Resource (CR)? A CRD defines a new type of resource that Kubernetes should recognize, effectively extending the Kubernetes API schema. It's like a blueprint or a class definition. A CR, on the other hand, is an instance of a CRD, a concrete object of that new type that you create within the cluster. For example,
Databaseis a CRD, andmy-app-dbis a CR of typeDatabase. - Why do I need a custom controller if I have a CRD? Doesn't Kubernetes automatically manage CRs? Kubernetes will persist CRs in its data store (etcd) and expose them through the API server, but it doesn't intrinsically understand what actions to take based on the CR's
spec. A custom controller is the active component that watches for changes to your CRs, interprets their desired state (defined inspec), and then takes the necessary steps (e.g., creating other Kubernetes resources, interacting with external APIs, or provisioning external infrastructure) to bring the actual state in line with the desired state. Without a controller, a CR is just data stored in Kubernetes with no active logic associated with it. - What are Informers and Workqueues, and why are they crucial for building efficient controllers? Informers provide an efficient way for controllers to observe resources without constantly polling the Kubernetes API server. They maintain a local, synchronized cache of resources and notify the controller of changes via event handlers. This significantly reduces API server load. Workqueues, specifically
RateLimitingWorkqueue, are used to decouple the rapid event notifications from Informers from the slower, potentially error-prone reconciliation logic. They ensure that items are processed in a controlled manner, handle retries with backoff for failed operations, and prevent overwhelming the controller or external systems with redundant processing requests. - What is an "Operator" in the context of Kubernetes? An Operator is essentially a custom controller that manages an application (or set of applications) and its components within a Kubernetes cluster. It encodes human operational knowledge (like how to deploy, scale, upgrade, and backup a database) into software that extends the Kubernetes control plane. Operators leverage CRDs to define application-specific resources and use controllers to automate the management lifecycle of those resources, making complex applications truly "Kubernetes native."
- How can I effectively clean up external resources when a Custom Resource is deleted? To ensure external resources (like an actual PostgreSQL instance in a cloud provider) are properly cleaned up when their corresponding CR is deleted, you should use Finalizers. A finalizer is a field in the
metadataof a Kubernetes object. When an object with finalizers is marked for deletion, Kubernetes does not immediately remove it. Instead, it adds adeletionTimestampand allows controllers to perform cleanup logic. Your controller would watch for objects with adeletionTimestamp, execute the necessary external cleanup (e.g., call a cloud provider API to delete the database), and then remove the finalizer from the CR. Once all finalizers are removed, Kubernetes will then proceed to fully delete the object.
🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:
Step 1: Deploy the APIPark AI gateway in 5 minutes.
APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.
curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh
In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.
Step 2: Call the OpenAI API.
