By apipark — 11 Apr 2026

Kubernetes Controller: How to Watch CRD Changes

controller to watch for changes to crd

Kubernetes has firmly established itself as the de facto operating system for the cloud-native era, fundamentally transforming how applications are built, deployed, and managed. At its core, Kubernetes operates on a declarative model, where users define the desired state of their applications and the system continuously works to achieve and maintain that state. This powerful paradigm is driven by a sophisticated architecture built around controllers – specialized control loops that watch for changes in the cluster's state and act upon them. While Kubernetes offers a rich set of built-in resources like Pods, Deployments, and Services, the true power of its extensibility comes from Custom Resource Definitions (CRDs). CRDs allow users to define their own resource types, extending the Kubernetes API to manage domain-specific objects as first-class citizens.

However, defining a CRD is only half the battle. To bring these custom resources to life, one needs a Kubernetes controller that can observe changes to instances of these CRDs – known as Custom Resources (CRs) – and enact the necessary logic. This article will embark on an extensive journey, meticulously exploring the mechanisms, best practices, and intricate details involved in building a Kubernetes controller that effectively watches for changes in CRDs. We will delve into the underlying API server interactions, the indispensable client-go library, and higher-level abstractions, providing a comprehensive guide for developers aiming to harness the full power of Kubernetes extensibility. Furthermore, we will illustrate how these custom controllers become critical orchestrators for advanced workloads, including those involving specialized AI infrastructure, where concepts like AI Gateway, LLM Gateway, and Model Context Protocol play pivotal roles in managing complex machine learning deployments.

Understanding the Core of Kubernetes: The Controller Pattern

Before we dive into the specifics of watching CRD changes, it is crucial to solidify our understanding of what a Kubernetes controller is and its fundamental operating principles. A Kubernetes controller is essentially a reconciliation loop – a continuous process that monitors the cluster's actual state and works to bring it closer to the desired state, as specified by users through Kubernetes API objects.

The Control Loop Paradigm

The control loop is a foundational concept in control theory, applied ingeniously within Kubernetes. Imagine a thermostat in your home: you set a desired temperature (the desired state). The thermostat constantly measures the current room temperature (the actual state). If there's a discrepancy, the thermostat turns the heater or air conditioner on or off (the actions) until the desired temperature is reached. This continuous feedback mechanism is precisely what Kubernetes controllers emulate.

In Kubernetes, the "desired state" is declared in API objects (like a Deployment YAML file specifying 3 replicas). The "actual state" is what is currently running in the cluster (e.g., how many Pods are actually up). A controller's job is to observe the desired state, compare it with the actual state, identify any discrepancies, and then take corrective actions to bridge the gap. This process is often referred to as the "reconciliation loop."

Key Characteristics of Kubernetes Controllers:

Declarative Nature: Controllers operate on declarative specifications. Users declare what they want, not how to achieve it. The controller figures out the how. This abstracts away operational complexities and makes the system more robust and self-healing.
Event-Driven: Controllers don't constantly poll for changes. Instead, they are typically notified by the Kubernetes API server about relevant events (resource creation, update, deletion). This makes them efficient and reactive.
Idempotency: A controller's reconciliation logic must be idempotent. This means that applying the same reconciliation multiple times, or receiving the same event multiple times, should produce the same outcome as applying it once. This is crucial for resilience, as controllers might re-process events due to network issues or restarts.
Convergence: The ultimate goal of a controller is to ensure that the actual state converges to the desired state. If the desired state changes, the controller works to converge to the new desired state.
Autonomous Operation: Once deployed, controllers operate autonomously, continuously monitoring and adjusting the cluster state without human intervention. This is what enables Kubernetes' self-healing capabilities.

Components of a Typical Controller

While the internal implementations can vary, most Kubernetes controllers, especially those built using the client-go library or controller-runtime, share a common architectural pattern comprising several key components:

Informer: This is perhaps the most critical component for watching resources. An informer maintains a local, in-memory cache of Kubernetes API objects. It uses the Kubernetes API server's WATCH mechanism to receive real-time updates and ensures its cache is always consistent with the API server. For efficient querying, the informer's cache provides an Indexer (for fetching objects by key) and a Lister (for fetching objects by label selectors).
Workqueue (RateLimitingQueue): The workqueue acts as a buffer and a decoupling mechanism. When the informer detects a change (add, update, delete) to a watched resource, it doesn't immediately trigger the reconciliation logic. Instead, it places the "key" (e.g., namespace/name) of the affected object into a workqueue. This allows the controller to process events asynchronously and to rate-limit or retry failed reconciliation attempts using mechanisms like exponential backoff.
Reconcile Function (or Sync Handler): This is the heart of the controller's business logic. A worker goroutine continuously pulls keys from the workqueue. For each key, the reconcile function is invoked. Its primary responsibilities are:
- Fetching the latest state of the object corresponding to the key from the informer's cache.
- Comparing this desired state with the current actual state of related resources (e.g., Pods, Deployments, external services).
- Taking necessary actions to reconcile the differences (e.g., creating, updating, or deleting dependent resources).
- Updating the status subresource of the custom resource to reflect the current state of the controller's operations.
Event Handlers: These are callback functions registered with the informer (AddFunc, UpdateFunc, DeleteFunc). When an event occurs for a watched resource, the corresponding handler is invoked. Typically, these handlers simply extract the object's key and add it to the workqueue.

These components work in concert to provide a robust and scalable mechanism for managing resources within a Kubernetes cluster. The beauty of this pattern lies in its ability to extend Kubernetes' capabilities without modifying the core system, allowing for the creation of an infinite array of custom operators.

Deep Dive into Custom Resource Definitions (CRDs)

To extend Kubernetes beyond its built-in resource types, the concept of Custom Resource Definitions (CRDs) was introduced. CRDs allow cluster administrators to define new, unique API object kinds, complete with their own schemas, validation rules, and versioning strategies. These custom types behave exactly like native Kubernetes resources, enabling the Kubernetes API to serve as a unified control plane for virtually any application or infrastructure component.

Why CRDs? Extending the Kubernetes API

Before CRDs, the primary method for extending Kubernetes was through API aggregation, which involved running a separate API server and registering it with the main Kubernetes API. While powerful, API aggregation is complex to implement and maintain. CRDs democratized API extension, making it significantly easier for developers to introduce new resource types.

The motivations for using CRDs are manifold:

Domain-Specific Abstractions: Representing application-specific concepts directly within the Kubernetes API. For instance, instead of managing a database as a collection of Pods, Services, and PersistentVolumes, one can define a Database CRD and manage database instances as single objects.
Operational Simplicity: Encapsulating complex operational knowledge into a single resource. A database controller, for example, would watch a Database CR and automatically provision storage, set up replication, manage backups, and handle upgrades.
Unified Management: Leveraging Kubernetes tools (kubectl, RBAC, watch mechanisms) for custom resources just as one would for built-in resources. This provides a consistent management experience across the entire infrastructure.
Developer Empowerment: Enabling application developers to define their own abstractions that directly map to their application architecture, fostering a more self-service model.

Anatomy of a CRD

A CRD itself is a Kubernetes API object. When you create a CRD, you're telling the Kubernetes API server about a new type of resource it should accept. Here's a breakdown of its key fields:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.stable.example.com # Must be in the format <plural>.<group>
spec:
  group: stable.example.com       # The API group for this CRD
  versions:
    - name: v1                    # The version of this CRD
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:                 # Definition of the custom resource's spec
              type: object
              properties:
                image:
                  type: string
                  description: "The Docker image to deploy"
                replicas:
                  type: integer
                  minimum: 1
                  default: 1
                port:
                  type: integer
                  minimum: 80
                  maximum: 65535
            status:               # Definition of the custom resource's status
              type: object
              properties:
                availableReplicas:
                  type: integer
  scope: Namespaced               # Can be Namespaced or Cluster
  names:
    plural: myapps                # Plural name used in API paths (e.g., /apis/stable.example.com/v1/myapps)
    singular: myapp               # Singular name (e.g., kubectl get myapp)
    kind: MyApp                   # The Kind of the custom resource (e.g., kind: MyApp)
    shortNames:                   # Optional short names
      - ma

Let's dissect the important fields:

metadata.name: This must be in the format <plural>.<group>. For our MyApp example, it's myapps.stable.example.com. This forms a unique identifier for your CRD.
spec.group: Defines the API group for your custom resources. This helps organize your APIs and avoids naming conflicts. Here, it's stable.example.com.
spec.versions: An array allowing you to define multiple versions of your custom resource (e.g., v1alpha1, v1).
- name: The name of the version (e.g., v1).
- served: A boolean indicating if this version is served via the API.
- storage: A boolean indicating if this version is used for storing the resource in etcd. There must be exactly one storage version.
- schema.openAPIV3Schema: This is where you define the structural schema for your custom resource using OpenAPI v3. This schema is critical for:
  - Validation: The API server will validate custom resources against this schema when they are created or updated, preventing malformed objects.
  - Client Generation: Tools can use this schema to generate client libraries for your custom resources.
  - spec: Defines the user-configurable fields for your custom resource.
  - status: Defines the fields that your controller will populate to report the current state of the resource. Kubernetes recommends making status a subresource to allow controllers to update it independently without triggering validation on the entire object.
spec.scope: Determines if custom resources of this type are Namespaced (like Pods and Deployments) or Cluster scoped (like Nodes and PersistentVolumes).
spec.names: Defines the various names for your custom resource:
- plural: Used in URLs (e.g., /apis/stable.example.com/v1/myapps).
- singular: The singular form.
- kind: The Kind field you'll use in your custom resource YAML files (e.g., kind: MyApp).
- shortNames: Optional, convenient abbreviations for kubectl commands (e.g., kubectl get ma).

Custom Resources (CRs): Instances of CRDs

Once a CRD is created and registered with the API server, you can then create instances of that custom type, which are called Custom Resources (CRs). A CR is simply a YAML or JSON document that adheres to the schema defined in its corresponding CRD.

Example of a Custom Resource myapp-sample.yaml:

apiVersion: stable.example.com/v1
kind: MyApp
metadata:
  name: myapp-frontend
  namespace: default
spec:
  image: "nginx:latest"
  replicas: 3
  port: 80

When you apply this YAML (kubectl apply -f myapp-sample.yaml), the Kubernetes API server will:

Receive the request.
Identify that stable.example.com/v1, kind: MyApp corresponds to the myapps.stable.example.com CRD.
Validate the myapp-frontend object against the openAPIV3Schema defined in the CRD.
If valid, store the object in etcd, just like any other Kubernetes resource.
Emit events (ADDED, MODIFIED) that controllers watching this CRD can pick up.

This seamless integration allows CRs to be managed, labeled, annotated, and secured using the same mechanisms as native Kubernetes objects, creating a truly extensible and unified control plane.

The Heart of the Matter: Watching CRD Changes

Now that we understand controllers and CRDs, let's turn our attention to the core mechanism: how a Kubernetes controller watches for changes to CRD instances (Custom Resources). This is where the Kubernetes API server's WATCH mechanism and the client-go library's informer pattern become paramount.

Kubernetes API Server's Role: The `WATCH` API

The Kubernetes API server is the central brain of the cluster, exposing a RESTful interface through which all communication flows. A crucial feature of this API is the WATCH mechanism. Instead of clients constantly polling the API server for changes (which would be inefficient and create unnecessary load), clients can establish a long-lived HTTP connection to the API server and "watch" for events pertaining to specific resource types.

When a resource is created, updated, or deleted, the API server sends a corresponding event (ADDED, MODIFIED, DELETED) down this watch connection to all interested clients. This real-time notification system is the foundation upon which all Kubernetes controllers operate. For CRDs, the API server handles watches on Custom Resources just as it does for built-in resources.

Client-Go Library for Controllers: Informers

While one could implement a watch mechanism by directly interacting with the Kubernetes API server's WATCH endpoint (e.g., by making long-polling HTTP requests), this approach is fraught with challenges for robust, production-grade controllers:

Connection Management: Handling dropped connections, retries, and re-establishing watches.
State Management: Keeping track of the cluster's state locally without constantly querying the API server, which would be inefficient and lead to throttling.
Error Handling: Differentiating transient errors from persistent ones.
Resource Versioning: Ensuring that no events are missed during connection interruptions.

This is where the official Kubernetes client-go library comes in. Client-go provides a set of powerful abstractions that simplify controller development, with informers being the cornerstone for watching resources.

Shared Informers: The Efficient Watcher

At the heart of client-go's watch mechanism is the SharedInformer pattern. A SharedInformer is designed for efficiency and collaboration:

Single Watch Stream: For a given resource type (e.g., MyApp CRs), only one WATCH request is sent to the Kubernetes API server by the Reflector component within the informer. This reduces the load on the API server.
Shared Cache: The informer maintains a local, in-memory cache of all objects of that type. This cache is automatically kept up-to-date by processing events from the watch stream. Any component within the controller (or even multiple controllers within the same process) can share this cache. This avoids redundant API calls and improves performance significantly.
Event Handlers: The SharedInformer allows you to register ResourceEventHandler callbacks (AddFunc, UpdateFunc, DeleteFunc) which are invoked when changes are detected in the cached objects.

Let's break down the internal flow and components of a SharedInformer:

Reflector: This is the lowest-level component of an informer. The Reflector is responsible for:
- Performing an initial LIST operation to populate the informer's cache with all existing objects of the watched type.
- Establishing and maintaining the WATCH connection to the Kubernetes API server.
- Handling connection drops and automatically re-establishing the watch, ensuring no events are missed by providing the ResourceVersion of the last seen object during reconnections.
- Receiving raw WATCH events (ADDED, MODIFIED, DELETED) from the API server.
- Pushing these events to an internal delta FIFO queue.
Delta FIFO Queue: This queue buffers the raw events received from the Reflector. It helps in ensuring processing order and can manage deduplication or merging of events for the same object if they arrive rapidly.
Controller (Informer's Internal Controller): This internal controller pulls events from the Delta FIFO queue and processes them. Its primary job is to:
- Update the Indexer (the in-memory cache) with the latest state of the object.
- Invoke the registered ResourceEventHandler callbacks (AddFunc, UpdateFunc, DeleteFunc) that you provide in your custom controller.
Indexer and Lister:
- Indexer: The actual in-memory store. It allows efficient retrieval of objects by their namespace/name key and supports custom indexing (e.g., by labels).
- Lister: A convenient interface built on top of the Indexer that provides read-only access to the cache. It allows you to "list" objects (all or by label selector) and "get" a specific object, without ever hitting the API server once the cache is populated. This is crucial for the reconciliation loop, as it makes fetching the desired state extremely fast.

Building a Controller with Client-Go Informers (Conceptual)

The typical flow for setting up a controller to watch CRDs using client-go involves these steps:

Generate Clientset: You'll use k8s.io/code-generator to generate a clientset, informers, and listers for your custom resources from your CRD definitions. This automates the creation of strongly typed Go interfaces for your CRDs.
Create a SharedInformerFactory: This factory allows you to create informers for various resource types (built-in and custom) and ensures they share the underlying watch connections and caches where possible.```go // Example: Using the generated clientset cfg, err := rest.InClusterConfig() // or clientcmd.BuildConfigFromFlags // ... error handling kubeClient, err := kubernetes.NewForConfig(cfg) // ... error handling myAppClient, err := myappclientset.NewForConfig(cfg) // myappclientset is generated for your CRD // ... error handling// Create a SharedInformerFactory for your custom resource // Resync period controls how often the cache is re-listed from API server // even if no watch events occur. Useful for detecting missed events or self-healing. myAppInformerFactory := myappinformers.NewSharedInformerFactory(myAppClient, time.Second*30)// Get the informer for your MyApp custom resource myAppInformer := myAppInformerFactory.Stable().V1().MyApps() ```
Register Event Handlers: Attach ResourceEventHandler functions to your informer. These functions will be called when an object is added, updated, or deleted. Inside these handlers, you'll typically push the object's key into your controller's workqueue.go myAppInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { // Convert the obj to your MyApp type, extract its namespace/name, // and add it to your controller's workqueue. key, err := cache.MetaNamespaceKeyFunc(obj) // ... error handling c.workqueue.Add(key) }, UpdateFunc: func(oldObj, newObj interface{}) { // Same logic as AddFunc, potentially with checks if relevant fields changed key, err := cache.MetaNamespaceKeyFunc(newObj) // ... error handling c.workqueue.Add(key) }, DeleteFunc: func(obj interface{}) { // Same logic, handle deletion (e.g., ensure associated resources are cleaned up) key, err := cache.MetaNamespaceKeyFunc(obj) // ... error handling c.workqueue.Add(key) }, })
Start the Informers and Factory:```go stopCh := make(chan struct{}) defer close(stopCh)// Start all informers in the factory. This kicks off the Reflector goroutines. myAppInformerFactory.Start(stopCh)// Wait for all caches to be synced. This ensures the informer's cache // is populated before your controller starts processing events. if !cache.WaitForCacheSync(stopCh, myAppInformer.Informer().HasSynced) { // ... error handling return }// Now your controller's worker goroutines can start processing items from the workqueue. ```

Controller-Runtime: A Higher-Level Abstraction

While client-go provides the fundamental building blocks, directly using it for complex controllers can still be verbose. The controller-runtime library (used by Operator SDK and Kubebuilder) offers a higher-level abstraction that significantly simplifies controller development. It wraps client-go informers, workqueues, and leader election into a more opinionated and developer-friendly framework.

Key features of controller-runtime:

Manager: Orchestrates multiple controllers, webhooks, and client connections.
Controller interface: Provides a simple Reconcile(context.Context, reconcile.Request) (reconcile.Result, error) method where you put your core logic.
Watch API: Simplified way to specify which resources a controller should watch (For(&MyApp{}).Watches(&appsv1.Deployment{})).
CachingClient: A client that transparently uses informers for reads and directly hits the API server for writes, balancing performance and consistency.

For most modern custom controller development, controller-runtime is the recommended path as it handles much of the boilerplate, allowing developers to focus on the core reconciliation logic.

Direct Watch (Less Common for Controllers)

It's worth briefly mentioning that client-go also provides a more direct client.Watch() function. This function allows you to get a raw watch.Interface which streams watch.Event objects directly from the API server.

// Example of direct watch (simplified)
watcher, err := myAppClient.StableV1().MyApps("default").Watch(context.TODO(), metav1.ListOptions{})
// ... error handling

for event := range watcher.ResultChan() {
    // Process event.Type (ADDED, MODIFIED, DELETED) and event.Object
    // This requires manual cache management, error handling, etc.
}

While this can be useful for simple, one-off scripts or debugging, it is not recommended for production-grade Kubernetes controllers. The informer pattern, with its caching, error handling, and shared watch capabilities, is vastly superior for building resilient and efficient controllers. Direct watch would lead to:

Increased load on the API server due to repeated LIST operations or lack of efficient watch reconnection logic.
Slower reconciliation cycles without a local cache.
More complex error handling and state management for the controller developer.

Controller Implementation Patterns and Best Practices

Building a robust Kubernetes controller involves more than just watching CRD changes. Several patterns and best practices are essential for ensuring stability, efficiency, and correct behavior.

Workqueue Management and Error Handling

The workqueue is central to decoupling event handling from reconciliation. Proper management is critical:

Rate-limiting: Controllers often use a RateLimitingQueue to prevent overwhelming the API server or dependent services. This queue implements exponential backoff for retries, meaning that if a reconciliation fails, the item is requeued and retried after an increasingly longer delay. This prevents tight loops on problematic resources.
Max Retries: Define a maximum number of retries. If an item consistently fails to reconcile after several attempts, it might indicate a fundamental issue (e.g., malformed CR, persistent external service error). In such cases, the item should be dropped from the workqueue (or marked as failed) to prevent it from blocking other items. An EventRecorder can be used to log events on the CR indicating the persistent failure.
Shutdown: Gracefully shutting down the workqueue and its worker goroutines is crucial during controller termination.

Idempotency of the Reconcile Function

As mentioned, the reconciliation logic must be idempotent. This means that if the reconcile function is called multiple times with the same desired state, it should have the same effect as being called once.

Avoid creating resources if they already exist: Before creating a Deployment, check if a Deployment with the expected name and owner reference already exists.
Update existing resources if they differ: If a resource exists, but its spec (e.g., image, replicas) does not match the desired state from the CR, update it.
Delete resources if they are no longer desired: If a CR is deleted, or a dependent resource is no longer specified by the CR, ensure it's removed.

Status Subresource: Reporting Controller State

Every Kubernetes resource, including CRs, has a status field. This field is read-only for users and is meant to be populated by controllers to report the current state of the resource and the actions taken by the controller.

status as a subresource: For CRDs, it's highly recommended to enable the status subresource. This allows controllers to update the status field without needing to update the entire object, which would trigger validation checks on the spec and could lead to race conditions if users are simultaneously modifying the spec.
Meaningful Status Fields: The status should provide clear, actionable information about the resource's health, progress, and any encountered errors. Common fields include:
- conditions: An array of conditions (e.g., Ready, Available, Progressing) with a status (True, False, Unknown), reason, and message.
- observedGeneration: The metadata.generation of the CR that the controller last processed. This helps users understand if the controller has acted on their latest changes.
- replicas, availableReplicas: For resources managing Pods.

Finalizers: Ensuring Proper Cleanup

Kubernetes provides finalizers as a mechanism to ensure that resources are not deleted until specific cleanup operations are performed, typically by a controller.

How they work: When a resource has finalizers, deleting it via the API server doesn't immediately remove it from etcd. Instead, the resource's metadata.deletionTimestamp is set, and the object remains in the cluster until all finalizers are removed from its metadata.finalizers list.
Controller's role: When a controller observes a resource with deletionTimestamp set, it knows the resource is marked for deletion. It then performs its cleanup logic (e.g., deleting external cloud resources, unregistering webhooks). Once cleanup is complete, the controller removes its finalizer from the resource. The API server then garbage collects the resource.
Use cases: Crucial for managing external resources (cloud databases, S3 buckets, DNS records) that Kubernetes doesn't automatically garbage collect. Without finalizers, a CR might be deleted, leaving orphaned external resources.

Owner References: Automatic Garbage Collection

Owner references establish a parent-child relationship between Kubernetes objects. This mechanism is primarily used for automatic garbage collection.

How they work: When a controller creates a dependent resource (e.g., a Deployment for a MyApp CR), it sets an OwnerReference on the dependent resource, pointing back to the MyApp CR.
Garbage Collector's role: The Kubernetes garbage collector, a built-in controller, watches for resources that have owner references. If an owner resource is deleted, the garbage collector automatically deletes all its dependent resources (children), unless explicitly configured otherwise (blockOwnerDeletion: false).
Benefit: Simplifies cleanup. When your MyApp CR is deleted, the associated Deployment, Service, and ConfigMaps are automatically removed without explicit controller action.

Leader Election: For High-Availability Controllers

For controllers deployed in a high-availability setup (multiple replicas), leader election is essential to prevent multiple controller instances from simultaneously trying to reconcile the same resource, which can lead to race conditions, conflicting actions, or wasted resources.

Mechanism: Kubernetes uses a Lease object (or ConfigMap/Endpoints in older versions) to coordinate leader election. Only the leader controller instance performs reconciliation.
Client-Go/Controller-Runtime integration: Both client-go and controller-runtime provide built-in mechanisms for leader election, simplifying its implementation. When a leader fails, another replica automatically takes over.

Testing Controllers

Comprehensive testing is vital for controller reliability:

Unit Tests: Test individual functions and reconciliation logic in isolation, mocking Kubernetes API interactions.
Integration Tests: Test the controller against a real (but isolated) Kubernetes API server (e.g., envtest from controller-runtime). This allows testing informer setup, workqueue interactions, and client-go calls.
End-to-End (E2E) Tests: Deploy the controller and CRDs to a test cluster and verify its behavior from a user's perspective, observing cluster state changes.

Practical Use Cases and Advanced Scenarios

The ability to define custom resources and build controllers to manage them unlocks an enormous potential for extending Kubernetes into virtually any domain. From automating infrastructure to orchestrating complex AI/ML workloads, CRDs and controllers serve as the foundational building blocks.

Automating Infrastructure Provisioning

One of the most common and impactful use cases for custom controllers is the automation of infrastructure provisioning. Instead of manually creating cloud resources through cloud provider APIs or Infrastructure-as-Code tools, developers can define CRDs for these resources.

Example: Database as a Service:
- Define a Database CRD (e.g., PostgreSQLInstance).
- A controller watches PostgreSQLInstance CRs.
- When a PostgreSQLInstance CR is created, the controller interacts with an external cloud API (e.g., AWS RDS, Azure Database) to provision a new database instance.
- It updates the CR's status with connection details (endpoint, credentials via Secret).
- When the CR is deleted, the controller deprovisions the external database.
- This effectively brings external infrastructure under Kubernetes' declarative management.

Managing Application Deployments with Custom Logic

While Kubernetes Deployments are robust, sometimes applications require highly specialized deployment strategies that go beyond what a standard Deployment offers.

Example: Blue/Green or Canary Deployments:
- Define a CanaryDeployment CRD.
- A controller watches CanaryDeployment CRs.
- Instead of just creating a ReplicaSet, the controller orchestrates a phased rollout:
  - Deploys a small percentage of new Pods.
  - Waits for health checks and metrics (e.g., error rates, latency).
  - Gradually shifts traffic (by updating Service selectors or ingress rules).
  - Rolls back if issues are detected.
- This complex logic is encapsulated within the controller, making advanced deployments as simple as applying a CanaryDeployment CR.

Orchestrating AI/ML Workloads: The Frontier of CRD Controllers

The realm of Artificial Intelligence and Machine Learning presents some of the most intricate and resource-intensive workloads in modern computing. Managing the entire lifecycle of an ML model – from data preprocessing and training to deployment and serving – often requires specialized hardware (GPUs, TPUs), complex data pipelines, and intelligent traffic routing. Kubernetes, extended with CRDs and custom controllers, is becoming the preferred platform for orchestrating these sophisticated AI/ML operations.

Defining AI/ML Resources with CRDs:
- TrainingJob CRD: Specifies dataset locations, model architectures, hyper-parameters, GPU requirements. A controller watches this to spin up GPU-enabled Pods, run training scripts, and store model artifacts.
- InferenceService CRD: Declares a trained model to be served, desired replica count, traffic splitting rules for A/B testing, and pre/post-processing logic. A controller orchestrates the deployment of model servers (e.g., KServe, Seldon Core), configures load balancers, and sets up scaling policies.
- FeatureStore CRD: Defines access patterns and storage for features used in ML models.
The Role of AI Gateway and LLM Gateway in Managed AI Ecosystems: As the number and complexity of AI models grow, especially with the proliferation of Large Language Models (LLMs), directly integrating each model into an application becomes a significant burden. Different models have varying APIs, authentication schemes, rate limits, and data formats. This complexity gives rise to the need for a unified AI Gateway or, specifically for LLMs, an LLM Gateway.
- AI Gateway/LLM Gateway CRDs: Imagine defining AIGatewayConfig or LLMGatewayPolicy CRDs. A controller could watch these CRDs to dynamically configure an AI Gateway or LLM Gateway instance. For example, a AIGatewayConfig CR might specify routing rules, authentication mechanisms (e.g., API keys, OAuth), and rate limits for various AI services exposed through the gateway. The controller would then apply these configurations to the actual gateway proxy.
- Benefits: These gateways abstract away the underlying AI model complexities from application developers. They provide:
  - Unified API Endpoint: A single endpoint for all AI services.
  - Centralized Authentication and Authorization: Enforcing security policies across all models.
  - Rate Limiting and Quota Management: Preventing abuse and ensuring fair usage.
  - Cost Tracking: Monitoring API calls and associated costs per user or application.
  - Observability: Centralized logging, metrics, and tracing for AI invocations.
  - A/B Testing and Traffic Management: Routing requests to different model versions or providers.
Model Context Protocol: Standardizing Interaction: Further enhancing the utility of AI Gateways is the concept of a Model Context Protocol. Different AI models, especially LLMs, might expect varied input structures (e.g., roles like "system", "user", "assistant" for chat models; specific JSON schemas for vision models) and produce different output formats. A Model Context Protocol aims to standardize this interaction by providing a common abstraction layer.
- ModelContextMapping CRD: A controller could watch a ModelContextMapping CRD, where each CR defines how a generic input format should be transformed into a specific model's expected input, and how the model's output should be transformed back into a generic output format. This CR could specify data transformations, prompt templating, and response parsing rules.
- Gateway Integration: The AI Gateway or LLM Gateway would then leverage these ModelContextMapping configurations, applied by the controller, to intelligently translate requests and responses on the fly. This ensures that changes in an underlying AI model's API do not break dependent applications, significantly reducing maintenance overhead.
APIPark: An Open Source AI Gateway & API Management PlatformFor enterprises grappling with the increasing complexity of integrating and managing a diverse portfolio of AI models, an advanced AI Gateway is not just a luxury, but a necessity. This is precisely the problem that APIPark solves. APIPark is an open-source AI gateway and API developer portal that is specifically designed to streamline the management, integration, and deployment of both AI and traditional REST services.Imagine having your custom Kubernetes controllers orchestrating your InferenceService CRs, which then need to expose these services to a wider array of applications. APIPark steps in as the unified front-end for these services. It facilitates the quick integration of over 100 AI models, offering a unified API format for AI invocation. This means your applications interact with a standardized API provided by APIPark, abstracting away the specifics of individual models – a tangible implementation of a sophisticated Model Context Protocol. A controller watching an InferenceService CR could, upon successful model deployment, automatically register the new model endpoint with APIPark via its API, making it instantly discoverable and manageable.Furthermore, APIPark allows for prompt encapsulation into REST APIs, enabling users to combine AI models with custom prompts to create new, domain-specific APIs (e.g., sentiment analysis, translation). This aligns perfectly with the goal of custom controllers to define higher-level abstractions. APIPark's comprehensive features, including end-to-end API lifecycle management, team-based sharing, independent tenant configurations, and robust security (API access approval), complement the capabilities of Kubernetes custom controllers. While controllers manage the deployment and operational state of AI resources within Kubernetes, APIPark handles the exposure, governance, and consumption of these AI services to external applications and users, making it an indispensable component in a modern AI infrastructure stack. Its impressive performance, rivaling Nginx with over 20,000 TPS, and detailed logging and data analysis capabilities further solidify its position as a critical platform for any organization serious about AI. The fact that it can be deployed in just 5 minutes with a single command (curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh) makes it an accessible and powerful tool for developers and enterprises alike, bridging the gap between Kubernetes orchestration and real-world AI service consumption.

The Lifecycle of a CRD Change and Controller Reaction

To consolidate our understanding, let's walk through the complete lifecycle of a custom resource change and how a controller built with client-go informers reacts to it. This chain of events showcases the intricate dance between the user, the Kubernetes API server, and the custom controller.

User Initiates Change: A user (or another automated system) creates, modifies, or deletes an instance of a Custom Resource (CR) using kubectl apply, kubectl create, or kubectl delete. bash kubectl apply -f myapp-frontend.yaml This command sends a HTTP POST/PUT/DELETE request to the Kubernetes API server.
API Server Receives and Processes Request:
- The API server receives the request.
- It performs authentication (who is making the request?), authorization (is the user allowed to perform this action on this resource?), and admission control (does the resource conform to policies, schema validation via CRD openAPIV3Schema?).
- If all checks pass, the API server persists the new or updated CR object into its underlying data store (etcd).
API Server Emits WATCH Event:
- Crucially, after successfully persisting the change, the API server sends a WATCH event (of type ADDED, MODIFIED, or DELETED) to all clients that have an active watch connection open for that specific CRD type.
Controller's Reflector Receives Event:
- Inside our custom controller, the Reflector component of the SharedInformer (which maintains the watch connection) receives this WATCH event.
- The Reflector extracts the object and pushes it into its internal Delta FIFO Queue.
Informer's Internal Controller Processes Event:
- The informer's internal controller (distinct from your custom controller's reconciliation logic) continuously pulls items from the Delta FIFO Queue.
- For each item, it updates the Indexer (the local in-memory cache) with the latest state of the object. This ensures the cache is always eventually consistent with the API server.
Informer Invokes Event Handlers:
- After updating the cache, the informer's internal controller invokes the ResourceEventHandler functions that your custom controller has registered (AddFunc, UpdateFunc, DeleteFunc).
- These handlers receive the (old and/or new) object corresponding to the event.
Event Handler Enqueues Key:
- Inside your AddFunc, UpdateFunc, or DeleteFunc, the typical action is to extract the object's unique key (usually namespace/name) and add it to your controller's workqueue (e.g., RateLimitingQueue).
- This is a critical decoupling step, ensuring that the event handling (which needs to be fast) doesn't block the reconciliation logic (which can be long-running).
Controller Worker Dequeues Key:
- One of your controller's worker goroutines continuously pulls keys from the workqueue.
- When a key is pulled, it marks it as "in progress" and prepares to call the reconciliation logic.
Reconcile Function Executes:
- The controller's core Reconcile function is invoked with the retrieved key.
- Fetch Desired State: The first action is usually to fetch the latest version of the custom resource corresponding to the key from the informer's Lister (the read-only cache). This is an extremely fast operation as it doesn't hit the API server.
- Compare States: The reconcile function then compares the spec of the fetched CR (the desired state) with the current actual state of the cluster or external systems (e.g., existing Deployments, Services, cloud resources).
- Perform Actions: Based on the comparison, the controller performs the necessary actions:
  - Creates new dependent resources (Pods, Deployments, external databases).
  - Updates existing dependent resources (e.g., changes an image in a Deployment, updates a Service port).
  - Deletes dependent resources if the CR is deleted or modified to no longer require them.
  - Interacts with external APIs (cloud providers, specialized AI Gateway like APIPark, or LLM providers).
- Update Status: Finally, the controller updates the status subresource of the Custom Resource to reflect the outcome of its reconciliation, providing feedback to the user about its current state and operations.
Workqueue Management (Completion/Retry):
- If the reconciliation was successful, the item is removed from the workqueue.
- If an error occurred, the item is requeued (often with exponential backoff) for a retry, allowing for transient issues to resolve.

This complete cycle, often executed within milliseconds for simple changes or seconds for complex operations, demonstrates the power and responsiveness of the Kubernetes control plane. It's a continuous, self-healing loop that keeps the cluster state aligned with the user's declared intentions.

Table: Comparison of Kubernetes Resource Interaction Methods

When dealing with Kubernetes resources, especially in the context of automation and custom controllers, there are various ways to interact with the API server. Understanding their differences is crucial for choosing the right tool for the job.

Feature / Method	`kubectl get --watch`	`client.Watch()` (Direct Client-Go)	`SharedInformer` (Client-Go)	`controller-runtime` (Manager/Controller)
Purpose	Ad-hoc observation by human	Programmatic raw event stream	Foundation for robust controllers, caching	High-level framework for production controllers
API Interaction	Direct API server watch	Direct API server watch	Single `LIST` then single `WATCH` per resource type	Wraps `SharedInformer`, uses caching client
Caching	None (streams events only)	None (streams events only)	Yes, in-memory cache (`Indexer`, `Lister`)	Yes, built-in CachingClient for reads
Local State Management	None	Manual, complex	Automatic, built-in for objects being watched	Automatic, managed by Manager
Event Handling	Prints events to console	Manual iteration over `ResultChan`	`AddFunc`, `UpdateFunc`, `DeleteFunc` callbacks	`Reconcile` function triggered by events
Error Handling/Retries	None (user re-runs command)	Manual, complex (re-establish watch, track RV)	Automatic watch reconnection, `RateLimitingQueue` for reconciliation logic	Built-in workqueue, exponential backoff, leader election
Scalability (API Server)	High load if many users watch	Can create high load if many clients use directly	Very efficient (single watch per resource type shared)	Very efficient (leverages `SharedInformer`)
Concurrent Processing	N/A	Manual, complex	Handled by controller's workqueue and workers	Handled by Manager, concurrency settings
Ease of Use	Very high	Low for production scenarios	Medium (requires some boilerplate)	High (abstracts away much boilerplate)
Use Case	Debugging, quick observation	Custom event processing in simple scripts	Building custom controllers, operators	Building robust operators and controllers, webhooks
Production Readiness	No	No (for general controllers)	Yes (with correct controller logic)	Yes (recommended)

This table clearly highlights why SharedInformer and controller-runtime are the preferred choices for building production-grade Kubernetes controllers. They provide the necessary abstractions, caching, and resilience features that are missing from simpler interaction methods, allowing developers to focus on the core business logic of their controllers.

Security Considerations for Custom Controllers

Security is paramount in any system, and Kubernetes controllers, with their elevated privileges and ability to manage critical resources, require careful attention to security best practices. A compromised controller can have severe implications for the entire cluster.

Role-Based Access Control (RBAC)

RBAC is the primary mechanism for controlling who can do what in a Kubernetes cluster. Custom controllers run as Pods and are typically associated with a ServiceAccount. This ServiceAccount needs appropriate permissions to interact with the Kubernetes API server.

Least Privilege Principle: Controllers should only be granted the minimum necessary permissions to perform their function. Do not grant cluster-admin unless absolutely unavoidable.
ServiceAccount: Create a dedicated ServiceAccount for your controller Pods.
ClusterRole / Role: Define ClusterRoles (for cluster-scoped resources like CRDs) or Roles (for namespace-scoped resources like custom resources, Pods, Deployments) that specify the verbs (get, list, watch, create, update, delete, patch) allowed on specific resources (e.g., myapps.stable.example.com, pods, deployments).
ClusterRoleBinding / RoleBinding: Bind the ServiceAccount to the defined Role or ClusterRole.

Example RBAC for a MyApp Controller:

# myapp-controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-controller
  namespace: myapp-system
---
# myapp-controller-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: myapp-controller-role
rules:
- apiGroups: ["stable.example.com"] # For our custom MyApp resource
  resources: ["myapps", "myapps/status", "myapps/finalizers"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"] # For Deployments
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""] # For Pods, Services, ConfigMaps, Secrets
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""] # For Pods to get logs (optional)
  resources: ["pods/log"]
  verbs: ["get"]
- apiGroups: ["apiextensions.k8s.io"] # For CRD itself if the controller manages CRDs
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list", "watch"]
---
# myapp-controller-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: myapp-controller-binding
subjects:
- kind: ServiceAccount
  name: myapp-controller
  namespace: myapp-system
roleRef:
  kind: ClusterRole
  name: myapp-controller-role
  apiGroup: rbac.authorization.k8s.io

Admission Controllers: Advanced Validation and Mutation

While CRD schema validation provides basic structural checks, ValidatingWebhookConfiguration and MutatingWebhookConfiguration offer more powerful and dynamic control over custom resource objects. These are external webhooks that the API server calls before persisting an object to etcd.

ValidatingWebhookConfiguration:
- Allows you to define complex, arbitrary validation logic that cannot be expressed purely through OpenAPI schema.
- Example: Ensuring a specific field's value is within a dynamic range, cross-referencing values with other resources, or enforcing business logic constraints.
- If the webhook returns an error, the object creation/update is rejected.
MutatingWebhookConfiguration:
- Allows you to modify (mutate) an object before it is persisted.
- Example: Automatically injecting default values, adding labels/annotations, or setting owner references.
- Useful for enforcing conventions or simplifying user input.

Both types of webhooks are essentially HTTP servers that implement specific Kubernetes API contracts and are deployed within the cluster, often managed by the same controller framework (like controller-runtime).

Secrets Management

Controllers often need to access sensitive information, such as API keys for external services, database credentials, or image pull secrets. This information should never be hardcoded into the controller's code or YAML manifests.

Kubernetes Secrets: Store sensitive data in Kubernetes Secret objects.
Volume Mounts / Environment Variables: Controllers should access these secrets by mounting them as files into the Pod's filesystem or injecting them as environment variables (though file mounts are generally more secure as they avoid exposing secrets in /proc/self/environ).
Cloud Provider Secret Management: For even higher security, integrate with cloud provider secret managers (e.g., AWS Secrets Manager, Azure Key Vault, Google Secret Manager) using tools like External Secrets Operator.

Secure Communication

Controllers should communicate securely with the Kubernetes API server and any external services.

TLS: Use TLS for all communication. Client-go automatically handles TLS for API server communication when using rest.InClusterConfig().
Authentication: Use ServiceAccount tokens for authentication with the API server. For external services, use API keys or OAuth tokens, securely managed as described above.

Troubleshooting and Debugging Controllers

Developing custom controllers can be challenging due to their distributed and asynchronous nature. Effective troubleshooting and debugging strategies are essential.

Logging

High-quality logging is the first line of defense in diagnosing controller issues.

Structured Logging: Use structured logging (e.g., JSON format) to make logs machine-readable and easily searchable in logging aggregation systems (ELK stack, Splunk, Loki).
Contextual Information: Include relevant context in logs, such as the namespace/name of the CR being reconciled, the current phase of reconciliation, and any error messages.
Log Levels: Use appropriate log levels (debug, info, warn, error) to control verbosity. debug for detailed tracing, info for normal operation, error for critical failures.
Tracing: Integrate with distributed tracing systems (e.g., Jaeger, Zipkin) for complex inter-service communication, especially if the controller interacts with multiple external systems or other microservices.

Events

Kubernetes Events are lightweight, time-stamped records attached to objects, indicating "what has happened" to that object. Controllers should use an EventRecorder to generate events.

User Feedback: Events provide crucial feedback to users about the controller's actions or encountered problems. Users can check events using kubectl describe <resource-type>/<name>.
Types of Events: Use Normal for successful operations (e.g., "DeploymentCreated", "StatusUpdated") and Warning for issues (e.g., "ImagePullFailed", "ReconciliationError").
Example Event: A MyApp controller might emit an event like: Type Reason Age From Message ---- ------ ---- ---- ------- Normal DeploymentCreated 5m myapp-controller Created Deployment "myapp-frontend" for MyApp "myapp-frontend" Warning FailedToSync 2m myapp-controller Failed to create Service: Service "myapp-frontend" already exists

Metrics

Exposing Prometheus-compatible metrics from your controller is invaluable for monitoring its health, performance, and operational efficiency.

Controller-Runtime Metrics: controller-runtime automatically exposes useful metrics like:
- controller_runtime_reconcile_total: Total number of reconciliations.
- controller_runtime_reconcile_errors_total: Number of failed reconciliations.
- controller_runtime_reconcile_duration_seconds: Histogram of reconciliation durations.
- workqueue_adds_total, workqueue_depth, workqueue_retries_total: Metrics about the workqueue's state.
Custom Metrics: Add custom metrics to track domain-specific operations (e.g., "external_api_calls_total", "database_provision_time_seconds").
Alerting: Configure Prometheus alerts based on these metrics (e.g., high error rate, consistently long reconciliation times, workqueue depth exceeding a threshold).

`kubectl describe`

The kubectl describe command is a powerful tool for inspecting the state of any Kubernetes resource, including your Custom Resources.

Comprehensive View: It provides a comprehensive view of the resource, including its spec, status, metadata (labels, annotations, owner references, finalizers), and crucially, its Events.
Debugging Status: The status field, populated by your controller, should be the first place to look for information about what your controller is doing or why it's stuck.
Related Objects: You can often infer related objects (like Pods or Deployments) from the controller's logic and then kubectl describe those as well.

Future Trends and Ecosystem

The Kubernetes ecosystem around CRDs and custom controllers is vibrant and continuously evolving, pushing the boundaries of what Kubernetes can manage.

Operator Framework

The Operator Framework is a collection of tools and resources designed to make building, deploying, and managing Kubernetes native applications (Operators) easier. Operators are essentially custom controllers that manage specific applications (e.g., a database operator, a message queue operator). The framework includes:

Operator SDK / Kubebuilder: Tools to scaffold, develop, and test Operators. Both use controller-runtime as their underlying library.
Operator Lifecycle Manager (OLM): A tool to install, update, and manage the lifecycle of Operators on a cluster. It provides a "Kubernetes App Store" experience.
OperatorHub.io: A registry of community and commercially supported Operators.

Crossplane: Managing External Resources

Crossplane is an open-source Kubernetes add-on that enables you to manage and provision infrastructure from your cloud provider (or any external system) using kubectl. It does this by extending Kubernetes with CRDs for external resources (like S3 buckets, PostgreSQL databases, VPCs) and controllers that reconcile these CRs with the actual external cloud APIs. Crossplane effectively turns Kubernetes into a universal control plane for all your infrastructure, both inside and outside the cluster.

KubeVela: Application Delivery with OAM

KubeVela is a modern application delivery platform built on Kubernetes. It uses the Open Application Model (OAM) to define and deliver applications, abstracting away the underlying infrastructure complexities. KubeVela leverages CRDs to represent application components, traits (e.g., autoscaling, ingress), and workflows, and its controllers orchestrate the deployment of these applications across various environments and infrastructures.

The Increasing Reliance on CRDs for AI/ML and Edge Computing

The trend is clear: CRDs and custom controllers are becoming the go-to mechanism for extending Kubernetes into highly specialized and emerging domains.

AI/ML Orchestration: As seen with concepts like AI Gateway, LLM Gateway, and Model Context Protocol, controllers are instrumental in managing the entire ML pipeline, from resource provisioning for training to deploying and serving complex inference models. The ability to define model versions, experiment definitions, and data pipelines as CRs allows data scientists and ML engineers to interact with their infrastructure in a Kubernetes-native way.
Edge Computing: In edge environments, where resources are constrained and connectivity can be intermittent, CRDs and controllers allow for defining edge-specific application deployments, device management, and data synchronization rules. A controller at the edge can watch for local CR changes and reconcile them with local devices or a central control plane.
Serverless Frameworks: Many serverless platforms built on Kubernetes (like Knative) use CRDs to define serverless functions, event sources, and auto-scaling policies.

The continuous evolution of these extensions underscores the power and flexibility that Kubernetes offers through its API extensibility. As the ecosystem matures, the role of platforms like APIPark becomes even more critical. While custom controllers manage the internal operational details of AI services within Kubernetes, APIPark provides the essential layer for robust external exposure, comprehensive management, and standardized consumption of these services, ensuring that the innovations delivered by Kubernetes controllers are readily available and governable for the wider enterprise.

APIPark is a high-performance AI gateway that allows you to securely access the most comprehensive LLM APIs globally on the APIPark platform, including OpenAI, Anthropic, Mistral, Llama2, Google Gemini, and more.Try APIPark now! 👇👇👇

Install APIPark – it’s free

Conclusion

The journey into building Kubernetes controllers that watch CRD changes reveals the profound elegance and power of Kubernetes' extensibility model. We began by understanding the fundamental reconciliation loop that defines a controller's purpose, a continuous pursuit of convergence between desired and actual states. We then delved into Custom Resource Definitions (CRDs), the architectural keystone that empowers us to extend the Kubernetes API with domain-specific objects, transforming Kubernetes into a truly universal control plane.

The core of our exploration focused on the intricate mechanics of watching CRD changes, highlighting the Kubernetes API server's WATCH mechanism and the indispensable role of the client-go library's SharedInformer pattern. This pattern, with its efficient caching, robust error handling, and sophisticated event processing, provides the bedrock for building resilient and scalable custom controllers. We further acknowledged controller-runtime as the modern, higher-level abstraction that streamlines this development, allowing engineers to focus more on business logic and less on boilerplate.

Beyond the technical implementation, we explored critical best practices – idempotent reconciliation, strategic use of status subresources, the crucial role of finalizers for cleanup, and the power of owner references for automatic garbage collection. We also addressed the vital aspects of security through RBAC and admission controllers, and the art of troubleshooting with structured logging, events, and metrics.

Crucially, we illuminated how these custom controllers are not just theoretical constructs but pragmatic solutions for orchestrating some of today's most complex workloads. The management of AI/ML infrastructure, in particular, showcases the immense value of CRDs and controllers in defining and managing elements like training jobs, inference services, and the crucial layers of an AI Gateway, LLM Gateway, and Model Context Protocol. In this context, platforms like APIPark emerge as essential companions, providing the enterprise-grade API management and AI gateway capabilities that seamlessly consume and govern the specialized services orchestrated by Kubernetes controllers.

In essence, understanding how to watch CRD changes is not merely a technical skill; it's an understanding of how to unleash the full, unbridled potential of Kubernetes. It's about empowering developers and operators to mould Kubernetes to their precise needs, creating self-managing systems that are resilient, scalable, and inherently aligned with the declarative principles of the cloud-native world. The journey continues as the ecosystem evolves, promising even more innovative ways to extend and leverage this powerful platform.

Frequently Asked Questions (FAQs)

What is the primary difference between a Kubernetes Controller and an Operator? A Kubernetes Controller is a generic term for a reconciliation loop that watches Kubernetes resources and takes actions. An Operator is a specific type of controller that manages a single application or service (often a complex stateful application like a database) using CRDs to represent application-specific configurations and lifecycle events. All Operators are controllers, but not all controllers are Operators. Operators often encapsulate significant domain-specific operational knowledge.
Why should I use SharedInformer instead of directly calling client.Watch() in my controller? SharedInformer is highly recommended for production controllers because it provides a robust, efficient, and scalable way to watch resources. It maintains a local, consistent cache, performs a single WATCH request to the API server which is then shared among all listeners, handles watch connection drops and retries automatically, and provides efficient read access through Lister. Directly calling client.Watch() lacks these critical features, making it prone to errors, inefficient, and difficult to manage in a production environment.
How does a controller ensure it doesn't miss any events if the API server connection drops? The Reflector component within a SharedInformer is designed to handle this. When it establishes a WATCH connection, it provides a ResourceVersion parameter to the API server. If the connection drops and is re-established, the Reflector provides the ResourceVersion of the last object it successfully processed. The API server then sends all events that occurred since that ResourceVersion, ensuring no events are missed. This mechanism, combined with periodic LIST and cache resyncs, guarantees eventual consistency.
What is the purpose of the status subresource for a CRD, and why is it important? The status subresource allows a controller to update only the status field of a Custom Resource without having to send the entire object to the API server. This is crucial for several reasons: it avoids triggering validation checks on the spec (which is managed by the user), prevents potential race conditions if the user is simultaneously modifying the spec, and clearly separates the user's desired state (spec) from the controller's observed state and progress (status). It provides transparent feedback to users about the controller's operations.
How does an AI Gateway like APIPark relate to Kubernetes controllers managing AI/ML CRDs? Kubernetes controllers managing AI/ML CRDs (e.g., InferenceService CRs) are responsible for the internal orchestration of AI model deployments within the Kubernetes cluster – provisioning resources, deploying model servers, managing scaling, etc. An AI Gateway like APIPark, on the other hand, focuses on the external exposure, management, and consumption of these AI services. It acts as a unified entry point, providing standardized APIs, centralized authentication, rate limiting, and observability for applications wanting to invoke these models. A controller could, for example, register a newly deployed inference service's internal endpoint with APIPark, making it discoverable and consumable through the gateway, thereby bridging the internal Kubernetes orchestration with the external application ecosystem.

🚀You can securely and efficiently call the OpenAI API on APIPark in just two steps:

Step 1: Deploy the APIPark AI gateway in 5 minutes.

APIPark is developed based on Golang, offering strong product performance and low development and maintenance costs. You can deploy APIPark with a single command line.

curl -sSO https://download.apipark.com/install/quick-start.sh; bash quick-start.sh

In my experience, you can see the successful deployment interface within 5 to 10 minutes. Then, you can log in to APIPark using your account.

Step 2: Call the OpenAI API.