Building Your First Kubernetes Controller with Kubebuilder: A Pod Annotator

If you already know kubectl and Kubernetes YAML but are new to Go and Kubernetes controllers, building a small controller is one of the best ways to understand how Kubernetes operators actually work.

In this tutorial, we’ll build a Pod Annotator controller using Kubebuilder.

Our controller will watch Kubernetes Pods and automatically add an annotation when a Pod matches a specific namespace and label.

For our example, the controller will look for:

• Namespace: tenant
• Label key: prometheus

When both conditions match, the controller adds:

metadata:
  annotations:
    telemetry-compass.com/reconciled: "true"

This project is deliberately small.

Instead of immediately jumping into Custom Resource Definitions (CRDs), we’ll first learn the most important concept behind Kubernetes operators:

The reconciliation loop.

By the end of this tutorial, we’ll have a controller running inside Kubernetes and automatically modifying matching Pods.

What Are We Building?

The desired behavior is simple:

Pod Created / Updated
        |
        v
Controller sees event
        |
        v
Is namespace "tenant"?
        |
       Yes
        |
        v
Does Pod have the "prometheus" label?
        |
       Yes
        |
        v
Does it already have
telemetry-compass.com/reconciled="true"?
        |
       No
        |
        v
Add annotation
        |
        v
Update Pod

For example, suppose this Pod is created:

apiVersion: v1
kind: Pod
metadata:
  name: prometheus-test
  namespace: tenant
  labels:
    prometheus: prometheus-server
spec:
  containers:
    - name: nginx
      image: nginx

Our controller should automatically add:

metadata:
  annotations:
    telemetry-compass.com/reconciled: "true"
  labels:
    prometheus: prometheus-server

Nobody needs to manually run kubectl annotate.

The controller does it for us.

1. What Is a Kubernetes Controller?

Before touching Kubebuilder, it helps to understand what we are actually building.

Think of a Kubernetes controller like a thermostat.

You configure a thermostat with:

Keep the room at 22°C.

The thermostat continuously compares:

Desired temperature: 22°C
Actual temperature:  19°C

There is a difference, so the thermostat turns on the heating.

Eventually:

Desired temperature: 22°C
Actual temperature:  22°C

Now it doesn’t need to do anything.

A Kubernetes controller follows essentially the same pattern:

Observe current state
        |
        v
Compare with desired state
        |
        v
Something wrong?
        |
       Yes
        |
        v
Make a change

This process is called reconciliation.

Our desired state is:

Every Pod in the tenant namespace with the prometheus label should contain the annotation telemetry-compass.com/reconciled: "true".

Our controller’s job is to make that statement true.

2. Prerequisites

Before starting, verify that the required tools are installed.

You should have:

• Go
• Kubebuilder
• kubectl
• Access to a Kubernetes cluster
• GNU Make
• Git
• Docker or another compatible container runtime

Check Go:

go version

Check Kubebuilder:

kubebuilder version

Check kubectl:

kubectl version --client

Check Make:

make --version

And verify that Kubernetes is accessible:

kubectl get nodes

If that works, we’re ready to build the controller.

3. Create the Kubebuilder Project

Create a project directory:

mkdir -p /opt/pod-annotator
cd /opt/pod-annotator

Initialize the Kubebuilder project:

kubebuilder init \
  --domain example.com \
  --repo github.com/machani/pod-annotator

Kubebuilder creates the basic project structure for us.

It will look roughly like:

pod-annotator/
├── cmd/
│   └── main.go
├── config/
│   ├── default/
│   ├── manager/
│   └── rbac/
├── internal/
│   └── controller/
├── Dockerfile
├── Makefile
├── PROJECT
└── go.mod

Don’t worry if this looks like a lot.

We only need to understand a few important pieces initially.

cmd/main.go

This starts the controller manager.

Think of the manager as the process responsible for running our controllers.

internal/controller/

This is where our reconciliation logic lives.

Most of the interesting code we’ll write will be here.

config/

This contains Kubernetes manifests used to deploy the controller.

That includes things such as:

• RBAC
• Deployment configuration
• ServiceAccount
• Kustomize configuration

Makefile

Kubebuilder gives us convenient commands such as:

make run
make test
make manifests
make docker-build
make docker-push
make deploy

We’ll use these throughout the project.

4. Do We Need a CRD?

When people hear Kubebuilder, they often immediately think about Custom Resource Definitions.

For example:

apiVersion: web.example.com/v1
kind: Website

But our controller doesn’t need a custom resource yet.

We’re watching an existing Kubernetes resource:

apiVersion: v1
kind: Pod

So we only need a controller.

We don’t need to invent a new Kubernetes API.

This makes the Pod Annotator a useful first project because we can concentrate on understanding reconciliation before adding CRDs, API schemas, status fields, finalizers, and other operator concepts.

5. Scaffold the Pod Controller

Generate the controller:

kubebuilder create api \
  --group core \
  --version v1 \
  --kind Pod \
  --controller=true \
  --resource=false

The most important part here is:

--resource=false

We’re telling Kubebuilder:

Create controller scaffolding, but don’t create a new custom resource.

Why?

Because Pod already exists in Kubernetes.

6. Meet the Reconcile Function

The heart of our controller is a function similar to:

func (r *PodReconciler) Reconcile(
    ctx context.Context,
    req ctrl.Request,
) (ctrl.Result, error)

This is the function controller-runtime calls when a relevant Pod event occurs.

The request identifies the resource that needs reconciliation.

Conceptually, it contains information like:

Namespace: tenant
Name:      prometheus-test

One important thing to understand is that the event itself isn’t our source of truth.

Instead, the controller uses the request to retrieve the current version of the object from the Kubernetes API.

That’s a fundamental controller pattern:

Event says:
"Something changed!"
        |
        v
Controller asks Kubernetes:
"What does the object look like right now?"

7. Define the Desired Rules

Let’s make our controller’s configuration clear.

We want:

Namespace:
tenant
Required label key:
prometheus
Annotation:
telemetry-compass.com/reconciled=true

In Go, we can represent these as constants:

const (
    targetNamespace = "tenant"
    targetLabelKey = "prometheus"
    annotationKey   = "telemetry-compass.com/reconciled"
    annotationValue = "true"
)

Using constants makes our reconciliation logic easier to understand.

Instead of repeatedly writing:

"telemetry-compass.com/reconciled"

we can write:

annotationKey

8. Fetch the Pod

When reconciliation begins, we first need the current Pod.

Create a Pod object:

pod := &corev1.Pod{}

Then retrieve it from Kubernetes:

if err := r.Get(ctx, req.NamespacedName, pod); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
}

This is roughly equivalent to asking Kubernetes:

Give me the Pod identified by:
namespace + name

But why do we use:

client.IgnoreNotFound(err)

instead of simply returning every error?

Because Kubernetes resources can disappear at any moment.

Imagine this sequence:

Pod created
     |
     v
Event queued
     |
     v
Pod deleted
     |
     v
Controller processes event
     |
     v
GET Pod
     |
     v
Not Found

Nothing is actually wrong.

The Pod simply disappeared before we processed its event.

Controllers must expect this kind of behavior.

9. Filter by Namespace

Now we check whether the Pod belongs to our target namespace.

if pod.Namespace != targetNamespace {
    return ctrl.Result{}, nil
}

If it isn’t in:

tenant

we don’t care about it.

We simply stop reconciliation.

Notice the style we’re using:

Wrong namespace?
    |
   Yes
    |
    v
Return

Instead of nesting everything inside multiple if blocks, controller code commonly uses early returns.

That keeps the reconciliation flow easy to follow.

10. Filter by Label

Next, check whether the Pod contains the prometheus label.

if _, ok := pod.Labels[targetLabelKey]; !ok {
    return ctrl.Result{}, nil
}

We’re effectively asking:

Does the Pod contain a label with the key:
prometheus

For example, this matches:

labels:
  prometheus: prometheus-server

And so does:

labels:
  prometheus: monitoring

The important part for this controller is the presence of the prometheus label key.

If the label isn’t present, we ignore the Pod.

Our logic now looks like:

Get Pod
   |
   v
Correct namespace?
   |
   +--- No ---> Stop
   |
  Yes
   |
   v
Has "prometheus" label?
   |
   +--- No ---> Stop
   |
  Yes
   |
   v
Continue

11. Check Whether the Annotation Already Exists

Now comes one of the most important concepts in controller development:

Idempotency.

Before modifying the Pod, check whether it is already correct:

if pod.Annotations != nil &&
    pod.Annotations[annotationKey] == annotationValue {
    return ctrl.Result{}, nil
}

Suppose the Pod already contains:

annotations:
  telemetry-compass.com/reconciled: "true"

There is nothing to do.

So we return.

Why is this important?

Imagine that we updated the Pod every time reconciliation ran.

An update can cause another event:

Controller updates Pod
        |
        v
Pod changed
        |
        v
New event
        |
        v
Reconcile
        |
        v
Controller updates Pod again
        |
        v
Another event

That’s exactly the kind of unnecessary loop we want to avoid.

A good reconciliation loop asks:

Is the current state already what I want?

If yes:

Do nothing.

That is idempotent behavior.

12. Add the Annotation

Now we know:

• The namespace matches.
• The prometheus label exists.
• The annotation isn’t already correct.

So we need to add it.

There’s one Go detail we need to handle first.

The annotation map may be nil.

So initialize it when necessary:

if pod.Annotations == nil {
    pod.Annotations = make(map[string]string)
}

Now add our annotation:

pod.Annotations[annotationKey] = annotationValue

Our in-memory Pod now contains:

annotations:
  telemetry-compass.com/reconciled: "true"

But changing the Go object isn’t enough.

We need to send the change back to Kubernetes:

if err := r.Update(ctx, pod); err != nil {
    return ctrl.Result{}, err
}

And finally:

return ctrl.Result{}, nil

Reconciliation is complete.

13. The Complete Reconciliation Logic

Putting the important pieces together, the core logic looks like this:

func (r *PodReconciler) Reconcile(
    ctx context.Context,
    req ctrl.Request,
) (ctrl.Result, error) {
    pod := &corev1.Pod{}
    if err := r.Get(ctx, req.NamespacedName, pod); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if pod.Namespace != targetNamespace {
        return ctrl.Result{}, nil
    }
    if _, ok := pod.Labels[targetLabelKey]; !ok {
        return ctrl.Result{}, nil
    }
    if pod.Annotations != nil &&
        pod.Annotations[annotationKey] == annotationValue {
        return ctrl.Result{}, nil
    }
    if pod.Annotations == nil {
        pod.Annotations = make(map[string]string)
    }
    pod.Annotations[annotationKey] = annotationValue
    if err := r.Update(ctx, pod); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

The entire algorithm can be summarized as:

Fetch Pod
    |
    v
Wrong namespace? -------------> Stop
    |
    v
No "prometheus" label? -------> Stop
    |
    v
Annotation already correct? --> Stop
    |
    v
Initialize annotation map
    |
    v
Add telemetry-compass.com/reconciled=true
    |
    v
Update Pod

This is the core of our controller.

14. Tell Kubebuilder to Watch Pods

Writing reconciliation logic isn’t enough.

We also need to tell controller-runtime:

Which Kubernetes resource should trigger this controller?

That’s done in SetupWithManager.

For our controller, it looks similar to:

func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&corev1.Pod{}).
        Named("pod").
        Complete(r)
}

The critical line is:

For(&corev1.Pod{})

That tells controller-runtime to watch Pods.

Conceptually:

Kubernetes API
      |
      | Pod events
      v
controller-runtime
      |
      v
Work Queue
      |
      v
Reconcile()

The controller doesn’t need to continuously run:

kubectl get pods

controller-runtime handles the watch mechanism and reconciliation queue for us.

15. Add RBAC Permissions

There is another important piece.

Watching Pods doesn’t automatically mean our controller is allowed to access them.

Kubernetes authorization still applies.

Our controller needs permissions to:

• Get Pods
• List Pods
• Watch Pods
• Update Pods
• Patch Pods

Kubebuilder lets us describe those permissions using markers in Go code.

Add an RBAC marker similar to:

// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;update;patch

Notice:

groups=""

Why is the group empty?

Because Pods belong to the Kubernetes core API group.

Now generate the RBAC manifests:

make manifests

Kubebuilder’s tooling reads the markers and generates the appropriate Kubernetes RBAC YAML.

This gives us an important distinction.

This:

For(&corev1.Pod{})

means:

Watch Pods.

While this:

// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;update;patch

means:

Kubernetes should authorize this controller to perform these operations on Pods.

You generally need both.

16. Format and Validate the Code

Go has standardized formatting.

Run:

make fmt

Then:

make vet

We can also run the project’s tests:

make test

A Kubebuilder test workflow may involve:

controller-gen
     |
     v
go fmt
     |
     v
go vet
     |
     v
envtest
     |
     v
go test

Kubebuilder commonly uses envtest for controller tests.

envtest provides lightweight Kubernetes control-plane components such as an API server and etcd for tests.

That lets our tests interact with a Kubernetes API without requiring an entire production cluster.

17. Run the Controller Locally

Now comes the fun part.

We can run the controller directly from our development machine while it talks to our Kubernetes cluster.

First check the current context:

kubectl config current-context

Then run:

make run

The controller starts locally and uses your kubeconfig to communicate with Kubernetes.

Keep that terminal running.

Open another terminal for testing.

18. Create a Test Pod

Make sure our target namespace exists:

kubectl create namespace tenant

If it already exists, Kubernetes will simply tell you so.

Now create:

test-pod.yaml

with:

apiVersion: v1
kind: Pod
metadata:
  name: annotation-test
  namespace: tenant
  labels:
    prometheus: prometheus-server
spec:
  containers:
    - name: nginx
      image: nginx

Apply it:

kubectl apply -f test-pod.yaml

Now inspect the Pod:

kubectl get pod annotation-test \
  -n tenant \
  -o yaml

Look at:

metadata:
  annotations:

We should see:

annotations:
  telemetry-compass.com/reconciled: "true"

Congratulations.

We just built a working Kubernetes controller.

19. Test the Negative Cases

Testing only the happy path isn’t enough.

Our controller has several conditions, so we should verify them.

Test 1: Correct Namespace + Prometheus Label

Namespace:
tenant
Label:
prometheus=prometheus-server

Expected:

telemetry-compass.com/reconciled: "true"

Test 2: Correct Namespace + No Prometheus Label

For example:

labels:
  app: nginx

Expected:

No annotation.

Test 3: Wrong Namespace + Prometheus Label

For example:

metadata:
  namespace: default
  labels:
    prometheus: prometheus-server

Expected:

No annotation.

Test 4: Annotation Already Exists

Suppose the Pod already contains:

annotations:
  telemetry-compass.com/reconciled: "true"

Expected:

No unnecessary update.

These tests verify the actual policy implemented by our controller.

20. Local Controller vs In-Cluster Controller

At this point our architecture looks like:

Laptop / Development Machine
make run
   |
   | kubeconfig
   |
   v
Kubernetes API Server
   |
   v
Pods

This is excellent during development.

But eventually we want:

Kubernetes Cluster
      |
      +--- Pod Annotator Controller
      |
      +--- Application Pods

To do that, we need to package the controller into a container image.

21. Build the Controller Image

Build the image:

make docker-build IMG=<your-registry>/pod-annotator:v0.1.0

For example:

make docker-build IMG=ghcr.io/example/pod-annotator:v0.1.0

The Kubebuilder project already includes a Dockerfile.

The build process compiles our Go controller and packages it into a container image.

Then push it:

make docker-push IMG=ghcr.io/example/pod-annotator:v0.1.0

Replace the example registry and repository with your own.

Using explicit version tags such as:

v0.1.0
v0.2.0
v1.0.0

also makes deployments easier to understand than relying entirely on:

latest

22. Deploy the Controller to Kubernetes

Once the image is available in the registry, deploy it:

make deploy IMG=ghcr.io/example/pod-annotator:v0.1.0

Kubebuilder uses the manifests under:

config/

to install the required Kubernetes resources.

These include the controller manager and its RBAC configuration.

Check the deployment:

kubectl get pods -n pod-annotator-system

Eventually we should see something similar to:

NAME                                               READY   STATUS    RESTARTS   AGE
pod-annotator-controller-manager-xxxxxxxxxx-xxxxx  1/1     Running   0          2m

Now our controller is running inside Kubernetes.

23. Check the Controller Logs

We can inspect its logs with:

kubectl logs \
  -n pod-annotator-system \
  deployment/pod-annotator-controller-manager

Logs become extremely useful when debugging controllers.

When something isn’t working, some of the first things worth checking are:

kubectl get pods -n pod-annotator-system

Then:

kubectl logs \
  -n pod-annotator-system \
  deployment/pod-annotator-controller-manager

And:

kubectl describe pod \
  -n pod-annotator-system \
  <controller-pod-name>

24. A Common Deployment Problem: go: No Such File or Directory

During development, you might encounter something like:

make: go: No such file or directory

followed by a controller-gen error.

This can be confusing because you might think:

I’m deploying a container. Why does my machine need Go?

The reason is that:

make deploy

does more than simply execute:

kubectl apply

The Makefile may first run generation steps such as:

controller-gen

And controller-gen may need the Go toolchain to inspect and load your Go packages.

So the machine where you run Kubebuilder’s Makefile should have Go correctly installed and available in:

$PATH

Check:

which go

and:

go version

If Go is installed but those commands fail, inspect:

echo $PATH

This is an important distinction between:

Running the compiled controller

and:

Running the Kubebuilder development/build tooling.

The final controller container doesn’t need your development Go installation.

But your development environment does.

25. The Complete Architecture

Once deployed, our system looks like this:

                  Kubernetes API Server
                           |
                           | Pod events
                           v
                 controller-runtime watch
                           |
                           v
                     Work Queue
                           |
                           v
                  Reconcile(request)
                           |
                           v
                      Fetch Pod
                           |
                           v
              Check namespace + label
                           |
                           v
               Check current annotation
                           |
                           v
                    Update Pod
                           |
                           v
                  Kubernetes API

There are several important ideas hidden inside this simple diagram.

The controller is:

Event driven

It reacts to changes rather than repeatedly running kubectl.

Declarative

We describe the state we want.

Idempotent

If the object is already correct, we don’t modify it again.

Eventually consistent

The system continuously works toward the desired state.

26. Controller vs Operator

We’ve built a Kubernetes controller.

So is it also an operator?

The terms are closely related, but they aren’t exactly identical.

A controller implements a reconciliation loop.

An operator typically uses controllers to automate operational knowledge for an application or system.

Operators frequently introduce their own Custom Resource Definitions.

For example, instead of hardcoding this:

targetNamespace = "tenant"

we could eventually create something like:

apiVersion: automation.example.com/v1alpha1
kind: PodAnnotationPolicy
metadata:
  name: prometheus-policy
spec:
  namespace: tenant
  labelSelector:
    labelKey: prometheus
  annotations:
    telemetry-compass.com/reconciled: "true"

Now users could configure the controller through Kubernetes YAML instead of modifying Go code.

That’s where Kubebuilder becomes even more powerful.

27. What We Learned

Even though our Pod Annotator is small, we’ve already covered many of the concepts used by real Kubernetes operators.

We learned how to:

• Initialize a Kubebuilder project
• Scaffold a controller
• Watch a built-in Kubernetes resource
• Understand the reconciliation loop
• Fetch objects from the Kubernetes API
• Filter Pods by namespace
• Filter Pods by labels
• Modify annotations
• Write idempotent reconciliation logic
• Understand controller-runtime watches
• Define RBAC using Kubebuilder markers
• Generate manifests
• Run controller tests
• Run a controller locally
• Build a controller container
• Push the image to a registry
• Deploy the controller into Kubernetes
• Inspect the running controller

But the most important lesson is the mental model.

We started with the familiar Kubernetes world:

kubectl
+
YAML

Now we can start thinking like a controller:

Observe
   |
   v
Compare
   |
   v
Reconcile
   |
   v
Repeat

That mental model is far more important than memorizing Kubebuilder commands.

28. Where Should We Go Next?

Our current controller has one obvious limitation.

These values are hardcoded:

Namespace
Label
Annotation

Changing the policy requires changing Go code and rebuilding the controller.

Kubernetes gives us a much better way.

We can define our own Kubernetes API.

For example:

apiVersion: web.example.com/v1alpha1
kind: Website
metadata:
  name: my-site
spec:
  image: nginx:latest
  replicas: 2

Then our controller could automatically create:

Website
   |
   +--- Deployment
   |       |
   |       +--- Pods
   |
   +--- Service

That project would introduce several major Kubebuilder concepts:

• Custom Resource Definitions
• API types
• Spec
• Status
• Generated DeepCopy code
• CRD validation
• Owner references
• Status conditions
• Finalizers
• Watching owned resources
• More advanced reconciliation

Our Pod Annotator therefore isn’t the end of the journey.

It’s the foundation.

Conclusion

Kubebuilder can look intimidating when you first generate a project.

Suddenly there are:

Go files
Makefiles
RBAC markers
Kustomize manifests
controller-runtime
Managers
Reconcilers
Clients
Schemes

But underneath all those abstractions, the core idea is remarkably simple:

Observe current state
        |
        v
Compare with desired state
        |
        v
Make them match

Our Pod Annotator demonstrates exactly that.

A Pod in the tenant namespace with a prometheus label appears.

The controller observes it.

It discovers that:

telemetry-compass.com/reconciled: "true"

is missing.

It updates the Pod.

Then, when reconciliation happens again, it sees that the Pod is already correct and does nothing.

That simple loop is the foundation on which sophisticated Kubernetes operators are built.

And once that idea clicks, Kubebuilder starts making a lot more sense.

Github Repo link: https://github.com/machani/pod-annotator

Cheers!