Deploy on GitHub Release with a Kubernetes Operator (Helm + KEDA)

Every release of a Kafka-listener service used to end the same way. Someone cuts a tag on GitHub. CI builds and pushes the image. Then a second person bumps the image tag in the Helm chart and runs the upgrade.

That last step is pure transcription. The tag already exists, the image is already in the registry, and a human is copying one string from one place to another. It is also a queue: the release is done, but it does not ship until someone has time.

This post builds the small piece of software that removes that step: a Kubernetes operator, written in Go with Operator SDK, that notices a new GitHub release and rolls the matching Deployment to the new image. The service is a Kafka consumer scaled by KEDA, which turns out to matter, and the Deployment is installed by Helm, which matters even more.

I built and ran everything below on a local kind cluster before writing it up, and I will say plainly where the test setup differs from production. The most useful findings are not in the happy path. They are what happens when Helm and an operator both think they own the same field.

What we are building

A Deployment opts in with three annotations. The operator reconciles every opted-in Deployment on a timer:

1. Ask GitHub for the latest release of the repo named in the annotation.
2. If it matches the running image tag, do nothing.
3. If it differs, check that the image exists in the registry. A release can be published before CI finishes pushing the image, and rolling to a tag the node cannot pull gives you ImagePullBackOff.
4. Pause the KEDA ScaledObject, patch the container image, and wait for the rollout to finish.
5. Resume the ScaledObject.

There is no custom resource. The unit of configuration is the Deployment itself, which keeps the whole thing to about 200 lines of Go.

Why not Argo CD, Flux, or a CI push

If you already run Argo CD Image Updater or Flux image automation, use that. They do this job and have had years of hardening.

An operator earns its place when you want three specific properties at once. Pull, not push: the cluster reaches out to GitHub, so CI never holds cluster credentials. No new control plane: one small Deployment, no GitOps stack to adopt for a single service family. And custom rollout hooks: pausing a Kafka autoscaler around a rollout is a few lines in your own reconciler and awkward everywhere else.

Scaffold the project

You need Go, Docker, kubectl, kind and the Operator SDK CLI. I used Operator SDK 1.42 and Kubernetes 1.37 on kind.

operator-sdk init --domain example.com --repo example.com/release-operator --plugins go/v4

# a controller for a type we do not own: no CRD, no API package
operator-sdk create api --group apps --version v1 --kind Deployment \
  --resource=false --controller=true \
  --external-api-path=k8s.io/api/apps/v1 --external-api-domain=k8s.io

That scaffolds a DeploymentReconciler with an empty Reconcile. Three things in the scaffold needed fixing for this use case:

• The RBAC marker names the wrong API group. It generates groups=apps.k8s.io. Deployments live in the apps group. Fix the marker, then run make manifests so config/rbac/role.yaml is regenerated.
• The Dockerfile copies a directory that does not exist. It has COPY api/ api/, but --resource=false means there is no api/ package, so docker build fails. Delete that line.
• The Go version drifts. Adding the registry client bumped go.mod to go 1.25, while the Dockerfile still pinned golang:1.24. Bump the builder image.

The reconciler

Configuration lives in annotations on the Deployment:

metadata:
  annotations:
    release.example.com/repo: acme/listener          # GitHub owner/name; opts the Deployment in
    release.example.com/image: nginx                 # image repository, no tag
    release.example.com/container: app              # which container to update
    release.example.com/scaledobject: listener      # KEDA ScaledObject to hold during rollouts

Here is the core of Reconcile. I kept the real code, not a sketch:

func (r *DeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := logf.FromContext(ctx)

	var d appsv1.Deployment
	if err := r.Get(ctx, req.NamespacedName, &d); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}
	repo, imageRepo := d.Annotations[annRepo], d.Annotations[annImage]
	if repo == "" || imageRepo == "" {
		return ctrl.Result{}, nil
	}
	idx := containerIndex(&d, d.Annotations[annContainer])
	if idx < 0 {
		log.Info("container not found, skipping", "container", d.Annotations[annContainer])
		return ctrl.Result{}, nil
	}
	c := d.Spec.Template.Spec.Containers[idx]
	soName := d.Annotations[annScaledObject]

	// Mid-rollout: leave everything alone and look again shortly.
	if !rolloutComplete(&d) {
		return ctrl.Result{RequeueAfter: rolloutPoll}, nil
	}
	// Rollout finished: hand scaling back to KEDA.
	if soName != "" {
		if err := r.setScalerPaused(ctx, d.Namespace, soName, false); err != nil {
			return ctrl.Result{}, err
		}
	}

	tag, err := r.Releases.LatestTag(ctx, repo)
	if err != nil {
		return ctrl.Result{}, err
	}
	tag = strings.TrimPrefix(tag, "v")
	if tag == imageTag(c.Image) {
		return ctrl.Result{RequeueAfter: r.PollInterval}, nil
	}

	// A release exists before CI has finished pushing the image.
	ref := imageRepo + ":" + tag
	ok, err := r.Images.Exists(ctx, ref)
	if err != nil {
		return ctrl.Result{}, err
	}
	if !ok {
		log.Info("release found but image not in registry yet", "image", ref)
		return ctrl.Result{RequeueAfter: time.Minute}, nil
	}

	// Hold the autoscaler before the rollout starts, not after.
	if soName != "" {
		if err := r.setScalerPaused(ctx, d.Namespace, soName, true); err != nil {
			return ctrl.Result{}, err
		}
	}
	log.Info("rolling out", "from", c.Image, "to", ref)
	return ctrl.Result{RequeueAfter: rolloutPoll}, r.setImage(ctx, &d, c.Name, ref)
}

Two design choices are worth naming.

The reconciler holds no state. There is no "rollout in progress" field. Every pass re-derives what to do from the cluster: is the rollout complete, is the release different, is the scaler paused by us. Crash the operator mid-rollout and the next pass finishes the job. This is the main reason to write it as a level-triggered reconciler instead of a webhook handler.

Release tags are mapped to image tags by trimming a leading v. v1.27.2 on GitHub means nginx:1.27.2 in the registry. If your pipeline tags differently, this is the one line to change.

Patch only the image, with server-side apply

The obvious way to change the image is r.Update(ctx, &d) with the whole object, or a merge patch. Both make the operator a co-owner of the entire Deployment spec. With server-side apply you send only the fields you care about and Kubernetes records you as the owner of exactly those:

func (r *DeploymentReconciler) setImage(ctx context.Context, d *appsv1.Deployment, container, image string) error {
	u := &unstructured.Unstructured{Object: map[string]any{
		"apiVersion": "apps/v1",
		"kind":       "Deployment",
		"metadata":   map[string]any{"name": d.Name, "namespace": d.Namespace},
		"spec": map[string]any{"template": map[string]any{"spec": map[string]any{
			"containers": []any{map[string]any{"name": container, "image": image}},
		}}},
	}}
	return r.Patch(ctx, u, client.Apply, client.FieldOwner(fieldOwner), client.ForceOwnership)
}

I used an unstructured object on purpose. A typed appsv1.Deployment serializes zero values (creationTimestamp: null, resources: {}, status: {}), and applying those would claim ownership of fields you never meant to touch.

After a rollout, kubectl get deploy listener --show-managed-fields shows the split. The operator owns the container name and image. Helm still owns everything else, such as the readiness probe and pull policy, and it no longer owns the image.

Pausing KEDA around a rollout

KEDA scales the Deployment by Kafka consumer lag. A rollout replaces pods, which restarts consumers, which makes lag move, which makes KEDA want a different replica count in the middle of the rollout. Pausing the ScaledObject for the duration removes that feedback loop. KEDA supports this with an annotation, and kubectl get scaledobject shows PAUSED=True while it is set.

func (r *DeploymentReconciler) setScalerPaused(ctx context.Context, ns, name string, pause bool) error {
	so := &unstructured.Unstructured{}
	so.SetGroupVersionKind(scaledObjectGVK)
	if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: name}, so); err != nil {
		return client.IgnoreNotFound(err)
	}
	ann := so.GetAnnotations()
	if ann == nil {
		ann = map[string]string{}
	}
	switch {
	case pause && ann[kedaPaused] != "true":
		ann[kedaPaused], ann[annPausedBy] = "true", fieldOwner
	case !pause && ann[annPausedBy] == fieldOwner:
		delete(ann, kedaPaused)
		delete(ann, annPausedBy)
	default:
		return nil
	}
	so.SetAnnotations(ann)
	return r.Update(ctx, so)
}

The paused-by marker matters. The operator only resumes a pause it set itself, so an engineer who paused scaling by hand during an incident does not get surprised when a rollout finishes. Again I use unstructured, which avoids importing KEDA's types, and the operator does nothing at all for Deployments without the annotation.

The two lookups

Both sources sit behind small interfaces so the reconciler can be tested without the network.

type ReleaseSource interface {
	LatestTag(ctx context.Context, repo string) (string, error)
}
type ImageChecker interface {
	Exists(ctx context.Context, ref string) (bool, error)
}

The GitHub side calls GET /repos/{repo}/releases/latest, which skips drafts and prereleases, and sends the last ETag back as If-None-Match. When nothing has changed GitHub answers 304 Not Modified with no body, and conditional requests that return 304 are documented as not counting against the rate limit. Set GITHUB_TOKEN for private repositories and for the higher authenticated limit.

The registry side is a manifest HEAD through go-containerregistry, which handles the token handshake that Docker Hub and GHCR require. A 404 means "not pushed yet" and anything else is an error:

_, err = remote.Head(r, remote.WithContext(ctx), remote.WithAuthFromKeychain(authn.DefaultKeychain))
var terr *transport.Error
if errors.As(err, &terr) && terr.StatusCode == http.StatusNotFound {
	return false, nil
}
return err == nil, err

The part that bites: Helm and the operator on one field

This is why I would not skip the test cluster. The Deployment is installed by Helm, and the chart sets image: nginx:1.27.0 from values.yaml. The operator then changes it to 1.27.1. What does the next helm upgrade do?

I ran it on Helm 3.19 with the operator already having moved the image to 1.27.1, and a chart that still said 1.27.0:

| Action | Image afterwards | |---|---| | helm upgrade with identical values | Reverted to 1.27.0 | | helm upgrade --set kafka.lagThreshold=9 (unrelated change) | 1.27.0 for about 3 seconds, then back to 1.27.1 | | Devops bumps the chart tag to 1.27.2 | 1.27.2, then the operator put it back to 1.27.1 |

Helm 3 builds a three-way merge that includes live state, so it resets the image to whatever the chart says on every upgrade, even for a change that has nothing to do with images. The operator notices within a poll interval and rolls forward again. The end state is right, but every Helm upgrade causes a pointless rollout back and a second one forward. On a Kafka consumer group that is two rebalances you did not need.

The fix is to make the chart stop managing the tag after the first install. Helm's lookup function reads the live Deployment at upgrade time:

{{- /* image.tag only bootstraps the first install; after that the release operator owns the image. */ -}}
{{- $image := printf "%s:%s" .Values.image.repository .Values.image.tag -}}
{{- $live := lookup "apps/v1" "Deployment" .Release.Namespace .Release.Name -}}
{{- if $live -}}{{- $image = (index $live.spec.template.spec.containers 0).image -}}{{- end }}
...
          image: {{ $image | quote }}

With that in place, I repeated the unrelated upgrade. The image stayed at 1.27.1 for the full 15 seconds I watched, and the lagThreshold change applied normally. The trade-offs are real, so decide before you adopt it:

• values.yaml image.tag becomes a bootstrap value. Bumping it on an existing release is silently ignored. That is the point, but document it in the chart.
• lookup returns nothing under helm template and --dry-run, so those render the chart value. And tools that render charts without cluster access, Argo CD among them, never see the live image. If you use one of those, lookup is not available to you.
• I only tested Helm 3.19. Helm 4 can apply with server-side apply, and I have not checked how its field-manager conflicts interact with the operator's ownership.

Test it yourself

My setup: a kind cluster, KEDA 2.21 installed with Helm, a single-node Kafka (apache/kafka, KRaft mode), and a Helm chart that renders the Deployment plus a ScaledObject with a Kafka trigger. The service image was plain nginx, a stand-in for a real consumer, since the point is rollout behavior and not message handling. The honest differences from production:

• GitHub was a fake. A short Python script serves /repos/acme/listener/releases/latest from a file, with an ETag, so I could flip releases on demand. The real-GitHub path is the same HTTP call, but I did not exercise real rate limiting.
• The registry was real. The existence check talked to Docker Hub, so a missing tag really returned a 404.

Results, in the order I ran them:

1. Release equals running image. The fake server's log showed If-None-Match on every poll after the first, and the image did not change.
2. Release v9.9.9, image missing. The operator logged release found but image not in registry yet and left the Deployment alone.
3. Release v1.27.1, image exists, under load. I loaded 300 messages into the topic and let KEDA scale the listener to 5 replicas, then flipped the release. During the rollout the Deployment showed 7 total, 3 updated and 4 available pods, and the ScaledObject carried autoscaling.keda.sh/paused: "true" with paused-by: release-operator. When the rollout finished the annotations were gone.
4. Same flow, operator running in the cluster. I built the image, loaded it into kind and used make deploy, which applies the generated RBAC. The release v1.27.2 rolled out with zero errors in the operator log, which confirms the corrected apps group and the patch and update verbs are enough.

The unit tests cover the two pure helpers, imageTag and rolloutComplete. The imageTag test exists because a naive strings.Split(image, ":") breaks on registries with a port, like localhost:5000/app.

What I would change before production

• A stuck rollout leaves the autoscaler paused. The operator resumes scaling only when the Deployment reports a complete rollout. If the new pods crash-loop, it never does, and KEDA stays paused. Use the Deployment's progressDeadlineSeconds condition to resume the scaler, and decide whether to roll back.
• ECR needs its own credentials in the cluster. authn.DefaultKeychain reads a Docker config, which is enough for public images and a workstation. In-cluster against ECR, add an ECR credential helper keychain with an IAM role for the service account.
• RBAC is cluster-wide. The generated role lets the operator patch Deployments in every namespace. Namespace-scope it or restrict it with an admission policy if that is too broad.
• Polling is a latency choice. A 2 minute interval means up to 2 minutes between tag and rollout. A GitHub webhook would make it near-instant, at the cost of exposing an endpoint, which undoes the pull-only property. Start with polling.
• No rollback and no semver policy. It follows releases/latest. If you want to hold back majors, add a tag pattern annotation and filter on it.

Takeaways

• A small operator is a reasonable way to remove the "copy the tag into Helm" step, and a Deployment with annotations is enough configuration. You do not need a CRD.
• Check that the image exists before you patch. A release exists before its image does.
• Use server-side apply with a named field owner and send only the image field, so the operator and Helm own disjoint fields.
• Helm will fight you on that field. Test helm upgrade against a live operator before you trust the setup, and let the chart read the live image with lookup if you can.
• Pause the autoscaler around a rollout, mark your pause as yours, and make sure something eventually resumes it.