Certwatcher only detects cert changes on leader

[hrak]

What broke? What's expected?

On a controller generated with kubebuilder 4.5.2, we are using secure metrics and a deployment with 2 replicas, and noticed that only the leader notices & acts on cert changes, any non-leaders will not reload the cert.

In main.go the metrics certwatcher is added to the manager as a runnable using mgr.Add. It seems like this runnable is acting like a LeaderElectionRunnable and thus only operates on the leader.

logs from the leader:

2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"/tmp/k8s-metrics-server/metrics-certs/tls.key\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "CHMOD         \"/tmp/k8s-metrics-server/metrics-certs/tls.crt\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate
2025-04-30T15:10:24Z    DEBUG    controller-runtime.certwatcher    certificate event    {"event": "REMOVE        \"\""}
2025-04-30T15:10:24Z    INFO    controller-runtime.certwatcher    Updated current TLS certificate

logs from a non leader (instantly starts logging bad certificate after rotation, can only be solved by restarting the pod):

I0425 13:16:25.693120       1 leaderelection.go:250] attempting to acquire leader lease eco-system/etcd-cluster-operator-controller-leader-election-helper...
2025/04/30 15:50:42 http: TLS handshake error from 192.168.2.129:45812: remote error: tls: bad certificate

We were expecting non-leader pods to reload the cert too.

Reproducing this issue

No response

KubeBuilder (CLI) Version

kubebuilder 4.5.2

PROJECT version

3

Plugin versions

go.kubebuilder.io/v4

Other versions

No response

Extra Labels

No response

[camilamacedo86]

Hey, thank you so much for raising this and sharing the detailed logs — really appreciated! 🙌

From what you've described, I think it is the expected behavior of controller-runtime when leader election is enabled.

Only the leader pod runs the manager, which includes components like the metrics server and certwatcher. So when you add the certwatcher with mgr.Add(...), it only runs in the leader, and that’s why only it detects and reloads cert changes.

Non-leader pods stay idle—they don’t start the manager or any runnables, including the certwatcher. I understand that:

✅ When the non-leader becomes leader, it should:

• Start the manager,
• Load the latest certs from disk,
• And start watching for changes again.

🚨 If the new leader fails to pick up valid certs, or continues to throw tls: bad certificate errors, and does not work without intervention then we might need to look on that.

1. Could you confirm whether everything works as expected after the non-leader becomes leader?
2. Could you outline the step-by-step process you're using to reproduce this issue in a fresh Kubebuilder project?

That way we can try to reproduce it on our end and help identify whether there's a gap in the scaffolding or some subtle setup issue.

Thanks again for taking the time to report this — looking forward to your reply! 🙏

[hrak]

Hi @camilamacedo86

If this is expected behavior, for the webhook server that's not an issue, as only the leader's webhook server will be active and handling requests. But for metrics this is a different story. Monitoring metrics just breaks for the non-leader pods after rotation, as Prometheus can't connect anymore due to a bad certificate (192.168.2.129:45812 mentioned in the non-leader log above is our Prometheus instance).

I believe we tested the non-leader becoming a leader scenario yesterday, and it worked (as in: it picked up the new cert after becoming a leader), but we will verify this once more and report back.

[camilamacedo86]

Hi @hrak,

Please let me know if I misunderstood any part of your setup, but I’d like to recap the scenario as I understand it:

✅ Reproduction Steps

• Create a new project using Kubebuilder.
• Enable metrics with TLS support.
• The project sets up the certwatcher to reload metrics certificates.
• Enable Prometheus for metrics scraping using TLS and certs rotated with certwatcher
  (default scaffold with Prometheus, [METRICS-WITH-CERTS][PROMETHEUS-WITH-CERTS] all enabled)

All works as expected when running with a single replica.

🐛 The issue seems to occur only after modifying the controller Deployment to use replicas: 2. From your description, it sounds like:

• Prometheus stops being able to scrape metrics from the non-leader pod after cert rotation (due to outdated or invalid certs).
• But metrics continue to work correctly on the leader.

Is that accurate? Or do you still have access to metrics via the leader, and the non-leader errors are just noise in the logs?

---

🤔 A Few Follow-up Questions & Thoughts

a) If you keep the manager at a single replica (i.e., no manual change to replicas: 2), does everything behave as expected, with proper cert rotation and availability via the leader?

b) If so, then perhaps the best HA pattern would be to use the LeaderElectionReleaseOnCancel = true option in the manager config. This allows graceful leadership handoff without needing multiple concurrently running manager instances.

c) That said, since controller-runtime allows us to configure TLS for metrics and supports setting up a cert watcher, we could reasonably expect this to work regardless of leader status. Should it properly work when we have two replicas?

kubebuilder/testdata/project-v4/cmd/main.go

Lines 160 to 185 in 6790714

	// If the certificate is not specified, controller-runtime will automatically
	// generate self-signed certificates for the metrics server. While convenient for development and testing,
	// this setup is not recommended for production.
	//
	// TODO(user): If you enable certManager, uncomment the following lines:
	// - [METRICS-WITH-CERTS] at config/default/kustomization.yaml to generate and use certificates
	// managed by cert-manager for the metrics server.
	// - [PROMETHEUS-WITH-CERTS] at config/prometheus/kustomization.yaml for TLS certification.
	if len(metricsCertPath) > 0 {
	setupLog.Info("Initializing metrics certificate watcher using provided certificates",
	"metrics-cert-path", metricsCertPath, "metrics-cert-name", metricsCertName, "metrics-cert-key", metricsCertKey)
	var err error
	metricsCertWatcher, err = certwatcher.New(
	filepath.Join(metricsCertPath, metricsCertName),
	filepath.Join(metricsCertPath, metricsCertKey),
	)
	if err != nil {
	setupLog.Error(err, "to initialize metrics certificate watcher", "error", err)
	os.Exit(1)
	}
	metricsServerOptions.TLSOpts = append(metricsServerOptions.TLSOpts, func(config *tls.Config) {
	config.GetCertificate = metricsCertWatcher.GetCertificate
	})
	}

kubebuilder/testdata/project-v4/cmd/main.go

Lines 187 to 205 in 6790714

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
	Scheme: scheme,
	Metrics: metricsServerOptions,
	WebhookServer: webhookServer,
	HealthProbeBindAddress: probeAddr,
	LeaderElection: enableLeaderElection,
	LeaderElectionID: "da1d9c86.testproject.org",
	// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
	// when the Manager ends. This requires the binary to immediately end when the
	// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
	// speeds up voluntary leader transitions as the new leader don't have to wait
	// LeaseDuration time first.
	//
	// In the default scaffold provided, the program ends immediately after
	// the manager stops, so would be fine to enable this option. However,
	// if you are doing or is intended to do any operation such as perform cleanups
	// after the manager stops then its usage might be unsafe.
	// LeaderElectionReleaseOnCancel: true,
	})

So perhaps this highlights an improvement opportunity in controller-runtime — to make the metrics TLS watcher function correctly across replicas, not just the leader. It may make sense to raise this for further discussion with maintainers if it turns out to be an architectural limitation. @sbueringer @alvaroaleman @vincepri Could you please give a hand on this one? WDYT?

Thanks again for surfacing this — looking forward to your thoughts!

[camilamacedo86]

HI @hrak

Please let us know if you can check the above comment.
Could you help us by providing the steps to reproduce the scenario with the latest version?