Prevent moco-controller startup cache sync failures from disabling failover and switchover

[arosh]

Summary

After upgrading controller-runtime to 0.23.x, moco-controller can fail to start when a cluster has a large number of MySQLCluster resources and secondary resources. In production, this caused all moco-controller replicas to become unavailable, which disabled MOCO failover and switchover.

We need to introduce multiple safeguards so that moco-controller remains available during startup, upgrade, and leader failover even in large-scale environments.

Context

controller-runtime 0.23.x changed controller source startup behavior to wait not only for informer caches to sync, but also for event handlers to process initial objects. This exposed expensive secondary resource handlers in MOCO.

The current MySQLClusterReconciler watches secondary resources such as ConfigMap and BackupPolicy. For each secondary resource event, the handler lists all MySQLCluster objects in the namespace and scans them to find matching references. With around 1000 MySQLCluster resources, this can make controller startup slow enough to hit cache sync timeout errors such as:

failed to wait for mysqlcluster caches to sync kind source
timed out waiting for cache to be synced for Kind *v1.ConfigMap

When this happens during rollout or restart, moco-controller replicas may fail readiness/startup and the controller fleet can become unavailable. Since moco-controller is responsible for MOCO operational workflows, this directly impacts failover and switchover.

Relevant upstream context:

• ✨ Delay reconciliation until handlers sync kubernetes-sigs/controller-runtime#3406 changed source startup to wait for handler sync.
• Handlers sync leads to cache sync timeouts kubernetes-sigs/controller-runtime#3466 discusses regressions caused by slow handlers during initial sync.
• Warm replica support can reduce leader failover downtime, but it does not by itself reduce the total work done by expensive handlers.

Plan

• Add a regression test that demonstrates the current startup failure or severe slowdown with around 1000 MySQLCluster resources.
• Add a configurable CacheSyncTimeout flag for moco-controller.
• Optimize secondary resource handlers in MySQLClusterReconciler.
• Make readiness depend on controller warmup completion during moco-controller RollingUpdate.
• Tighten PodWatcher filtering so it only watches or enqueues MOCO-managed Pods.

Add a large-scale startup regression test

Introduce a test that reproduces the pathological behavior with many MySQLCluster resources and secondary resources.

The test should validate that the current handler pattern does not scale and should protect future changes from reintroducing the same issue. We should evaluate whether this is best implemented with envtest or with a fake client/cache setup. Envtest is likely closer to the real controller-runtime cache behavior, while a fake setup may be faster and easier to run in unit tests.

The test should cover at least the ConfigMap watch path, because that is one of the observed timeout kinds. If practical, also cover BackupPolicy.

Add a configurable CacheSyncTimeout flag

Expose a moco-controller flag to configure controller-runtime Controller.CacheSyncTimeout.

This is a mitigation, not the primary fix. It gives operators a way to recover or tune behavior in large environments while deeper handler optimizations are rolled out.

The default should preserve current controller-runtime behavior unless we intentionally choose a safer MOCO-specific default.

Optimize secondary resource handlers

Replace the current list-and-scan handlers with indexed lookups.

The ConfigMap handler should not list all MySQLCluster objects and scan spec.mysqlConfigMapName for every ConfigMap event. Instead, register a field index for MySQLCluster.spec.mysqlConfigMapName and use client.MatchingFields.

The BackupPolicy handler should follow the same pattern with MySQLCluster.spec.backupPolicyName.

This should reduce initial handler sync cost from roughly secondary resource count * MySQLCluster count to indexed lookups for only matching clusters.

Gate readiness on warmup completion

During RollingUpdate, a new moco-controller Pod should not become ready before its controllers have completed source/cache warmup. Otherwise Kubernetes may continue the rollout while the replacement replica is not yet capable of taking over failover/switchover duties quickly.

Investigate controller-runtime warm replica support and readiness integration. If EnableWarmup is used, readiness should only return OK after the relevant controllers have completed warmup. If controller-runtime does not expose enough state directly, introduce MOCO-side tracking around controller startup/warmup completion.

This should be designed carefully because warm replicas increase API server List/Watch load.

Tighten PodWatcher filtering

PodWatcher currently watches Pods broadly and filters later in reconcile. In large clusters, this can create unnecessary initial events and queue pressure.

Restrict the watch path to MOCO-managed Pods as early as possible, using predicates and/or cache selectors where appropriate. The goal is to avoid processing unrelated Pods during startup and normal operation.

Note

Warm replicas are useful for reducing downtime during leader failover, but they are not a substitute for optimizing slow event handlers. The primary root cause is the synchronous startup cost of secondary resource handlers after controller-runtime 0.23.x.