Prevent MySQLCluster controller startup slowdown in large clusters

[arosh]

Summary

This PR fixes the main cause of issue #922.
It reduces the startup cost of moco-controller when many MySQLCluster and secondary resources exist in the same namespace.

The current implementation scanned all MySQLCluster objects for each ConfigMap and BackupPolicy event. This became expensive after the controller-runtime upgrade and could make cache sync fail during controller startup. In that case, MOCO failover and switchover could be unavailable because all controller replicas might fail to become ready.

Context

After upgrading to controller-runtime 0.23.x, controller startup became more sensitive to slow event handlers during initial cache sync.

In MOCO, the MySQLClusterReconciler watches secondary resources such as ConfigMap and BackupPolicy. The old handlers used a list-and-scan pattern, so startup work increased too much in large-scale environments. This was the root cause behind the cache sync timeout described in issue #922.

Implementation

This PR replaces the list-and-scan logic with field index lookups.

MySQLCluster objects are now indexed by:

• spec.mysqlConfigMapName
• spec.backupPolicyName

During ConfigMap and BackupPolicy events, the reconciler now lists only matching MySQLCluster objects by using client.MatchingFields, instead of scanning every cluster in the namespace.

This PR also adds a startup regression test for the MySQLCluster controller. The test creates 1000 MySQLCluster objects and 1000 ConfigMap objects before the manager starts, then checks that the controller can still start and process initial events in time.

Note

In addition, test startup goroutines were updated to use GinkgoRecover so failures are reported correctly, and focused test instructions were added to make the new test easier to run.

[Copilot]

Pull request overview

This PR addresses MOCO controller startup slowdowns in namespaces with many MySQLCluster objects and secondary resources (e.g., ConfigMap, BackupPolicy) by replacing expensive “list all clusters then scan” watch handlers with field-index-based lookups. It also adds an envtest regression test to ensure the controller can start promptly even with large numbers of pre-existing objects, and improves Ginkgo goroutine failure reporting.

Changes:

• Index MySQLCluster by spec.mysqlConfigMapName and spec.backupPolicyName, and use client.MatchingFields in secondary-resource event handlers.
• Add a large-scale startup regression test that creates 1000 MySQLCluster + 1000 ConfigMap objects before starting the manager.
• Update test goroutines to use defer GinkgoRecover() and add focused-test execution instructions.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
controllers/mysqlcluster_controller.go	Replaces secondary watch list-and-scan handlers with field-index lookups for ConfigMap/BackupPolicy references.
cmd/moco-controller/cmd/run.go	Passes a context into MySQLClusterReconciler.SetupWithManager and reuses the signal context consistently.
controllers/mysqlcluster_controller_slowdown_test.go	Adds an envtest regression test covering large-scale controller startup behavior.
controllers/mysqlcluster_controller_test.go	Updates controller setup call signature and improves goroutine failure reporting with GinkgoRecover.
controllers/pod_watcher_test.go	Improves goroutine failure reporting with GinkgoRecover and Gomega assertions.
controllers/partition_controller_test.go	Improves goroutine failure reporting with GinkgoRecover and Gomega assertions.
clustering/manager_test.go	Improves goroutine failure reporting with GinkgoRecover and Gomega assertions.
.github/instructions/focused-tests.instructions.md	Documents preferred Makefile-based focused test execution paths for MOCO.

[yoheinbb]

@arosh
There are merge conflicts in this PR. Could you please resolve them?

[arosh]

@yoheinbb
resolved 78e1c9e