TravisFarrellMCC commented on issue #824: URL: https://github.com/apache/solr-operator/issues/824#issuecomment-6332039095
If the fix provided by @autokctrl-team are valid, please merge his PR into the code. This is a serverly disruptive and critial issue for a large enterprise with a large and very busy Solr Cloud. I'm adding another production data point. This is hitting us repeatedly on recurring backups, and it's silent. **Environment:** solr-operator v0.8.1, Solr 9.8.1, GKE, GCS backup repository, one recurring `SolrBackup` (`@hourly`, `maxSaved: 48`) over 41 collections, in two separate clusters (primary and DR). **What we see over 11 days (Oct 10–21):** - 7 separate stalls in the primary cluster. Each time, a subset of collections in one scheduled run (2 to 36 of 41) ended with `asyncBackupStatus: notfound` and `finished: null`, and the run never went terminal. Top-level `finished`/`successful` stayed `null`, so no further runs were scheduled. - Gaps ranged from ~2.5 hours to ~92 hours. Only 48 runs completed in a window that should have had ~260. - Our DR cluster stalled the same way after 9 hourly runs and stayed stalled until we noticed. - In the retained history, `notfound` appears only on collections in stalled runs (74 of 2,009 collection statuses). It never appears on a collection that finished. In every case we checked, Solr had actually written the backup (`backup_N.properties` and shard metadata are present in GCS). Only the operator's tracking was lost. **Why this matters:** nothing alerts. The operator reports no error, the CR just stays `inProgress`, and the backup schedule silently stops. We found out because we built external freshness monitoring on the GCS bucket. Before that, our longest gap was nearly four days without a backup. We also confirmed that `controllers/util/backup_util.go` and `controllers/util/solr_api/backup.go` are unchanged between v0.8.1 and `main`, so upgrading to v0.9.1 doesn't help. This matches @benediktarnold's reports above on v0.9.1. +1 for #846. Treating `notfound` as terminal and unsuccessful, while keeping the status visible, is the right behavior: a failed run is far better than a frozen schedule. Happy to test a build of #846 against our setup and report back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
