autokctrl-team opened a new issue, #848: URL: https://github.com/apache/solr-operator/issues/848
# Environment - solr-operator built from `ed5c5c7d28a4c1189d19f581259e05385c0d4b20` - Solr 9.7.0 - Kubernetes v1.35.0 (kind v0.31.0, 3 nodes) - A `SolrBackup` against a healthy single-node SolrCloud # What happened A `SolrBackup` submitted a backup to Solr, but the operator failed to save `InProgress=true` to the apiserver. The backup then remained stuck. This happens if the operator restarts after Solr accepts the backup but before the status update is saved. After the restart, the operator reads `InProgress=false` and submits the same backup again. Solr rejects the duplicate async ID, so the operator never polls or cleans up the original request. The backup only recovered after I manually called `DELETESTATUS`. The next submission then succeeded and the backup completed. # Where the source code is wrong [`reconcileSolrCollectionBackup`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L285-L300) decides what to do from the saved `InProgress` value. If the value is `false`, it submits a backup without first checking Solr: ```go // controllers/solrbackup_controller.go:285-300 if collectionBackupStatus.Finished { return true, nil } else if !collectionBackupStatus.InProgress { started, err = util.StartBackupForCollection(...) // no pre-check if err != nil { return true, err } collectionBackupStatus.InProgress = started // in-memory only ... } else if collectionBackupStatus.InProgress { // REQUESTSTATUS poll, and DELETESTATUS cleanup on finish } ``` `InProgress` is only persisted [at the end of `Reconcile`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solrbackup_controller.go#L178-L181): ```go // controllers/solrbackup_controller.go:178-181 if !reflect.DeepEqual(unmodifiedBackupResource.Status, backup.Status) { err = r.Status().Patch(ctx, backup, client.MergeFrom(unmodifiedBackupResource)) } ``` If this patch does not complete after Solr accepts the backup, etcd still contains `InProgress=false`. Every later reconcile submits the same async ID. Solr rejects it, and the function returns before setting `InProgress=true`. This repeats indefinitely. Solr keeps completed async records until `DELETESTATUS` is called. However, the operator only calls `DELETESTATUS` when `InProgress=true`, so it cannot clean up the record. The [cluster-operation code](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L564-L570) avoids this problem by checking Solr before submitting: ```go // controllers/util/solr_update_util.go:564-570 // First check to see if the Async Replace request has started if asyncState, message, asyncErr := solr_api.CheckAsyncRequest(ctx, solrCloud, requestId); asyncErr != nil { ... } else if asyncState == "notfound" { // Submit new Replace Node request ``` This code can detect an existing request even if an operator status update was lost. The backup code does not perform this check. # Expected behavior A failed status update should not leave the backup stuck after Solr has accepted it. Before submitting, the backup code should call `CheckAsyncRequest` with the async ID: - `notfound` → submit the backup - running / completed / failed → set `InProgress=true` and use the existing polling path -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
