HoustonPutman opened a new pull request, #910:
URL: https://github.com/apache/solr-operator/pull/910
Fixes #824
**This replaces #846.** #846 treats a `notfound` async status as a terminal
failure. This PR instead submits the backup again, so a recurring backup keeps
going and still produces a backup, rather than recording a failed run. It
builds on #907, which handles a backup's async ID being present in Solr before
it is submitted.
### The problem
A collection backup whose async status Solr reports as `notfound` stayed
`inProgress` forever. Because the SolrBackup never finished, a recurring backup
was never scheduled again and `maxSaved` pruning stopped. Nothing was reported.
### Changes
- **Resubmit on `notfound`.** Once a collection backup has been `notfound`
for more than 5 seconds after it was submitted, it is submitted again, with the
same name and async ID. Its start time is reset on every submission, so the 5
second grace also applies after a resubmission.
- **Delete the async status only after the result is saved.** Solr's async
status for a finished collection backup used to be deleted before the
SolrBackup status recording the result was saved. If that save failed, the next
reconcile found the backup still in progress, with Solr reporting it as
`notfound`. That is one way into #824.
- `Reconcile` now fetches the SolrCloud and builds its auth context.
- `reconcileSolrCloudBackup` returns the collections whose async requests
finished.
- `Reconcile` deletes their async statuses only once the status patch
succeeds.
- **Don't finish without a collection that failed to start.** If a
collection's BACKUP call failed, its status was never recorded. Once the other
collections finished, the SolrBackup was marked finished, and successful,
without it. Its status is now recorded as not started, so the SolrBackup waits
for it.
- **Error handling.**
- A failure for one collection no longer stops the other collections from
being checked. The errors are returned together.
- A failure to list the collections ends the reconcile.
- Errors are only recorded as events once, as `BackupError` by
`Reconcile`. The duplicate `BackupCloudNotReady` and
`BackupCollectionListError` events are removed. Events were added in #836,
which has not been released.
### Testing
- New unit tests in `controllers/solrbackup_controller_unit_test.go`,
against a fake Solr Collections API:
- resubmitting a `notfound` backup and seeing it through to completion;
- no resubmit within the grace period, including right after a
resubmission;
- the async status is deleted only after the status is saved, and kept
when the save fails;
- a collection that failed to start holds up the SolrBackup;
- a failure to list collections ends the reconcile.
- `events_test.go`: the cloud-not-ready test now goes through `Reconcile`,
and checks that exactly one warning event is recorded.
- #907's tests in `solrbackup_async_recovery_test.go` are updated for the
deletion now happening after the status save, and for the status entry now
being recorded when the start fails.
- Each new test fails without its fix. All plain controller tests pass, and
`go vet` is clean, at every commit.
- Not yet run: the envtest suite (`make test`) and the backup e2e tests
(`make e2e-tests TEST_FILES=backups_test.go`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]