[
https://issues.apache.org/jira/browse/SOLR-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyi Dai updated SOLR-18584:
----------------------------
Description:
When RESTORE targets a collection that doesn't exist, RestoreCmd
(RestoreOnANewCollection) creates the collection, marks all its shards
CONSTRUCTION, creates one replica per shard, installs the backed up data on
each shard, and only then marks the shards ACTIVE. If installing the data fails
for a shard, requestShardsToRestore() throws and markAllShardsAsActive() is
never called. The collection is left in ZooKeeper with every shard in
CONSTRUCTION.
Unlike a failure to create the initial replicas, which deletes the collection,
nothing cleans this up. Retrying the same RESTORE now finds the collection, so
it takes the existing collection path (RestoreOnExistingCollection). That path
restores the data and reports success, but never marks CONSTRUCTION shards
ACTIVE. Updates to the collection are then rejected because no active shard
covers their hash range.
This was found through code analysis, and I reproduced it with a test using the
test framework's error-throwing backup repository. It happens when RESTORE is
called synchronously. An async RESTORE doesn't abort when restoring a shard
fails (see the TODO in ShardRequestTracker.processResponses): it reports
FAILED, but still marks the shards ACTIVE and leaves the collection in place
without the data that failed to restore.
*Steps to reproduce:*
# Back up a collection.
# Synchronously RESTORE the backup to a new collection T, with restoring the
shard data failing (e.g. the backup repository fails to copy an index file).
# Retry the same RESTORE once the repository works again.
# Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so updates
to T fail.
*Expected:* a failed restore to a new collection shouldn't leave a
half-restored collection behind, and a retry should produce a usable collection.
*Proposed fix:* under discussion. Deleting the collection on failure,
re-activating shards on retry, and refusing a retry when every shard is under
construction each have a problem; see the comments here and on the PR. Options
being considered are an explicit “restore in progress” marker on the new
collection or making shard activation synchronous.
was:
When RESTORE targets a collection that doesn't exist, RestoreCmd
(RestoreOnANewCollection) creates the collection, marks all its shards
CONSTRUCTION, creates one replica per shard, installs the backed up data on
each shard, and only then marks the shards ACTIVE. If installing the data fails
for a shard, requestShardsToRestore() throws and markAllShardsAsActive() is
never called. The collection is left in ZooKeeper with every shard in
CONSTRUCTION.
Unlike a failure to create the initial replicas, which deletes the collection,
nothing cleans this up. Retrying the same RESTORE now finds the collection, so
it takes the existing collection path (RestoreOnExistingCollection). That path
restores the data and reports success, but never marks CONSTRUCTION shards
ACTIVE. Updates to the collection are then rejected because no active shard
covers their hash range.
This was found through code analysis, and I reproduced it with a test using the
test framework's error-throwing backup repository. It happens when RESTORE is
called synchronously. An async RESTORE doesn't abort when restoring a shard
fails (see the TODO in ShardRequestTracker.processResponses): it reports
FAILED, but still marks the shards ACTIVE and leaves the collection in place
without the data that failed to restore.
*Steps to reproduce:*
# Back up a collection.
# Synchronously RESTORE the backup to a new collection T, with restoring the
shard data failing (e.g. the backup repository fails to copy an index file).
# Retry the same RESTORE once the repository works again.
# Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so updates
to T fail.
*Expected:* a failed restore to a new collection shouldn't leave a
half-restored collection behind, and a retry should produce a usable collection.
*Proposed fix:* if anything fails after the collection is created and before
its shards are marked ACTIVE, delete the collection, as is already done when
the initial replicas can't be created, and treat a shard restore failure
recorded by an async request the same way. A retry then restores to a new
collection again. I have a patch with a test and will open a PR.
> A failed RESTORE to a new collection leaves its shards in CONSTRUCTION, and
> retrying the restore doesn't fix them
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18584
> URL: https://issues.apache.org/jira/browse/SOLR-18584
> Project: Solr
> Issue Type: Bug
> Components: Backup/Restore
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When RESTORE targets a collection that doesn't exist, RestoreCmd
> (RestoreOnANewCollection) creates the collection, marks all its shards
> CONSTRUCTION, creates one replica per shard, installs the backed up data on
> each shard, and only then marks the shards ACTIVE. If installing the data
> fails for a shard, requestShardsToRestore() throws and
> markAllShardsAsActive() is never called. The collection is left in ZooKeeper
> with every shard in CONSTRUCTION.
> Unlike a failure to create the initial replicas, which deletes the
> collection, nothing cleans this up. Retrying the same RESTORE now finds the
> collection, so it takes the existing collection path
> (RestoreOnExistingCollection). That path restores the data and reports
> success, but never marks CONSTRUCTION shards ACTIVE. Updates to the
> collection are then rejected because no active shard covers their hash range.
> This was found through code analysis, and I reproduced it with a test using
> the test framework's error-throwing backup repository. It happens when
> RESTORE is called synchronously. An async RESTORE doesn't abort when
> restoring a shard fails (see the TODO in
> ShardRequestTracker.processResponses): it reports FAILED, but still marks the
> shards ACTIVE and leaves the collection in place without the data that failed
> to restore.
> *Steps to reproduce:*
> # Back up a collection.
> # Synchronously RESTORE the backup to a new collection T, with restoring the
> shard data failing (e.g. the backup repository fails to copy an index file).
> # Retry the same RESTORE once the repository works again.
> # Result: the retry succeeds, but T's shards stay in CONSTRUCTION, so
> updates to T fail.
> *Expected:* a failed restore to a new collection shouldn't leave a
> half-restored collection behind, and a retry should produce a usable
> collection.
> *Proposed fix:* under discussion. Deleting the collection on failure,
> re-activating shards on retry, and refusing a retry when every shard is under
> construction each have a problem; see the comments here and on the PR.
> Options being considered are an explicit “restore in progress” marker on the
> new collection or making shard activation synchronous.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
