Philipp Trulson created SOLR-18303:
--------------------------------------
Summary: Incremental backup failure-cleanup path masks the real
error with an unrelated `NoSuchFileException`
Key: SOLR-18303
URL: https://issues.apache.org/jira/browse/SOLR-18303
Project: Solr
Issue Type: Bug
Components: Backup/Restore
Affects Versions: 10.0
Reporter: Philipp Trulson
Disclaimer: Bug report was created with Claude, but I proof-read it and made
sure the issue is valid and well described.
When an incremental collection backup ({{{}action=BACKUP{}}}) fails during the
shard-copy phase, {{BackupCmd.call()}} logs the real exception and then
attempts to clean up the partially-created backup via
{{CollectionHandlingUtils.cleanBackup()}} →
{{{}DeleteBackupCmd.deleteBackupIds(){}}}. That cleanup code tries to delete
the backup's {{zk_backup_<id>}} directory defensively:
{code:java}
// DeleteBackupCmd.java:210-217 (10.0.0), :211-215 (main)
try {
for (BackupId backupId : backupIdsDeletes) {
repository.deleteDirectory(
repository.resolveDirectory(backupUri,
BackupFilePaths.getZkStateDir(backupId)));
}
} catch (FileNotFoundException e) {
// ignore this
}
{code}
{{zk_backup_<id>}} is only created later in {{BackupCmd.call()}} (via
{{{}backupMgr.createZkStateDir(){}}}, which runs *after* the shard-copy step).
If the shard-copy step fails, that directory was never created, so the delete
attempt above is expected to be a no-op — but
{{LocalFileSystemRepository.deleteDirectory()}} (via commons-io's
{{PathUtils.deleteDirectory()}} / NIO {{{}Files.walkFileTree(){}}}) throws
{{{}java.nio.file.NoSuchFileException{}}}, *not*
{{{}java.io.FileNotFoundException{}}}. These are unrelated exception types
({{{}NoSuchFileException extends FileSystemException extends IOException{}}};
it does not extend {{{}FileNotFoundException{}}}), so the catch clause never
fires.
The result: the {{NoSuchFileException}} propagates out of
{{{}cleanBackup(){}}}, which is itself uncaught in {{{}BackupCmd.call(){}}}'s
catch block, so it replaces the original {{throw e;}} that would have surfaced
the real {{{}SolrException{}}}. The client — and the server log — only ever see:
{noformat}
java.nio.file.NoSuchFileException: <location>/<name>/<collection>/zk_backup_0
{noformat}
with no indication of what actually failed. The real error is logged one line
earlier via {{{}log.error("Error happened during incremental backup for
collection: {}", ...){}}}, but is otherwise discarded from the API response and
easy to miss in the logs.
h3. Steps to Reproduce
Minimal, reliable repro — forces the shard-copy phase to fail for any reason:
# Fresh SolrCloud node, default {{solr.security.allow.paths}} (i.e. not set to
{{*}} and not covering the target location).
# Create a collection: {{citibike}} (or any collection with at least one
document).
# {{GET
/solr/admin/collections?action=BACKUP&name=citibike&collection=citibike&location=/tmp/some-backup-dir}}
where {{/tmp/some-backup-dir}} is outside the default allowed paths
({{{}SOLR_HOME{}}}/{{{}SOLR_DATA_HOME{}}}/{{{}coreRootDirectory{}}}).
h3. Expected Result
HTTP 500 with the real cause, e.g.:
{quote}Path ... must be relative to SOLR_HOME, SOLR_DATA_HOME
coreRootDirectory. Set system property 'solr.security.allow.paths' to add other
allowed paths.
{quote}
h3. Actual Result
HTTP 500 with {{msg: "<location>/citibike/citibike/zk_backup_0"}} (bare
{{{}NoSuchFileException{}}}, no explanation). The real cause above only appears
in the server log, one ERROR line earlier, and is easy to overlook since the
visible top-level exception is unrelated.
*Note:* any other failure during shard-copy reproduces the same masking — e.g.
this is effectively what happened in SOLR-15673 / SOLR-15696 for a
shard-split-related parsing bug; fixing that trigger did not fix the masking
itself, which is still present verbatim on {{main}} today.
h3. Suggested Fix
Two options, not mutually exclusive:
# *Minimal:* also catch {{java.nio.file.NoSuchFileException}} (or better,
check {{repository.exists(...)}} before calling {{{}deleteDirectory{}}}) at
{{{}DeleteBackupCmd.java:214{}}}.
# *More robust:* in {{{}BackupCmd.call(){}}}'s catch block, wrap the
{{cleanBackup()}} call in its own try/catch that logs-and-swallows *any*
exception from best-effort cleanup, so a secondary failure during rollback can
never suppress the original {{{}throw e;{}}}. This closes the whole class of
bug rather than just this one exception-type mismatch, and would have also
prevented the confusion in SOLR-15673/15696.
h3. Related
SOLR-15673, SOLR-15696 — same symptom ({{{}NoSuchFileException{}}} on
{{{}zk_backup_N{}}}), different root trigger (shard-split {{ShardBackupId}}
parsing), masking mechanism left unfixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
