Solr 9.8.1 Collections backup action fails on a large sharded collection, similar to SOLR-15673

Wed, 27 May 2026 12:25:10 -0700 

My issue is quite similar to the one reported in the email from Jordan Diehl 
back in September, 
2021<https://lists.apache.org/thread/8tq7tcj0j64g366yq4751ngrs8w0skr3> and 
logged as issue SOLR-15673<https://issues.apache.org/jira/browse/SOLR-15673>, 
which is supposed to have been fixed in Solr 9.0. We are running Solr 9.8.1 in 
a Windows server environment in cloud mode. We have a collection which was 
split into two shards immediately after creation, before we began to populate 
the index. Therefore, one of the conditions for SOLR-15673 is not met (backup 
before split). I have tested the same backup action on a much smaller sharded 
collection without seeing the failure.


The collection is now quite large (337GB in shard1_1 and 340GB in shard1_0):
[cid:e3041f1f-25de-4781-b69a-69a8a4fd0715]

The backup command
(https://localhost:8983/solr/admin/collections?action=BACKUP&collection=i2ksearch&name=i2ksearch-incremental-backup&ma
xNumBackupPoints=7&location=E:\solrsnapshots&async=13252cb7-1615-40fd-b16d-216f1b13b1e8)
starts and runs for about 90 minutes, then fails with a NoSuchFileException on 
a file named "zk_backup_0". Just before failing, the backup location appears to 
contain data that corresponds to both shards, but when it fails, only 340GB 
remain, evidently from shard1_0.

I have attached an edited Solr.log. Since Solr was processing other requests 
during the backup I have edited out many "o.a.s.u.p.LogUpdateProcessorFactory 
webapp=/solr path=/update" entries. Note that in the log there are 2 "Done 
creating backup snapshot" entries, one for each shard.


The status reported for the async backup operation shows this failure:
{
  "responseHeader":{
    "status":0,
    "QTime":1
  },
  "success":{
    "10.67.137.143:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":13
      }
    },
    "10.67.137.143:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":0
      }
    },
    "10.67.137.143:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":0
      },
      "STATUS":"completed",
      "response":{
        "startTime":"2026-05-14T18:29:37.176802700Z",
        "indexFileCount":412,
        "uploadedIndexFileCount":412,
        "indexSizeMB":345551.196,
        "uploadedIndexFileMB":345551.196,
        "shard":"shard1_1",
        "endTime":"2026-05-14T20:03:35.962806700Z",
        "shardBackupId":"md_shard1_1_0"
      }
    }
  },
  "afc5de6b-c016-4db2-99e7-65f897202c192982632021917200":{
    "responseHeader":{
      "status":0,
      "QTime":0
    },
    "STATUS":"completed",
    "response":{
      "startTime":"2026-05-14T18:29:37.176802700Z",
      "indexFileCount":412,
      "uploadedIndexFileCount":412,
      "indexSizeMB":345551.196,
      "uploadedIndexFileMB":345551.196,
      "shard":"shard1_1",
      "endTime":"2026-05-14T20:03:35.962806700Z",
      "shardBackupId":"md_shard1_1_0"
    }
  },
  "Operation backup caused 
exception:":"java.nio.file.NoSuchFileException:java.nio.file.NoSuchFileException:
 E:\\solrsnapshots\\i2ksearch-incremental-backup\\i2ksearch\\zk_backup_0",
  "exception":{
    
"msg":"E:\\solrsnapshots\\i2ksearch-incremental-backup\\i2ksearch\\zk_backup_0",
    "rspCode":-1
  },
  "status":{
    "state":"failed",
    "msg":"found [afc5de6b-c016-4db2-99e7-65f897202c19] in failed tasks"
  }
}


Mack



David Mack Endres,

Technical Advisor,

i2k Connect

[email protected]<mailto:[email protected]>

i2kconnect.com<https://urldefense.com/v3/__http:/i2kconnect.com/__;!!HTp9Kzo!kadyADft8ijimFr4TUhHWoaCw9QzETEvZZ-3P80mixJio8rtc4vFwNBjTzoCamtMvuqLxlCNKTypoT3cLshKO7e-$>



2026-05-14 18:29:37.155 INFO  (OverseerThreadFactory-17-thread-5) [c:i2ksearch 
s: r: x: t:] o.a.s.c.a.c.BackupCmd Starting backup of collection=i2ksearch with 
backupName=i2ksearch-incremental-backup at 
location=file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch
2026-05-14 18:29:37.159 INFO  (qtp1376151044-25690-null-992913) [c:i2ksearch s: 
r: x: t:null-992913] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/collections 
params={async=afc5de6b-c016-4db2-99e7-65f897202c19&maxNumBackupPoints=7&name=i2ksearch-incremental-backup&action=BACKUP&location=E:\solrsnapshots&collection=i2ksearch}
 status=0 QTime=24
2026-05-14 18:29:37.165 INFO  (qtp1376151044-25107-null-992914) [c: s: r: 
x:i2ksearch_shard1_0_replica_n3 t:null-992914] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={core=i2ksearch_shard1_0_replica_n3&async=afc5de6b-c016-4db2-99e7-65f897202c192982632022203400&qt=/admin/cores&name=shard1_0&shardBackupId=md_shard1_0_0&action=BACKUPCORE&location=file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch&incremental=true&wt=javabin&version=2}
 status=0 QTime=0
2026-05-14 18:29:37.175 INFO  (qtp1376151044-25115-null-992914) [c: s: r: 
x:i2ksearch_shard1_1_replica_n4 t:null-992914] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={core=i2ksearch_shard1_1_replica_n4&async=afc5de6b-c016-4db2-99e7-65f897202c192982632021917200&qt=/admin/cores&name=shard1_1&shardBackupId=md_shard1_1_0&action=BACKUPCORE&location=file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch&incremental=true&wt=javabin&version=2}
 status=0 QTime=13
2026-05-14 18:29:37.176 INFO  
(parallelCoreAdminAPIExpensiveExecutor-19-thread-14-processing-10.67.137.143:8983_solr
 i2ksearch_shard1_1_replica_n4 null-992914 
afc5de6b-c016-4db2-99e7-65f897202c192982632021917200 backupcore) [c:i2ksearch 
s:shard1_1 r:core_node6 x:i2ksearch_shard1_1_replica_n4 t:null-992914] 
o.a.s.h.IncrementalShardBackup Creating backup snapshot at 
file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch 
shardBackupMetadataFile:org.apache.solr.core.backup.ShardBackupId@430a0b46
2026-05-14 18:29:37.176 INFO  (qtp1376151044-25583-null-992916) [c: s: r: x: 
t:null-992916] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={requestid=afc5de6b-c016-4db2-99e7-65f897202c19&action=REQUESTSTATUS} 
status=0 QTime=1
2026-05-14 18:29:37.177 INFO  
(parallelCoreAdminAPIExpensiveExecutor-19-thread-13-processing-10.67.137.143:8983_solr
 i2ksearch_shard1_0_replica_n3 null-992914 
afc5de6b-c016-4db2-99e7-65f897202c192982632022203400 backupcore) [c:i2ksearch 
s:shard1_0 r:core_node5 x:i2ksearch_shard1_0_replica_n3 t:null-992914] 
o.a.s.h.IncrementalShardBackup Creating backup snapshot at 
file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch 
shardBackupMetadataFile:org.apache.solr.core.backup.ShardBackupId@58e686af
2026-05-14 18:29:37.179 INFO  (qtp1376151044-25690-null-992914) [c: s: r: x: 
t:null-992914] o.a.s.h.a.CoreAdminOperation Checking request status for : 
afc5de6b-c016-4db2-99e7-65f897202c192982632021917200
2026-05-14 18:29:37.179 INFO  (qtp1376151044-25690-null-992914) [c: s: r: x: 
t:null-992914] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&requestid=afc5de6b-c016-4db2-99e7-65f897202c192982632021917200&action=REQUESTSTATUS&wt=javabin&version=2}
 status=0 QTime=0
...
2026-05-14 20:03:35.912 INFO  (qtp1376151044-25949-null-992914) [c: s: r: x: 
t:null-992914] o.a.s.h.a.CoreAdminOperation Checking request status for : 
afc5de6b-c016-4db2-99e7-65f897202c192982632021917200
2026-05-14 20:03:35.912 INFO  (qtp1376151044-25949-null-992914) [c: s: r: x: 
t:null-992914] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&requestid=afc5de6b-c016-4db2-99e7-65f897202c192982632021917200&action=REQUESTSTATUS&wt=javabin&version=2}
 status=0 QTime=0
2026-05-14 20:03:35.962 INFO  
(parallelCoreAdminAPIExpensiveExecutor-19-thread-14-processing-10.67.137.143:8983_solr
 i2ksearch_shard1_1_replica_n4 null-992914 
afc5de6b-c016-4db2-99e7-65f897202c192982632021917200 backupcore) [c:i2ksearch 
s:shard1_1 r:core_node6 x:i2ksearch_shard1_1_replica_n4 t:null-992914] 
o.a.s.h.IncrementalShardBackup Done creating backup snapshot at 
file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch 
shardBackupMetadataFile:org.apache.solr.core.backup.ShardBackupId@430a0b46
...
2026-05-14 20:03:55.641 ERROR (OverseerThreadFactory-17-thread-5) [c:i2ksearch 
s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Collection i2ksearch, 
operation backup failed => java.nio.file.NoSuchFileException: 
E:\solrsnapshots\i2ksearch-incremental-backup\i2ksearch\zk_backup_0
      at 
java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
java.nio.file.NoSuchFileException: 
E:\solrsnapshots\i2ksearch-incremental-backup\i2ksearch\zk_backup_0
      at 
java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
 ~[?:?]
      at 
java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
 ~[?:?]
      at 
java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
 ~[?:?]
      at 
java.base/sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:53)
 ~[?:?]
      at 
java.base/sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:38)
 ~[?:?]
      at 
java.base/sun.nio.fs.WindowsFileSystemProvider.readAttributes(WindowsFileSystemProvider.java:199)
 ~[?:?]
      at 
java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
      at 
java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220) 
~[?:?]
      at 
java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
      at 
java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
      at 
java.base/java.nio.file.Files.walkFileTree(Files.java:2804) ~[?:?]
      at 
java.base/java.nio.file.Files.walkFileTree(Files.java:2882) ~[?:?]
      at 
org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:1653) 
~[commons-io-2.15.1.jar:2.15.1]
      at 
org.apache.commons.io.file.PathUtils.lambda$deleteDirectory$0(PathUtils.java:502)
 ~[commons-io-2.15.1.jar:2.15.1]
      at 
org.apache.commons.io.file.PathUtils.withPosixFileAttributes(PathUtils.java:1777)
 ~[commons-io-2.15.1.jar:2.15.1]
      at 
org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:501) 
~[commons-io-2.15.1.jar:2.15.1]
      at 
org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:486) 
~[commons-io-2.15.1.jar:2.15.1]
      at 
org.apache.solr.core.backup.repository.LocalFileSystemRepository.deleteDirectory(LocalFileSystemRepository.java:98)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.api.collections.DeleteBackupCmd.deleteBackupIds(DeleteBackupCmd.java:212)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.api.collections.CollectionHandlingUtils.cleanBackup(CollectionHandlingUtils.java:348)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.api.collections.BackupCmd.call(BackupCmd.java:132) 
~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564)
 ~[solr-core-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - houston 
- 2025-03-06 13:59:17]
      at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
 ~[solr-solrj-9.8.1.jar:9.8.1 dab835e4a3249780230e5b1cefebffdf15ad5c96 - 
houston - 2025-03-06 13:59:17]
      at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 ~[?:?]
      at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 ~[?:?]
      at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
...
2026-05-14 20:03:55.667 INFO  (OverseerThreadFactory-17-thread-5) [c:i2ksearch 
s: r: x: t:] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000057 doesn't exist. Requestor may 
have disconnected from ZooKeeper
2026-05-14 20:03:55.773 INFO  
(OverseerCollectionConfigSetProcessor-72208516633526272-10.67.137.143:8983_solr-n_0000000018)
 [c: s: r: x: t:] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000057 doesn't exist. Requestor may 
have disconnected from ZooKeeper
...
2026-05-14 20:04:00.515 INFO  
(parallelCoreAdminAPIExpensiveExecutor-19-thread-13-processing-10.67.137.143:8983_solr
 i2ksearch_shard1_0_replica_n3 null-992914 
afc5de6b-c016-4db2-99e7-65f897202c192982632022203400 backupcore) [c:i2ksearch 
s:shard1_0 r:core_node5 x:i2ksearch_shard1_0_replica_n3 t:null-992914] 
o.a.s.h.IncrementalShardBackup Done creating backup snapshot at 
file:///E:/solrsnapshots/i2ksearch-incremental-backup/i2ksearch 
shardBackupMetadataFile:org.apache.solr.core.backup.ShardBackupId@58e686af

Reply via email to