Hello, I have been unsuccessfully trying to find a way to restore a collection using the Collections API for the last several days. *I would greatly appreciate any help as I am now stuck.* I am using Solr 8.2 in cloud mode. To simplify things, I have only a single Solr node on my local machine (though I have also tried multi-node setups). I am able to successfully create collections, index documents, and run searches against the index. General gist of what I am trying to do is to be able to backup an existing collection, delete that collection, and then restore the collection from backup when needed. Our use case requires creating thousands of collections, which I believe is highly not recommended in Solr Cloud (is that correct?), so this backup-delete-restore mechanism is our way of reducing the number of collections in Solr at any given time. Just FYI, in our use case it is ok for searches to take a couple of minutes to produce results.
So far I have not been able to restore a single collection using the collection RESTORE API. I am able to create backups by running the following HTTP command in Solr admin console: http://localhost:8983/solr/admin/collections?action=BACKUP&name=test1_backup2&collection=test1_20200407_170438_20200407_170441&location=C:\TEST\DELETE\BACKUPS The backup appears to be successfully created. It contains a snapshot from "shard1", which is the only shard in the cluster. It also contains the "zk_backup" directory, which contains my Solr config and "collection_state.json" file. And it contains a "backup.properties" file. Everything looks good to me. However, when I attempt to restore the collection using the following HTTP command: http://localhost:8983/solr/admin/collections?action=RESTORE&name=test1_backup2&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW I get the same extremely vague error messages in Solr logs: *RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core [test1_backup_NEW_shard1_replica_n1] Caused by: nullERROR (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica * Below is the full Solr log of the restore command: 2020-04-08 21:10:56.753 INFO (qtp1099855928-24) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :restore with params name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW and sendToOCPQueue=true 2020-04-08 21:10:56.783 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Using existing config AutopsyConfig 2020-04-08 21:10:56.783 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Starting restore into collection=test1_backup_NEW with backup_name=test1_backup2 at location=file:///C:/TEST/DELETE/BACKUPS/ 2020-04-08 21:10:56.784 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd Create collection test1_backup_NEW 2020-04-08 21:10:56.900 WARN (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (test1_backup_NEW) without cores. 2020-04-08 21:10:56.908 INFO (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [ ] o.a.s.c.o.SliceMutator Update shard state invoked for collection: test1_backup_NEW with message: { "shard1":"construction", "collection":"test1_backup_NEW", "operation":"updateshardstate"} 2020-04-08 21:10:56.908 INFO (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [ ] o.a.s.c.o.SliceMutator Update shard state shard1 to construction 2020-04-08 21:10:56.925 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.RestoreCmd Adding replica for shard=shard1 collection=DocCollection(test1_backup_NEW//collections/test1_backup_NEW/state.json/0)={ "pullReplicas":0, "replicationFactor":1, "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false", "nrtReplicas":1, "tlogReplicas":0} 2020-04-08 21:10:56.928 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Node Identified localhost:8983_solr for creating new replica of shard shard1 for collection test1_backup_NEW 2020-04-08 21:10:56.939 INFO (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command. 2020-04-08 21:10:56.945 INFO (OverseerStateUpdate-72057883336638464-localhost:8983_solr-n_0000000003) [ ] o.a.s.c.o.SliceMutator createReplica() { "operation":"addreplica", "collection":"test1_backup_NEW", "shard":"shard1", "core":"test1_backup_NEW_shard1_replica_n1", "state":"down", "base_url":"http://localhost:8983/solr";, "node_name":"localhost:8983_solr", "type":"NRT"} 2020-04-08 21:10:57.348 INFO (qtp1099855928-19) [ x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT 2020-04-08 21:10:57.373 WARN (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrConfig Couldn't add files from C:\total\crap\dir\ignored filtered by null to classpath: C:\total\crap\dir\ignored 2020-04-08 21:10:57.374 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrResourceLoader [test1_backup_NEW_shard1_replica_n1] Added 56 libs to classloader, from paths: [/C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/clustering/lib, /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/extraction/lib, /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/langid/lib, /C:/Bitnami/solr-8.2.0-2/apache-solr/contrib/velocity/lib] 2020-04-08 21:10:57.395 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.2.0 2020-04-08 21:10:57.424 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.s.IndexSchema [test1_backup_NEW_shard1_replica_n1] Schema name=Autopsy Keyword Search 2020-04-08 21:10:57.451 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema Autopsy Keyword Search/2.2 with uniqueid field id 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'test1_backup_NEW_shard1_replica_n1' using configuration from collection test1_backup_NEW, trusted=true 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.test1_backup_NEW.shard1.replica_n1' (registry 'solr.core.test1_backup_NEW.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@214b199c 2020-04-08 21:10:57.473 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore [[test1_backup_NEW_shard1_replica_n1] ] Opening new SolrCore at [C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1], dataDir=[C:\Bitnami\solr-8.2.0-2\apache-solr\server\solr\test1_backup_NEW_shard1_replica_n1\data\] 2020-04-08 21:10:57.491 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if 15000 uncommitted docs; 2020-04-08 21:10:57.527 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled 2020-04-08 21:10:57.535 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1] main] 2020-04-08 21:10:57.536 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/AutopsyConfig 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/AutopsyConfig 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers 2020-04-08 21:10:57.537 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01} 2020-04-08 21:10:57.540 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@3a4eb909 2020-04-08 21:10:57.541 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} 2020-04-08 21:10:57.541 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.test1_backup_NEW.shard1.replica_n1, tag=3a4eb909 2020-04-08 21:10:57.542 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@54eb018f: rootName = null, domain = solr.core.test1_backup_NEW.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.test1_backup_NEW.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@55bc77fd 2020-04-08 21:10:57.542 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.S.Request [test1_backup_NEW_shard1_replica_n1] webapp=null path=null params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher} hits=0 status=0 QTime=1 2020-04-08 21:10:57.542 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.QuerySenderListener QuerySenderListener done. 2020-04-08 21:10:57.542 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2020-04-08 21:10:57.542 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak 2020-04-08 21:10:57.542 INFO (searcherExecutor-22-thread-1-processing-n:localhost:8983_solr x:test1_backup_NEW_shard1_replica_n1 c:test1_backup_NEW s:shard1 r:core_node2) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.c.SolrCore [test1_backup_NEW_shard1_replica_n1] Registered new searcher Searcher@59a2b753[test1_backup_NEW_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} 2020-04-08 21:10:57.552 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.test1_backup_NEW.shard1.leader, tag=3a4eb909 2020-04-08 21:10:57.560 ERROR (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core [test1_backup_NEW_shard1_replica_n1] Caused by: null at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1195) at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92) at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360) at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:311) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:170) at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:125) at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:348) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Unable to create core [test1_backup_NEW_shard1_replica_n1] at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1266) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1156) ... 50 more Caused by: org.apache.solr.common.SolrException at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1055) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:882) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1245) ... 51 more Caused by: java.lang.NullPointerException at org.apache.solr.core.SolrCore.bufferUpdatesIfConstructing(SolrCore.java:1097) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1029) ... 53 more 2020-04-08 21:10:57.561 INFO (qtp1099855928-19) [c:test1_backup_NEW s:shard1 r:core_node2 x:test1_backup_NEW_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=AutopsyConfig&name=test1_backup_NEW_shard1_replica_n1&action=CREATE&collection=test1_backup_NEW&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=400 QTime=213 2020-04-08 21:10:57.577 ERROR (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: http://localhost:8983/solr => org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Error CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core [test1_backup_NEW_shard1_replica_n1] Caused by: null at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727) org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Error CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core [test1_backup_NEW_shard1_replica_n1] Caused by: null at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746) ~[?:?] at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) ~[?:?] at org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238) ~[?:?] at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_222-1-ojdkbuild] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_222-1-ojdkbuild] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_222-1-ojdkbuild] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ~[metrics-core-4.0.5.jar:4.0.5] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222-1-ojdkbuild] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222-1-ojdkbuild] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222-1-ojdkbuild] 2020-04-08 21:10:57.578 ERROR (OverseerThreadFactory-9-thread-1-processing-n:localhost:8983_solr) [c:test1_backup_NEW ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: test1_backup_NEW operation: restore failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013) at org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177) at org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708) at org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264) at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at null: Error CREATEing SolrCore 'test1_backup_NEW_shard1_replica_n1': Unable to create core [test1_backup_NEW_shard1_replica_n1] Caused by: null at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:727) at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:400) at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) at org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238) at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ... 4 more 2020-04-08 21:10:57.604 INFO (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=test1_backup2&action=RESTORE&location=C:\TEST\DELETE\BACKUPS&collection=test1_backup_NEW} status=500 QTime=850 2020-04-08 21:10:57.605 ERROR (qtp1099855928-24) [c:test1_backup_NEW ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica at org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53) at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280) at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.lang.Thread.run(Thread.java:748) 2020-04-08 21:10:58.775 INFO (OverseerCollectionConfigSetProcessor-72057883336638464-localhost:8983_solr-n_0000000003) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000014 doesn't exist. Requestor may have disconnected from ZooKeeper 2020-04-08 21:11:16.690 INFO (qtp1099855928-21) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.test1_backup_NEW2.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_backup_NEW2.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW2.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:INDEX.sizeInBytes&key=solr.core.test1_backup_NEW.shard1.replica_n1:UPDATE./update.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:UPDATE./update.requests&key=solr.core.test1_20200407_170438_20200407_170441.shard1.replica_n1:QUERY./select.requests&key=solr.core.xp_20200408_162046_20200408_162051.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:QUERY./select.requests&key=solr.core.test1_backup_NEW.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=51 2020-04-08 21:11:16.731 INFO (qtp1099855928-24) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used} status=0 QTime=27
