Re: Solr cloud backup/restore not working
On 6/17/2020 8:55 PM, yaswanth kumar wrote: Caused by: javax.crypto.BadPaddingException: RSA private key operation failed Something appears to be wrong with the private key that Solr is attempting to use for a certificate. Best guess, incorporating everything I can see in the stacktrace, is that you have enabled certificate-based authentication, and the private key for the client certificate is malformed in some way. The error message originated in Java code, not Solr. https://docs.oracle.com/javase/8/docs/api/javax/crypto/BadPaddingException.html It sounds like the keystore has a problem. You would need to consult with someone who is an expert at Java crypto mechanisms. Thanks, Shawn
Re: Solr cloud backup/restore not working
On 6/16/2020 8:44 AM, yaswanth kumar wrote: I don't see anything related in the solr.log file for the same error. Not sure if there is anyother place where I can check for this. The underlying request that failed might be happening on one of the other nodes in the cloud. It might be necessary to check the solr.log file on multiple machines. The response here does NOT contain any information about what caused the problem. All it says is that an ADDREPLICA action necessary to complete the restore failed. You'll need to locate the node where the ADDREPLICA failed, and we will need to see the FULL error message. It is probably dozens of lines in length. I see that you've opened an issue in Jira. That is premature. The Solr project does NOT use Jira as a support portal. If we determine that you're running into a bug, then it would be appropriate to open an issue. Thanks, Shawn
Re: Solr cloud backup/restore not working
Hi Vinodh, Here is what I see when I tried with requestid, Collection: test operation: restore failed:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1030) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler$ShardRequestTracker.processResponses(OverseerCollectionMessageHandler.java:1013) at org.apache.solr.cloud.api.collections.AddReplicaCmd.lambda$addReplica$1(AddReplicaCmd.java:177) at org.apache.solr.cloud.api.collections.AddReplicaCmd$$Lambda$746/.run(Unknown Source) at org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:199) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.addReplica(OverseerCollectionMessageHandler.java:708) at org.apache.solr.cloud.api.collections.RestoreCmd.call(RestoreCmd.java:286) at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264) at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA private key operation failed at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325) at org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305) at org.apache.solr.security.PKIAuthenticationPlugin.access$200(PKIAuthenticationPlugin.java:61) at org.apache.solr.security.PKIAuthenticationPlugin$2.onQueued(PKIAuthenticationPlugin.java:239) at org.apache.solr.client.solrj.impl.Http2SolrClient.decorateRequest(Http2SolrClient.java:468) at org.apache.solr.client.solrj.impl.Http2SolrClient.makeRequest(Http2SolrClient.java:455) at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:364) at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:746) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) at org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:238) at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199) at org.apache.solr.handler.component.HttpShardHandler$$Lambda$529/.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ... 5 more Caused by: javax.crypto.BadPaddingException: RSA private key operation failed at java.base/sun.security.rsa.NativeRSACore.crtCrypt_Native(NativeRSACore.java:149) at java.base/sun.security.rsa.NativeRSACore.rsa(NativeRSACore.java:91) at java.base/sun.security.rsa.RSACore.rsa(RSACore.java:149) at java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:355) at java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392) at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2260) at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:323) Thanks, On Wed, Jun 17, 2020 at 8:08 AM Kommu, Vinodh K. wrote: > Hi, > > What is the log level defined for solr nodes? Did you used requestid in > restore command? If so, check the status of the requestid if that points to > any errors. > > Thanks & Regards, > Vinodh > > -Original Message- > From: yaswanth kumar > Sent: Wednesday, June 17, 2020 4:33 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr cloud backup/restore not working > > ATTENTION: External Email – Be Suspicious of Attachments, Links and > Requests for Login Information. > > Can someone please guide me on where can I get more detailed error of the > above exception while doing restore?? All that I see in solr.log was pasted > above > > Thanks, > > On Tue, Jun 16, 2020 at 10:44 AM yaswanth kumar > wrote: > > > I don't see anything related in the solr.log file for the same error. > > Not sure if there is anyother place where I can check for this. > > > > Thanks, > > > > On Tue, Jun 16, 2020 at 10:21 AM Shawn Heisey > wrote: > > > >> On
RE: Solr cloud backup/restore not working
Hi, What is the log level defined for solr nodes? Did you used requestid in restore command? If so, check the status of the requestid if that points to any errors. Thanks & Regards, Vinodh -Original Message- From: yaswanth kumar Sent: Wednesday, June 17, 2020 4:33 PM To: solr-user@lucene.apache.org Subject: Re: Solr cloud backup/restore not working ATTENTION: External Email – Be Suspicious of Attachments, Links and Requests for Login Information. Can someone please guide me on where can I get more detailed error of the above exception while doing restore?? All that I see in solr.log was pasted above Thanks, On Tue, Jun 16, 2020 at 10:44 AM yaswanth kumar wrote: > I don't see anything related in the solr.log file for the same error. > Not sure if there is anyother place where I can check for this. > > Thanks, > > On Tue, Jun 16, 2020 at 10:21 AM Shawn Heisey wrote: > >> On 6/12/2020 8:38 AM, yaswanth kumar wrote: >> > Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for >> > each >> > collection) >> > Enabled basic authentication and gave all access to the admin user >> > >> > Now trying to use solr cloud backup/restore API, backup is working >> great, >> > but when trying to invoke restore API its throwing the below error >> >> > "msg":"ADDREPLICA failed to create replica", >> > "trace":"org.apache.solr.common.SolrException: ADDREPLICA >> > failed to create replica\n\tat >> > >> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.j >> ava:53)\n\tat >> > >> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(Collect >> ionsHandler.java:280)\n\tat >> >> The underlying cause of this exception is not recorded here. Are >> there other entries in the Solr log with more detailed information >> from the ADDREPLICA attempt? >> >> Thanks, >> Shawn >> > > > -- > Thanks & Regards, > Yaswanth Kumar Konathala. > yaswanth...@gmail.com > -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
Re: Solr cloud backup/restore not working
Can someone please guide me on where can I get more detailed error of the above exception while doing restore?? All that I see in solr.log was pasted above Thanks, On Tue, Jun 16, 2020 at 10:44 AM yaswanth kumar wrote: > I don't see anything related in the solr.log file for the same error. Not > sure if there is anyother place where I can check for this. > > Thanks, > > On Tue, Jun 16, 2020 at 10:21 AM Shawn Heisey wrote: > >> On 6/12/2020 8:38 AM, yaswanth kumar wrote: >> > Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each >> > collection) >> > Enabled basic authentication and gave all access to the admin user >> > >> > Now trying to use solr cloud backup/restore API, backup is working >> great, >> > but when trying to invoke restore API its throwing the below error >> >> > "msg":"ADDREPLICA failed to create replica", >> > "trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to >> > create replica\n\tat >> > >> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat >> > >> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat >> >> The underlying cause of this exception is not recorded here. Are there >> other entries in the Solr log with more detailed information from the >> ADDREPLICA attempt? >> >> Thanks, >> Shawn >> > > > -- > Thanks & Regards, > Yaswanth Kumar Konathala. > yaswanth...@gmail.com > -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: Solr cloud backup/restore not working
I don't see anything related in the solr.log file for the same error. Not sure if there is anyother place where I can check for this. Thanks, On Tue, Jun 16, 2020 at 10:21 AM Shawn Heisey wrote: > On 6/12/2020 8:38 AM, yaswanth kumar wrote: > > Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each > > collection) > > Enabled basic authentication and gave all access to the admin user > > > > Now trying to use solr cloud backup/restore API, backup is working great, > > but when trying to invoke restore API its throwing the below error > > > "msg":"ADDREPLICA failed to create replica", > > "trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to > > create replica\n\tat > > > org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat > > > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat > > The underlying cause of this exception is not recorded here. Are there > other entries in the Solr log with more detailed information from the > ADDREPLICA attempt? > > Thanks, > Shawn > -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: Solr cloud backup/restore not working
On 6/12/2020 8:38 AM, yaswanth kumar wrote: Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each collection) Enabled basic authentication and gave all access to the admin user Now trying to use solr cloud backup/restore API, backup is working great, but when trying to invoke restore API its throwing the below error "msg":"ADDREPLICA failed to create replica", "trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to create replica\n\tat org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat The underlying cause of this exception is not recorded here. Are there other entries in the Solr log with more detailed information from the ADDREPLICA attempt? Thanks, Shawn
Re: Solr cloud backup/restore not working
Sure I pasted it below from the solr logfiles.. 2020-06-16 14:06:27.000 INFO (qtp1987693491-153) [c:test ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=test&action=RESTORE&location=/opt/$ 2020-06-16 14:06:27.001 ERROR (qtp1987693491-153) [c:test ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: ADDREPLICA failed to create replica at org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53) at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280) at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199) at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base/java.lang.Thread.run(Thread.java:834) Can you please review and let me know if I am missing something?? On Tue, Jun 16, 2020 at 3:15 AM Jörn Franke wrote: > Have you looked in the Solr logfiles? > > > Am 16.06.2020 um 05:46 schrieb yaswanth kumar : > > > > Can anyone here help on the posted question pls?? > > > >> On Fri, Jun 12, 2020 at 10:38 AM yaswanth kumar > >> wrote:
Re: Solr cloud backup/restore not working
Have you looked in the Solr logfiles? > Am 16.06.2020 um 05:46 schrieb yaswanth kumar : > > Can anyone here help on the posted question pls?? > >> On Fri, Jun 12, 2020 at 10:38 AM yaswanth kumar >> wrote: >> >> Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each >> collection) >> Enabled basic authentication and gave all access to the admin user >> >> Now trying to use solr cloud backup/restore API, backup is working great, >> but when trying to invoke restore API its throwing the below error >> >> { >> "responseHeader":{ >>"status":500, >>"QTime":349}, >> "Operation restore caused >> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: >> ADDREPLICA failed to create replica", >> "exception":{ >>"msg":"ADDREPLICA failed to create replica", >>"rspCode":500}, >> "error":{ >>"metadata":[ >> "error-class","org.apache.solr.common.SolrException", >> "root-error-class","org.apache.solr.common.SolrException"], >>"msg":"ADDREPLICA failed to create replica", >>"trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to >> create replica\n\tat >> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat >> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat >> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)\n\tat >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat >> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)\n\tat >> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)\n\tat >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)\n\tat >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat >> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat >> org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat >> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat >> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)\n\tat >> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)\n\tat >> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)\n\tat >> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat >> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat >> org.ecl
Re: Solr cloud backup/restore not working
Can anyone here help on the posted question pls?? On Fri, Jun 12, 2020 at 10:38 AM yaswanth kumar wrote: > Using solr 8.2.0 and setup a cloud with 2 nodes. (2 replica's for each > collection) > Enabled basic authentication and gave all access to the admin user > > Now trying to use solr cloud backup/restore API, backup is working great, > but when trying to invoke restore API its throwing the below error > > { > "responseHeader":{ > "status":500, > "QTime":349}, > "Operation restore caused > exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > ADDREPLICA failed to create replica", > "exception":{ > "msg":"ADDREPLICA failed to create replica", > "rspCode":500}, > "error":{ > "metadata":[ > "error-class","org.apache.solr.common.SolrException", > "root-error-class","org.apache.solr.common.SolrException"], > "msg":"ADDREPLICA failed to create replica", > "trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to > create replica\n\tat > org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)\n\tat > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)\n\tat > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)\n\tat > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)\n\tat > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat > org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat > org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)\n\tat > org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)\n\tat > org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)\n\tat > org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat > org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Que