jstaf commented on issue #653: URL: https://github.com/apache/solr-operator/issues/653#issuecomment-2731064995
As a workaround, you can manually add `/____v2/cluster/replicas/balance` in the security screen to the `k8s-oper` and `admin` users on a new cluster. However just a warning that rebalancing doesn't seem to work very well so you may just want to leave it off... it seems to just create a whole bunch of replicas then time out, then leaves a bunch of replicas on the new nodes without deleting the ones on the old node. After failing, it runs rebalance again, then instantly declares it was a "success" even though the replicas never got moved properly and now you have more replicas than you started with. ``` 2025-03-15 00:59:28.579 INFO (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Timed out waiting for 1 leader replicas to recover 2025-03-15 00:59:28.579 INFO (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Failed to create some replicas. Cleaning up all newly created replicas. org.apache.solr.common.SolrException: collection,shard,replica are required params " at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:86) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]" 2025-03-15 00:59:28.579 ERROR (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Operation balance_replicas failed => org.apache.solr.common.SolrException: collection,shard,replica are required params " at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:86) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.migrateReplicas(ReplicaMigrationUtils.java:217) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.BalanceReplicasCmd.call(BalanceReplicasCmd.java:81) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363) ~[solr-solrj-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]" 2025-03-15 00:59:28.579 WARN (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Error deleting replica => org.apache.solr.common.SolrException: collection,shard,replica are required params " at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196)" " at org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.migrateReplicas(ReplicaMigrationUtils.java:217) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.BalanceReplicasCmd.call(BalanceReplicasCmd.java:81) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363) ~[solr-solrj-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]" " at java.base/java.lang.Thread.run(Unknown Source) [?:?]" " at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196)" org.apache.solr.common.SolrException: collection,shard,replica are required params " at org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]" " at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]" " at java.base/java.lang.Thread.run(Unknown Source) [?:?]" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
