Hello fellow SolrCloud users,Hopefully someone can point me in the right direction as I can't figure out what is wrong.
I have 4 nodes. I just finished reindexing a very large collection and now I'm trying to add replicas.
nodes 1 and 2 contain the collection and I'm trying to replicate these shards to nodes 3 and 4.
Whenever I try and add a replica via the UI I'm getting No node can satisfy the rules "[] , where the rules list is empty.
It says to look in the log file, which I do (and is attached), and I see the autoscale diagnostics report followed up by the exception.
The violations value in the diagnostics output is empty.Originally the cluster rules were blank, but I changed them to the what is indicated in the autoscaling-diagnostics.json attachment, after getting this error the first time and seeing the blank'[]' rule set.
I'm a bit perplexed as to the actual issue and how to fix it.Is there possible a collection level setting that needs resetting? This is a new collection.
Solr 7.7.3 -Mike/NewsRx
autoscaling-diagnostics.json
Description: application/json
2021-11-05 13:31:57.878 ERROR (OverseerThreadFactory-33-thread-5-processing-n:solr-0003:8983_solr) [c:weeklies_2021-10-25_7_7_3 s:shard2 ] o.a.s.c.s.c.a.PolicyHelper errorId : AutoScaling.error.diagnostics.1797151067707752 {
"sortedNodes":[{
"node":"solr-0001:8983_solr",
"isLive":true,
"cores":4.0,
"freedisk":213.41428756713867,
"sysLoadAvg":54.0,
"totaldisk":1151.609375,
"replicas":{
"alldata":{"shard1":[{
"core_node62":{
"core":"alldata_shard1_replica_n61",
"shard":"shard1",
"collection":"alldata",
"node_name":"solr-0001:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0001:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":174.70748720970005}}]},
"weeklies_2021-10-25_7_7_3":{"shard2":[{
"core_node7":{
"core":"weeklies_2021-10-25_7_7_3_shard2_replica_n4",
"shard":"shard2",
"collection":"weeklies_2021-10-25_7_7_3",
"node_name":"solr-0001:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0001:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":670.2301744800061}}]},
"dailies_2021-1":{"shard2":[{
"core_node30":{
"core":"dailies_2021-1_shard2_replica_n29",
"shard":"shard2",
"collection":"dailies_2021-1",
"node_name":"solr-0001:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0001:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":61.86608923971653}}]},
"clinicaltrials_20211014":{"shard1":[{
"core_node36":{
"core":"clinicaltrials_20211014_shard1_replica_n35",
"shard":"shard1",
"collection":"clinicaltrials_20211014",
"node_name":"solr-0001:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0001:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":8.83115712646395}}]}}}
,{
"node":"solr-0002:8983_solr",
"isLive":true,
"cores":4.0,
"freedisk":199.5399932861328,
"sysLoadAvg":15.0,
"totaldisk":1151.609375,
"replicas":{
"alldata":{"shard1":[{
"core_node64":{
"core":"alldata_shard1_replica_n63",
"shard":"shard1",
"collection":"alldata",
"node_name":"solr-0002:8983_solr",
"type":"NRT",
"base_url":"http://solr-0002:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":174.71872491855174}}]},
"weeklies_2021-10-25_7_7_3":{
"shard2":[],
"shard1":[{
"core_node3":{
"core":"weeklies_2021-10-25_7_7_3_shard1_replica_n1",
"shard":"shard1",
"collection":"weeklies_2021-10-25_7_7_3",
"node_name":"solr-0002:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0002:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":669.8700903812423}}]},
"dailies_2021-1":{"shard2":[{
"core_node22":{
"core":"dailies_2021-1_shard2_replica_n21",
"shard":"shard2",
"collection":"dailies_2021-1",
"node_name":"solr-0002:8983_solr",
"type":"NRT",
"base_url":"http://solr-0002:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":61.86608923971653}}]},
"clinicaltrials_20211014":{"shard1":[{
"core_node28":{
"core":"clinicaltrials_20211014_shard1_replica_n27",
"shard":"shard1",
"collection":"clinicaltrials_20211014",
"node_name":"solr-0002:8983_solr",
"type":"NRT",
"base_url":"http://solr-0002:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":8.831211734563112}}]}}}
,{
"node":"solr-0004:8983_solr",
"isLive":true,
"cores":3.0,
"freedisk":965.3958778381348,
"sysLoadAvg":8.0,
"totaldisk":1151.609375,
"replicas":{
"alldata":{"shard2":[{
"solr-0004.alldata":{
"core":"alldata",
"shard":"shard2",
"collection":"alldata",
"node_name":"solr-0004:8983_solr",
"type":"NRT",
"base_url":"http://solr-0004:8983/solr";,
"state":"active",
"force_set_state":"false"}}]},
"weeklies_2021-10-25_7_7_3":{"shard2":[]},
"dailies_2021-1":{"shard1":[{
"core_node24":{
"core":"dailies_2021-1_shard1_replica_n23",
"shard":"shard1",
"collection":"dailies_2021-1",
"node_name":"solr-0004:8983_solr",
"type":"NRT",
"base_url":"http://solr-0004:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":0.5156466346234083}}]},
"clinicaltrials_20211014":{"shard2":[{
"core_node24":{
"core":"clinicaltrials_20211014_shard2_replica_n23",
"shard":"shard2",
"collection":"clinicaltrials_20211014",
"node_name":"solr-0004:8983_solr",
"type":"NRT",
"base_url":"http://solr-0004:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":9.176561694592237}}]}}}
,{
"node":"solr-0003:8983_solr",
"isLive":true,
"cores":3.0,
"freedisk":942.5447769165039,
"sysLoadAvg":1.0,
"totaldisk":1151.609375,
"replicas":{
"alldata":{"shard2":[{
"solr-0003.alldata":{
"core":"alldata",
"shard":"shard2",
"collection":"alldata",
"node_name":"solr-0003:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0003:8983/solr";,
"state":"active",
"force_set_state":"false"}}]},
"weeklies_2021-10-25_7_7_3":{"shard2":[]},
"dailies_2021-1":{"shard1":[{
"core_node3":{
"core":"dailies_2021-1_shard1_replica_n1",
"shard":"shard1",
"collection":"dailies_2021-1",
"node_name":"solr-0003:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0003:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":0.46995995100587606}}]},
"clinicaltrials_20211014":{"shard2":[{
"core_node22":{
"core":"clinicaltrials_20211014_shard2_replica_n21",
"shard":"shard2",
"collection":"clinicaltrials_20211014",
"node_name":"solr-0003:8983_solr",
"type":"NRT",
"leader":"true",
"base_url":"http://solr-0003:8983/solr";,
"state":"active",
"force_set_state":"false",
"INDEX.sizeInGB":9.17459019087255}}]}}}],
"liveNodes":["solr-0001:8983_solr",
"solr-0002:8983_solr",
"solr-0003:8983_solr",
"solr-0004:8983_solr"],
"violations":[],
"config":{
"cluster-preferences":[{
"minimize":"cores",
"precision":2}
,{
"maximize":"freedisk",
"precision":100}
,{
"minimize":"sysLoadAvg",
"precision":10}]}}
2021-11-05 13:31:57.879 ERROR (OverseerThreadFactory-33-thread-5-processing-n:solr-0003:8983_solr) [c:weeklies_2021-10-25_7_7_3 s:shard2 ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: weeklies_2021-10-25_7_7_3 operation: addreplica failed:org.apache.solr.cloud.api.collections.Assign$AssignmentException: Error getting replica locations : No node can satisfy the rules "[] More details from logs in node : solr-0003:8983_solr, errorId : AutoScaling.error.diagnostics.1797151067707752"
at org.apache.solr.cloud.api.collections.Assign.getPositionsUsingPolicy(Assign.java:394)
at org.apache.solr.cloud.api.collections.Assign$PolicyBasedAssignStrategy.assign(Assign.java:630)
at org.apache.solr.cloud.api.collections.Assign.getNodesForNewReplicas(Assign.java:368)
at org.apache.solr.cloud.api.collections.AddReplicaCmd.buildReplicaPositions(AddReplicaCmd.java:360)
at org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:146)
at org.apache.solr.cloud.api.collections.AddReplicaCmd.call(AddReplicaCmd.java:91)
at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:294)
at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:505)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: No node can satisfy the rules "[] More details from logs in node : solr-0003:8983_solr, errorId : AutoScaling.error.diagnostics.1797151067707752"
at org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getReplicaLocations(PolicyHelper.java:179)
at org.apache.solr.cloud.api.collections.Assign.getPositionsUsingPolicy(Assign.java:382)
... 11 more
