Is there still a valid reason to keep the inactive shards around? If shard splitting is robust, could not the split operation delete the inactive shard once the new shards are successfully loaded, just like what happens during an automated merge of segments?
-- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 18. jun. 2018 kl. 12:12 skrev Andrzej Białecki > <andrzej.biale...@lucidworks.com>: > > If I’m not mistaken the weird accounting of “inactive” shard cores is caused > also by the fact that individual cores that constitute replicas in the > inactive shard are still loaded, so they still affect the number of active > cores. If that’s the case then we should probably fix this to prevent loading > the cores from inactive (but still present) shards. > >> On 14 Jun 2018, at 04:27, Shalin Shekhar Mangar <shalinman...@gmail.com> >> wrote: >> >> Yes, I believe Noble is working on this. See >> https://issues.apache.org/jira/browse/SOLR-11985 >> >> On Wed, Jun 13, 2018 at 1:35 PM Jan Høydahl <jan....@cominvent.com> wrote: >> >>> Ok, get the meaning of preferences. >>> >>> Would there be a way to write a generic rule that would suggest moving >>> shards to obtain balance, without specifying absolute core counts? I.e. if >>> you have three nodes >>> A: 3 cores >>> B: 5 cores >>> C: 3 cores >>> >>> Then that rule would suggest two moves to end up with 4 cores on all three >>> (unless that would violate disk space or load limits)? >>> >>> -- >>> Jan Høydahl, search solution architect >>> Cominvent AS - www.cominvent.com >>> >>>> 12. jun. 2018 kl. 08:10 skrev Shalin Shekhar Mangar < >>> shalinman...@gmail.com>: >>>> >>>> Hi Jan, >>>> >>>> Comments inline: >>>> >>>> On Tue, Jun 12, 2018 at 2:19 AM Jan Høydahl <jan....@cominvent.com >>> <mailto:jan....@cominvent.com>> wrote: >>>> >>>>> Hi >>>>> >>>>> I'm trying to have Autoscaling move a shard to another node after >>> manually >>>>> splitting. >>>>> We have two nodes, one has a shard1 and the other node is empty. >>>>> >>>>> After SPLITSHARD you have >>>>> >>>>> * shard1 (inactive) >>>>> * shard1_0 >>>>> * shard1_1 >>>>> >>>>> For autoscaling we have the {"minimize" : "cores"} cluster preference >>>>> active. Because of that I'd expect that Autoscaling would suggest to >>> move >>>>> e.g. shard1_1 to the other (empty) node, but it doesn't. Then I create a >>>>> rule just to test {"cores": "<2", "node": "#ANY"}, but still no >>>>> suggestions. Not until I delete the inactive shard1, then it suggests to >>>>> move one of the two remaining shards to the other node. >>>>> >>>>> So my two questions are >>>>> 1. Is it by design that inactive shards "count" wrt #cores? >>>>> I understand that it consumes disk but it is not active otherwise, >>>>> so one could argue that it should not be counted in core/replica >>> rules? >>>>> >>>> >>>> Today, inactive slices also count towards the number of cores -- though >>>> technically correct, it is probably an oversight. >>>> >>>> >>>>> 2. Why is there no suggestion to move a shard due to the "minimize >>> cores" >>>>> reference itself? >>>>> >>>> >>>> The /autoscaling/suggestions end point only suggests if there are policy >>>> violations. Preferences such as minimize:cores are more of a sorting >>> order >>>> so they aren't really being violated. After you add the rule, the >>> framework >>>> still cannot give a suggestion that satisfies your rule. This is because >>>> even if shard1_1 is moved to node2, node1 still has shard1 and shard1_0. >>> So >>>> the system ends up not suggesting anything. You should get a suggestion >>> if >>>> you add a third node to the cluster though. >>>> >>>> Also see SOLR-11997 <https://issues.apache.org/jira/browse/SOLR-11997 < >>> https://issues.apache.org/jira/browse/SOLR-11997>> which >>>> will tell users that a suggestion could not be returned because we cannot >>>> satisfy the policy. There are a slew of other improvements to suggestions >>>> planned that will return suggestions even when there are no policy >>>> violations. >>>> >>>> >>>>> >>>>> -- >>>>> Jan Høydahl, search solution architect >>>>> Cominvent AS - www.cominvent.com <http://www.cominvent.com/> >>>>> >>>>> >>>> >>>> -- >>>> Regards, >>>> Shalin Shekhar Mangar. >>> >>> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >
