[ https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692886#comment-16692886 ]
Noble Paul edited comment on SOLR-12993 at 11/20/18 9:10 AM: ------------------------------------------------------------- Yeah, the example had it wrong. I just corrected bq.I would think the replica / shard "state", LIR term and "leader" flags would be good candidates Yes, both are suitable candidates. We can get rid of the LIR term directory altogether or alternately we can just add this data (status, leader) to the LIR term files . That way , we don't need to create any new files was (Author: noble.paul): Yeah, the example had it wrong. I just corrected bq.I would think the replica / shard "state", LIR term and "leader" flags would be good candidates Yes, both are suitable candidates. We can get rid of the LIR term directory altogether > Split the state.json into 2. a small frequently modified data + a large > unmodified data > --------------------------------------------------------------------------------------- > > Key: SOLR-12993 > URL: https://issues.apache.org/jira/browse/SOLR-12993 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Noble Paul > Priority: Major > > This a just a proposal to minimize the ZK load and improve scalability of > very large clusters. > Every time a small state change occurs for a collection/replica the following > file needs to be updated + read * n times (where n = no of replicas for this > collection ). The proposal is to split the main file into 2. > {code} > {"gettingstarted":{ > "pullReplicas":"0", > "replicationFactor":"2", > "router":{"name":"compositeId"}, > "maxShardsPerNode":"-1", > "autoAddReplicas":"false", > "nrtReplicas":"2", > "tlogReplicas":"0", > "shards":{ > "shard1":{ > "range":"80000000-ffffffff", > > "replicas":{ > "core_node3":{ > "core":"gettingstarted_shard1_replica_n1", > "base_url":"http://10.0.0.80:8983/solr", > "node_name":"10.0.0.80:8983_solr", > "state":"active", > "type":"NRT", > "force_set_state":"false", > "leader":"true"}, > "core_node5":{ > "core":"gettingstarted_shard1_replica_n2", > "base_url":"http://10.0.0.80:7574/solr", > "node_name":"10.0.0.80:7574_solr", > > "type":"NRT", > "force_set_state":"false"}}}, > "shard2":{ > "range":"0-7fffffff", > "state":"active", > "replicas":{ > "core_node7":{ > "core":"gettingstarted_shard2_replica_n4", > "base_url":"http://10.0.0.80:7574/solr", > "node_name":"10.0.0.80:7574_solr", > > "type":"NRT", > "force_set_state":"false"}, > "core_node8":{ > "core":"gettingstarted_shard2_replica_n6", > "base_url":"http://10.0.0.80:8983/solr", > "node_name":"10.0.0.80:8983_solr", > > "type":"NRT", > "force_set_state":"false", > "leader":"true"}}}}}} > {code} > another file {{status.json}} which is frequently updated and small. > {code} > { > "shard1": { > "state": "ACTIVE", > "core_node3": {"state": "active", "leader" : true}, > "core_node5": {"state": "active"} > }, > "shard2": { > "state": "active", > "core_node7": {"state": "active"}, > "core_node8": {"state": "active", "leader" : true}} > } > {code} > Here the size of the file is roughly one tenth of the other file. This leads > to a dramatic reduction in the amount of data written/read to/from ZK. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org