[jira] [Comment Edited] (SOLR-12993) Split the state.json into 2. a small frequently modified data + a large unmodified data
[
https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819976#comment-16819976
]
mosh edited comment on SOLR-12993 at 4/17/19 11:05 AM:
---
{quote}or alternately we can just add this data (status, leader) to the LIR
term files . That way , we don't need to create any new files
{quote}
ZkShardTerms(class that generates LIR files) resides in solr-core, while
ZkStateReader is in solrJ.
Since this proposal is to split state.json, there would be no way to find out
which replica is the leader,
since this information will reside inside the LIR term files.
I propose two possible forms of action:
# Move ZkShardTerms to solrJ, combining LIR terms, shard state status and
leader.
# Create new files as proposed by [~noble.paul], which will contain a small
subset of the split information.
[~noble.paul], [~gus_heck],
WDYT?
was (Author: moshebla):
{quote}or alternately we can just add this data (status, leader) to the LIR
term files . That way , we don't need to create any new files
{quote}
ZkShardTerms(class that generates LIR files) resides in solr-core, while
ZkStateReader is in solrJ.
Since this proposal is to split state.json, there would be no way to find out
which replica is the leader,
since this information will reside inside the LIR term files.
I propose two possible forms of action:
# Move ZkShardTerms to solrJ, and combine LIR terms
# Create new files as proposed by [~noble.paul], which will contain a small
subset of the split information.
[~noble.paul], [~gus_heck],
WDYT?
> Split the state.json into 2. a small frequently modified data + a large
> unmodified data
> ---
>
> Key: SOLR-12993
> URL: https://issues.apache.org/jira/browse/SOLR-12993
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
>
> This a just a proposal to minimize the ZK load and improve scalability of
> very large clusters.
> Every time a small state change occurs for a collection/replica the following
> file needs to be updated + read * n times (where n = no of replicas for this
> collection ). The proposal is to split the main file into 2.
> {code}
> {"gettingstarted":{
> "pullReplicas":"0",
> "replicationFactor":"2",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"2",
> "tlogReplicas":"0",
> "shards":{
> "shard1":{
> "range":"8000-",
>
> "replicas":{
> "core_node3":{
> "core":"gettingstarted_shard1_replica_n1",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"gettingstarted_shard1_replica_n2",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard2":{
> "range":"0-7fff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"gettingstarted_shard2_replica_n4",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"},
> "core_node8":{
> "core":"gettingstarted_shard2_replica_n6",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
>
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}
> {code}
> another file {{status.json}} which is frequently updated and small.
> {code}
> {
> "shard1": {
> "state": "ACTIVE",
> "core_node3": {"state": "active", "leader" : true},
> "core_node5": {"state": "active"}
> },
> "shard2": {
> "state": "active",
> "core_node7": {"state": "active"},
> "core_node8": {"state": "active", "leader" : true}}
> }
> {code}
> Here the size of the file is roughly one tenth of the other file. This leads
> to a dramatic reduction in the amount of data written/read to/from ZK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12993) Split the state.json into 2. a small frequently modified data + a large unmodified data
[
https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784973#comment-16784973
]
Noble Paul edited comment on SOLR-12993 at 3/5/19 10:17 PM:
We do that because of the explosion of the no:of watchers . large no:of
watchers are error prone. We don't know if a particular node is not in sync and
the cost of watching so many nodes
was (Author: noble.paul):
We do that because of the explosion of the no:of watchers . large no:of
watchers are error prone. We don't know if a particular node is not in sync or
the cost of watching so many nodes
> Split the state.json into 2. a small frequently modified data + a large
> unmodified data
> ---
>
> Key: SOLR-12993
> URL: https://issues.apache.org/jira/browse/SOLR-12993
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
>
> This a just a proposal to minimize the ZK load and improve scalability of
> very large clusters.
> Every time a small state change occurs for a collection/replica the following
> file needs to be updated + read * n times (where n = no of replicas for this
> collection ). The proposal is to split the main file into 2.
> {code}
> {"gettingstarted":{
> "pullReplicas":"0",
> "replicationFactor":"2",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"2",
> "tlogReplicas":"0",
> "shards":{
> "shard1":{
> "range":"8000-",
>
> "replicas":{
> "core_node3":{
> "core":"gettingstarted_shard1_replica_n1",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"gettingstarted_shard1_replica_n2",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard2":{
> "range":"0-7fff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"gettingstarted_shard2_replica_n4",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"},
> "core_node8":{
> "core":"gettingstarted_shard2_replica_n6",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
>
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}
> {code}
> another file {{status.json}} which is frequently updated and small.
> {code}
> {
> "shard1": {
> "state": "ACTIVE",
> "core_node3": {"state": "active", "leader" : true},
> "core_node5": {"state": "active"}
> },
> "shard2": {
> "state": "active",
> "core_node7": {"state": "active"},
> "core_node8": {"state": "active", "leader" : true}}
> }
> {code}
> Here the size of the file is roughly one tenth of the other file. This leads
> to a dramatic reduction in the amount of data written/read to/from ZK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12993) Split the state.json into 2. a small frequently modified data + a large unmodified data
[
https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692886#comment-16692886
]
Noble Paul edited comment on SOLR-12993 at 11/20/18 9:10 AM:
-
Yeah, the example had it wrong. I just corrected
bq.I would think the replica / shard "state", LIR term and "leader" flags would
be good candidates
Yes, both are suitable candidates. We can get rid of the LIR term directory
altogether
or alternately we can just add this data (status, leader) to the LIR term files
. That way , we don't need to create any new files
was (Author: noble.paul):
Yeah, the example had it wrong. I just corrected
bq.I would think the replica / shard "state", LIR term and "leader" flags would
be good candidates
Yes, both are suitable candidates. We can get rid of the LIR term directory
altogether
> Split the state.json into 2. a small frequently modified data + a large
> unmodified data
> ---
>
> Key: SOLR-12993
> URL: https://issues.apache.org/jira/browse/SOLR-12993
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
>
> This a just a proposal to minimize the ZK load and improve scalability of
> very large clusters.
> Every time a small state change occurs for a collection/replica the following
> file needs to be updated + read * n times (where n = no of replicas for this
> collection ). The proposal is to split the main file into 2.
> {code}
> {"gettingstarted":{
> "pullReplicas":"0",
> "replicationFactor":"2",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"2",
> "tlogReplicas":"0",
> "shards":{
> "shard1":{
> "range":"8000-",
>
> "replicas":{
> "core_node3":{
> "core":"gettingstarted_shard1_replica_n1",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"gettingstarted_shard1_replica_n2",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard2":{
> "range":"0-7fff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"gettingstarted_shard2_replica_n4",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"},
> "core_node8":{
> "core":"gettingstarted_shard2_replica_n6",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
>
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}
> {code}
> another file {{status.json}} which is frequently updated and small.
> {code}
> {
> "shard1": {
> "state": "ACTIVE",
> "core_node3": {"state": "active", "leader" : true},
> "core_node5": {"state": "active"}
> },
> "shard2": {
> "state": "active",
> "core_node7": {"state": "active"},
> "core_node8": {"state": "active", "leader" : true}}
> }
> {code}
> Here the size of the file is roughly one tenth of the other file. This leads
> to a dramatic reduction in the amount of data written/read to/from ZK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12993) Split the state.json into 2. a small frequently modified data + a large unmodified data
[
https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692886#comment-16692886
]
Noble Paul edited comment on SOLR-12993 at 11/20/18 9:08 AM:
-
Yeah, the example had it wrong. I just corrected
bq.I would think the replica / shard "state", LIR term and "leader" flags would
be good candidates
Yes, both are suitable candidates. We can get rid of the LIR term directory
altogether
was (Author: noble.paul):
Yeah, the example had it wrong. I just corrected
bq.I would think the replica / shard "state", LIR term and "leader" flags would
be good candidates
Yes, both are suitable candidates
> Split the state.json into 2. a small frequently modified data + a large
> unmodified data
> ---
>
> Key: SOLR-12993
> URL: https://issues.apache.org/jira/browse/SOLR-12993
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Noble Paul
>Priority: Major
>
> This a just a proposal to minimize the ZK load and improve scalability of
> very large clusters.
> Every time a small state change occurs for a collection/replica the following
> file needs to be updated + read * n times (where n = no of replicas for this
> collection ). The proposal is to split the main file into 2.
> {code}
> {"gettingstarted":{
> "pullReplicas":"0",
> "replicationFactor":"2",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"2",
> "tlogReplicas":"0",
> "shards":{
> "shard1":{
> "range":"8000-",
>
> "replicas":{
> "core_node3":{
> "core":"gettingstarted_shard1_replica_n1",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"gettingstarted_shard1_replica_n2",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard2":{
> "range":"0-7fff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"gettingstarted_shard2_replica_n4",
> "base_url":"http://10.0.0.80:7574/solr;,
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"},
> "core_node8":{
> "core":"gettingstarted_shard2_replica_n6",
> "base_url":"http://10.0.0.80:8983/solr;,
> "node_name":"10.0.0.80:8983_solr",
>
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}
> {code}
> another file {{status.json}} which is frequently updated and small.
> {code}
> {
> "shard1": {
> "state": "ACTIVE",
> "core_node3": {"state": "active", "leader" : true},
> "core_node5": {"state": "active"}
> },
> "shard2": {
> "state": "active",
> "core_node7": {"state": "active"},
> "core_node8": {"state": "active", "leader" : true}}
> }
> {code}
> Here the size of the file is roughly one tenth of the other file. This leads
> to a dramatic reduction in the amount of data written/read to/from ZK.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
