[
https://issues.apache.org/jira/browse/SOLR-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803537#comment-13803537
]
Mark Miller commented on SOLR-5381:
-----------------------------------
bq. The first step is to store the shards information in separate nodes and
each node can just listen to the shard node it belongs to. We may also need to
split each collection into its own node and the clusterstate.json just holding
the names of the collections .
We actually used to have a very fine grained layout similar to this - it had
some advantages, but it was *very* expensive it turned out, because of having
to do so many calls to load the state. I was also never very happy with the
number watchers that we were juggling. I think we want to find the right
balance, or perhaps see if ZooKeeper has gained the ability to read multiple
nodes in a single call.
> Split Clusterstate and scale
> -----------------------------
>
> Key: SOLR-5381
> URL: https://issues.apache.org/jira/browse/SOLR-5381
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: Noble Paul
> Assignee: Noble Paul
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> clusterstate.json is a single point of contention for all components in
> SolrCloud. It would be hard to scale SolrCloud beyond a few thousand nodes
> because there are too many updates and too many nodes need to be notified of
> the changes. As the no:of nodes go up the size of clusterstate.json keeps
> going up and it will soon exceed the limit impossed by ZK.
> The first step is to store the shards information in separate nodes and each
> node can just listen to the shard node it belongs to. We may also need to
> split each collection into its own node and the clusterstate.json just
> holding the names of the collections .
> This is an umbrella issue
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
