[
https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354965#comment-14354965
]
Shalin Shekhar Mangar commented on SOLR-7191:
---------------------------------------------
bq. I tested 4,000 cores on branch_5x and found better results.
Thanks, that is good to know.
bq. BTW: collection creation slows down the more collections you have in the
cloud. Starts with qtimes of ~3s; ending with ~6s. Solr 4.x was always steady
at ~3s.
Yes, that is expected with the current implementation. Each new collection
touches the /clusterstate.json as a way to inform all nodes about the existence
of a new collection (this is required for routing requests; this also happens
on collection deletion). This invokes a reload of state information for all
collections that have a core on that solr node. As the number of collections
increase (and because you have a lot of cores on the same node), the state
refresh takes longer. The Collection Create API waits for the new collection to
be visible in the cluster state and therefore takes a longer time as number of
collections increase.
The current implementation was made under the assumption that the system will
have a large number of collections but a node will only have a few replicas.
This isn't true in this case and therefore we're running into these bottlenecks.
> Improve stability and startup performance of SolrCloud with thousands of
> collections
> ------------------------------------------------------------------------------------
>
> Key: SOLR-7191
> URL: https://issues.apache.org/jira/browse/SOLR-7191
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 5.0
> Reporter: Shawn Heisey
> Labels: performance, scalability
> Attachments: lots-of-zkstatereader-updates-branch_5x.log
>
>
> A user on the mailing list with thousands of collections (5000 on 4.10.3,
> 4000 on 5.0) is having severe problems with getting Solr to restart.
> I tried as hard as I could to duplicate the user setup, but I ran into many
> problems myself even before I was able to get 4000 collections created on a
> 5.0 example cloud setup. Restarting Solr takes a very long time, and it is
> not very stable once it's up and running.
> This kind of setup is very much pushing the envelope on SolrCloud performance
> and scalability. It doesn't help that I'm running both Solr nodes on one
> machine (I started with 'bin/solr -e cloud') and that ZK is embedded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
