[
https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347834#comment-14347834
]
Shawn Heisey commented on SOLR-7191:
------------------------------------
Regarding jute.maxbuffer: The new stateFormat ensures that the clusterstate
won't break the default 1MB limit.
The giant queue I encountered was about 850000 entries, and resulted in a
packet length of a little over 14 megabytes. If I divide 850000 by 14, I know
that I can have about 60000 overseer queue entries in one znode before
jute.maxbuffer needs to be increased.
There are a couple of possibilities for managing really large overseer queues
within the default buffer size. One is to throttle the creation of new
overseer entries when the number of existing entries exceeds a certain
threshold ... I'm thinking 32768 ... so that hopefully the overseer can catch
up. Another is to change the structure of the queue so that it creates new
nodes under /overseer/queue and then puts actual entries inside those nodes,
limiting the number of queue entries in each one to 32768. With 32768 nodes
that each have 32768 entries, the queue could easily reach one billion entries
without breaking the buffer... although a queue that size might take days or
weeks to process.
A similar problem would exist with the /collections node ... so a limit of
32768 collections would also be prudent. I expect that users will run into
other problems (number of processes being the one that comes to mind) long
before they reach 32768 collections, though.
> Improve stability and startup performance of SolrCloud with thousands of
> collections
> ------------------------------------------------------------------------------------
>
> Key: SOLR-7191
> URL: https://issues.apache.org/jira/browse/SOLR-7191
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 5.0
> Reporter: Shawn Heisey
> Labels: performance, scalability
>
> A user on the mailing list with thousands of collections (5000 on 4.10.3,
> 4000 on 5.0) is having severe problems with getting Solr to restart.
> I tried as hard as I could to duplicate the user setup, but I ran into many
> problems myself even before I was able to get 4000 collections created on a
> 5.0 example cloud setup. Restarting Solr takes a very long time, and it is
> not very stable once it's up and running.
> This kind of setup is very much pushing the envelope on SolrCloud performance
> and scalability. It doesn't help that I'm running both Solr nodes on one
> machine (I started with 'bin/solr -e cloud') and that ZK is embedded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
