[ https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905254#comment-15905254 ]
albert vico oton commented on SOLR-5872: ---------------------------------------- Yea exactly, sorry if my comment was confusing but the problem is with the use that SolrCloud is doing of ZK not with ZK itself. {quote} When the number of collections gets large enough, Solr has a tendency to run into ZOOKEEPER-1162, because entries can be added to the overseer queue at a much faster rate than the overseer can process them. During my testing on SOLR-7191 with version 5, Solr generated an overseer queue with 850,000 entries in it, resulting in a ZK packet size of 14 megabytes. {quote} I believe that this is exactly what we were experiencing. > Eliminate overseer queue > ------------------------- > > Key: SOLR-5872 > URL: https://issues.apache.org/jira/browse/SOLR-5872 > Project: Solr > Issue Type: Improvement > Components: SolrCloud > Reporter: Noble Paul > Assignee: Noble Paul > > The overseer queue is one of the busiest points in the entire system. The > raison d'ĂȘtre of the queue is > * Provide batching of operations for the main clusterstate,json so that > state updates are minimized > * Avoid race conditions and ensure order > Now , as we move the individual collection states out of the main > clusterstate.json, the batching is not useful anymore. > Race conditions can easily be solved by using a compare and set in Zookeeper. > The proposed solution is , whenever an operation is required to be performed > on the clusterstate, the same thread (and of course the same JVM) > # read the fresh state and version of zk node > # construct the new state > # perform a compare and set > # if compare and set fails go to step 1 > This should be limited to all operations performed on external collections > because batching would be required for others -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org