[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707781#comment-14707781 ]
Gregory Chanan commented on SOLR-6760: -------------------------------------- After some more thought, I like the rename of OverseerCollectionQueue -> OverseerTaskQueue. It makes the term more generic, but makes it clear the scope should be limited to the Overseer (compared to my earlier suggestion of DistributedTaskQueue), which seems like a good idea given the limitations of the interface discussed in this issue. My latest version of the patch in SOLR-7789 makes this change. > New optimized DistributedQueue implementation for overseer > ---------------------------------------------------------- > > Key: SOLR-6760 > URL: https://issues.apache.org/jira/browse/SOLR-6760 > Project: Solr > Issue Type: Improvement > Components: SolrCloud > Reporter: Noble Paul > Assignee: Shalin Shekhar Mangar > Fix For: Trunk, 5.4 > > Attachments: SOLR-6760-branch_5x.patch, SOLR-6760.patch, > SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch > > > Currently the DQ works as follows > * read all items in the directory > * sort them all > * take the head and return it and discard everything else > * rinse and repeat > This works well when we have only a handful of items in the Queue. If the > items in the queue is much larger (in tens of thousands) , this is > counterproductive > As the overseer queue is a multiple producers + single consumer queue, We can > read them all in bulk and before processing each item , just do a > zk.exists(itemname) and if all is well we don't need to do the fetch all + > sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org