[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703549#comment-14703549 ]
Shalin Shekhar Mangar commented on SOLR-6760: --------------------------------------------- Thanks! Looks good to me. I'll commit this tomorrow morning my time. > New optimized DistributedQueue implementation for overseer > ---------------------------------------------------------- > > Key: SOLR-6760 > URL: https://issues.apache.org/jira/browse/SOLR-6760 > Project: Solr > Issue Type: Bug > Reporter: Noble Paul > Assignee: Shalin Shekhar Mangar > Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, > SOLR-6760.patch, deadlock.patch > > > Currently the DQ works as follows > * read all items in the directory > * sort them all > * take the head and return it and discard everything else > * rinse and repeat > This works well when we have only a handful of items in the Queue. If the > items in the queue is much larger (in tens of thousands) , this is > counterproductive > As the overseer queue is a multiple producers + single consumer queue, We can > read them all in bulk and before processing each item , just do a > zk.exists(itemname) and if all is well we don't need to do the fetch all + > sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org