[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704564#comment-14704564 ]
Ramkumar Aiyengar edited comment on SOLR-6760 at 8/20/15 9:17 AM: ------------------------------------------------------------------ In the output above, 'success' new way is 3 instead of 20k, some test bug? Or is it counting with the batching as one op? was (Author: andyetitmoves): In the output above, 'success' new way is 3 instead of 20k, some test bug? > New optimized DistributedQueue implementation for overseer > ---------------------------------------------------------- > > Key: SOLR-6760 > URL: https://issues.apache.org/jira/browse/SOLR-6760 > Project: Solr > Issue Type: Bug > Reporter: Noble Paul > Assignee: Shalin Shekhar Mangar > Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, > SOLR-6760.patch, deadlock.patch > > > Currently the DQ works as follows > * read all items in the directory > * sort them all > * take the head and return it and discard everything else > * rinse and repeat > This works well when we have only a handful of items in the Queue. If the > items in the queue is much larger (in tens of thousands) , this is > counterproductive > As the overseer queue is a multiple producers + single consumer queue, We can > read them all in bulk and before processing each item , just do a > zk.exists(itemname) and if all is well we don't need to do the fetch all + > sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org