[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-6760: ----------------------------- Description: Currently the DQ works as follows * read all items in the directory * sort them all * take the head and return it This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again was: Currently the DQ works as follows * read all items in the directory * sort them all * take the head and return it This works well when we have only a handful of items in the Queue. If the items in the queue is much larger in tens of thousands, this is counterprodcutive As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again > New optimized DistributedQueue implementation for overseer > ---------------------------------------------------------- > > Key: SOLR-6760 > URL: https://issues.apache.org/jira/browse/SOLR-6760 > Project: Solr > Issue Type: Bug > Reporter: Noble Paul > Assignee: Noble Paul > > Currently the DQ works as follows > * read all items in the directory > * sort them all > * take the head and return it > This works well when we have only a handful of items in the Queue. If the > items in the queue is much larger (in tens of thousands) , this is > counterproductive > As the overseer queue is a multiple producers + single consumer queue, We can > read them all in bulk and before processing each item , just do a > zk.exists(itemname) and if all is well we don't need to do the fetch all + > sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org