Hi folks,
I was wondering if people had experimented with something other than the
default Lucene search logic of completing one segment within a Slice at a
time, and then going on to the next segment in sequential order.
Here is the current logic in IndexSearcher#search(List<LeafReaderContext>
leaves, Weight weight, Collector collector):
```
for (LeafReaderContext ctx : leaves) {
...
scorer.score(leafCollector, ctx.reader().getLiveDocs());
...
}
```
For sorted indexes (e.g. time sorted data), for a query that asks for the
top-k results, it may be better to round robin among leaves (or expand more
into the leaf that has better values) within a Slice that uses just a
single thread?
In Amazon's Product Search context, our index is sorted in descending order by
a custom Score.
I wanted to experiment with something like:
```
while (isAnyCollectorNotTerminated) { // do a round robin
for (LeafReaderContext ctx : leaves) {
...
scorer.score(leafCollector, ctx.reader().getLiveDocs(), start, end);
...
}
}
```
However, I came across two issues until now:
1. DrillSidewaysScorer has a docId limitation of start=0 and end=Integer
.MAX_VALUE
2. FacetsCollector is designed in a way that it needs a leaf to be finished
before it can go on to
the next leaf. This does not work with my logic of round-robining.
I am working through them right now.
I was wondering if people had in general tried this approach, and whether
they knew of other
problems that might arise, potential re-routes, performance numbers or any
other experiences in general?
Are there any other Collectors that have such design decisions?
I see the current code has a comment like:
```
// TODO: should we make this
// threaded...? the Collector could be sync'd?
```
so I guess there were some ideas around making this logic smarter?
Thanks for the help!
-
Gautam Worah.
