[ https://issues.apache.org/jira/browse/LUCENE-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051640#comment-17051640 ]
Michael Sokolov commented on LUCENE-8929: ----------------------------------------- OK, hm I now believe the results posted above are somewhat suspect since `luceneutil` had a bug when measuring timing of multiple tasks under concurrent search. Since the use of multiple threads enables the test harness to consume all system resources, there can be contention among multiple tasks run concurrently, leading to starvation of the fast ones. I pushed a patch that works around this issue in luceneutil and did a new round of measurements. I also updated the PR (opened a new PR actually- #1316, since it had been so long it was hard to re-use the old one). Updates since the previous PR include: # Added tiebreak by docid to ensure we collect the right hits (there doesn't seem to be a test for this? Probably we should add one) # No longer terminate by max min scores; the impact is largely superseded by this change. I measured having both and it seemed to hurt in most cases. Possibly we should terminate based on max-min score when 0 < N < 100, and using this approach (min-min score) for N > 100 # This now bounds the thread count used to determine update frquency by the number of slices # I added some paranoid protections against a race condition that probably didn't exist, but I was going crazy with inconsistent results and this seemed like a good idea at the time.Might want to revisit this and remove some unneeded code in MaxScoreTerminator. Here's a better description of what this PR is: Performs early termination in TopFieldCollector (when query sort matches index sort) based on the minimum minimum score across all leaf collectos, providing nice gains for queries with large N. Once we have globally collected the number of hits required to satisfy the query (which is max(N, 1000)) then the minimum minimum score across threads is a global lower bound on the score that must be met by any hit. We already have sufficient hits to satisfy the query with score better than that, so any later hit with a worse score will be discarded, and any Collector retrieving such a hit can be terminated. Note that this is a looser bound than the maximum minimum computed in LUCENE-8978, but that bound can only be applied once a collector has collected N hits and all collectors together have collected 1000. When N is relatively large, this weaker bound often applies before N hits have been collected. Further, once a collector terminates, the same logic applies to the remaining collectors, which can result in raising the bound further. More precisely, the termination bound used is the minimum minimum score among the top collectors (ranked by their minimum scores) that together have at least max(N, 1000) hits. h2. N=1000 ||Task|| QPS before|| StdDev|| QPS after|| StdDev|| Pct diff|| |LowTermDayOfYearSort| 216.94| (0.3%)| 549.10| (15.9%)| 153.1% ( 136% - 169%)| |HighTermDayOfYearSort| 277.36| (0.9%)| 1107.12| (25.9%)| 299.2% ( 269% - 328%)| h2. N=500 ||Task|| QPS before|| StdDev|| QPS after|| StdDev|| Pct diff|| |LowTermDayOfYearSort| 385.47| (5.4%)| 649.01| (4.9%) | 68.4% ( 55% - 83%)| |HighTermDayOfYearSort| 499.03| (6.5%)| 1133.98| (27.5%) |127.2% ( 87% - 172%)| h2. N=100 ||Task|| QPS before|| StdDev|| QPS after|| StdDev|| Pct diff|| |LowTermDayOfYearSort| 578.19| (1.4%)| 589.86| (2.0%)|2.0% ( -1% - 5%)| |HighTermDayOfYearSort| 1444.25| (1.8%)| 1508.98| (9.6%)|4.5% ( -6% - 16%)| h2. N=20 ||Task|| QPS before|| StdDev|| QPS after|| StdDev|| Pct diff|| |HighTermDayOfYearSort| 1857.14 |(2.4%)|1766.53 |(2.0%)|-4.9% ( -9% - 0%)| |LowTermDayOfYearSort| 628.71| (2.3%)| 620.24| (1.6%) |-1.3% ( -5% - 2%)| > Early Terminating CollectorManager > ---------------------------------- > > Key: LUCENE-8929 > URL: https://issues.apache.org/jira/browse/LUCENE-8929 > Project: Lucene - Core > Issue Type: Sub-task > Reporter: Atri Sharma > Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > We should have an early terminating collector manager which accurately tracks > hits across all of its collectors and determines when there are enough hits, > allowing all the collectors to abort. > The options for the same are: > 1) Shared total count : Global "scoreboard" where all collectors update their > current hit count. At the end of each document's collection, collector checks > if N > threshold, and aborts if true > 2) State Reporting Collectors: Collectors report their total number of counts > collected periodically using a callback mechanism, and get a proceed or abort > decision. > 1) has the overhead of synchronization in the hot path, 2) can collect > unnecessary hits before aborting. > I am planning to work on 2), unless objections -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org