vsop-479 commented on PR #11888:
URL: https://github.com/apache/lucene/pull/11888#issuecomment-1759050886
Append some performance data. Note that the results have quite diversity
from different rounds.
# round1
Task QPS baseline StdDev QPS bsearch
StdDev Pct diff p-value
BrowseRandomLabelTaxoFacets 2601.88 (5.0%) 2437.93
(11.1%) -6.3% ( -21% - 10%) 0.021
BrowseRandomLabelSSDVFacets 1796.93 (7.1%) 1696.76
(8.0%) -5.6% ( -19% - 10%) 0.019
MedPhrase 4021.51 (6.7%) 3813.81
(7.6%) -5.2% ( -18% - 9%) 0.022
HighPhrase 821.97 (7.3%) 788.46
(6.3%) -4.1% ( -16% - 10%) 0.059
HighTerm 6221.04 (7.3%) 6032.11
(5.6%) -3.0% ( -14% - 10%) 0.138
Fuzzy2 276.46 (6.0%) 268.79
(4.8%) -2.8% ( -12% - 8%) 0.105
Respell 707.62 (5.9%) 692.48
(4.9%) -2.1% ( -12% - 9%) 0.211
BrowseDayOfYearSSDVFacets 6517.91 (5.4%) 6392.55
(6.0%) -1.9% ( -12% - 10%) 0.287
BrowseDateSSDVFacets 2195.69 (19.2%) 2155.70
(9.7%) -1.8% ( -25% - 33%) 0.705
BrowseMonthSSDVFacets 6724.35 (6.7%) 6606.24
(6.1%) -1.8% ( -13% - 11%) 0.386
Prefix3 3023.99 (4.5%) 2974.60
(5.3%) -1.6% ( -10% - 8%) 0.293
Fuzzy1 886.09 (5.1%) 877.56
(6.0%) -1.0% ( -11% - 10%) 0.587
PKLookup 396.87 (15.8%) 393.83
(19.2%) -0.8% ( -30% - 40%) 0.890
OrHighLow 3242.80 (6.0%) 3223.53
(7.2%) -0.6% ( -13% - 13%) 0.778
BrowseMonthTaxoFacets 5874.67 (6.1%) 5856.79
(5.1%) -0.3% ( -10% - 11%) 0.865
IntNRQ 2799.54 (4.7%) 2792.69
(5.4%) -0.2% ( -9% - 10%) 0.878
LowIntervalsOrdered 1336.37 (8.1%) 1333.20
(5.6%) -0.2% ( -12% - 14%) 0.914
HighSpanNear 2660.49 (6.7%) 2654.45
(6.1%) -0.2% ( -12% - 13%) 0.911
LowTerm 9965.56 (8.1%) 9961.77
(10.6%) -0.0% ( -17% - 20%) 0.990
AndHighHigh 3384.43 (7.0%) 3388.41
(11.3%) 0.1% ( -16% - 19%) 0.968
HighSloppyPhrase 1984.76 (5.8%) 1988.83
(5.3%) 0.2% ( -10% - 12%) 0.908
MedIntervalsOrdered 7914.54 (8.2%) 7944.02
(10.2%) 0.4% ( -16% - 20%) 0.899
AndHighMed 4097.29 (7.7%) 4121.75
(8.0%) 0.6% ( -14% - 17%) 0.811
LowSpanNear 5107.67 (9.3%) 5145.19
(7.6%) 0.7% ( -14% - 19%) 0.785
HighTermMonthSort 3221.73 (5.2%) 3245.54
(8.5%) 0.7% ( -12% - 15%) 0.739
HighIntervalsOrdered 1333.81 (7.6%) 1349.72
(5.3%) 1.2% ( -10% - 15%) 0.564
LowPhrase 5029.07 (8.3%) 5091.95
(11.1%) 1.3% ( -16% - 22%) 0.687
Wildcard 1327.36 (3.8%) 1346.91
(3.4%) 1.5% ( -5% - 9%) 0.197
AndHighLow 4382.38 (7.8%) 4447.59
(6.9%) 1.5% ( -12% - 17%) 0.524
OrHighMed 3121.72 (7.2%) 3169.60
(6.4%) 1.5% ( -11% - 16%) 0.478
HighTermDayOfYearSort 3766.72 (6.2%) 3825.04
(7.2%) 1.5% ( -11% - 15%) 0.467
MedTerm 8666.16 (7.3%) 8841.37
(8.1%) 2.0% ( -12% - 18%) 0.406
LowSloppyPhrase 3303.11 (6.5%) 3374.89
(7.8%) 2.2% ( -11% - 17%) 0.341
OrHighHigh 2458.90 (7.7%) 2512.82
(5.2%) 2.2% ( -9% - 16%) 0.289
BrowseDateTaxoFacets 6229.11 (5.5%) 6366.43
(5.7%) 2.2% ( -8% - 14%) 0.211
BrowseDayOfYearTaxoFacets 5695.81 (6.8%) 5830.89
(6.6%) 2.4% ( -10% - 16%) 0.265
MedSpanNear 3161.49 (6.8%) 3242.45
(5.4%) 2.6% ( -8% - 15%) 0.186
MedSloppyPhrase 3363.11 (7.0%) 3456.66
(7.6%) 2.8% ( -11% - 18%) 0.230
# round2
Task QPS baseline StdDev QPS bsearch
StdDev Pct diff p-value
LowTerm 9037.60 (7.0%) 8817.16
(7.1%) -2.4% ( -15% - 12%) 0.276
Respell 724.07 (6.8%) 706.54
(6.1%) -2.4% ( -14% - 11%) 0.236
AndHighLow 5165.41 (9.9%) 5045.20
(9.6%) -2.3% ( -19% - 18%) 0.449
OrHighLow 2393.11 (6.0%) 2341.81
(7.8%) -2.1% ( -15% - 12%) 0.329
LowIntervalsOrdered 3400.45 (5.4%) 3345.42
(6.6%) -1.6% ( -12% - 10%) 0.395
AndHighHigh 3465.41 (7.8%) 3429.20
(7.3%) -1.0% ( -14% - 15%) 0.661
IntNRQ 3252.71 (5.6%) 3236.81
(5.4%) -0.5% ( -10% - 11%) 0.778
HighIntervalsOrdered 4125.58 (4.1%) 4114.03
(6.3%) -0.3% ( -10% - 10%) 0.867
Fuzzy1 951.24 (5.9%) 949.09
(6.8%) -0.2% ( -12% - 13%) 0.911
HighPhrase 2552.87 (6.3%) 2547.78
(8.1%) -0.2% ( -13% - 15%) 0.931
BrowseMonthTaxoFacets 5806.39 (5.8%) 5799.97
(4.7%) -0.1% ( -10% - 11%) 0.947
HighTermDayOfYearSort 3677.43 (4.5%) 3676.63
(5.9%) -0.0% ( -10% - 10%) 0.990
Prefix3 2314.90 (3.9%) 2315.43
(4.6%) 0.0% ( -8% - 8%) 0.986
BrowseDayOfYearTaxoFacets 5734.92 (6.2%) 5739.04
(4.8%) 0.1% ( -10% - 11%) 0.967
LowPhrase 3718.54 (9.1%) 3727.36
(8.7%) 0.2% ( -16% - 19%) 0.933
MedSpanNear 5401.87 (6.7%) 5417.02
(8.4%) 0.3% ( -13% - 16%) 0.907
MedTerm 7734.00 (6.5%) 7767.21
(8.6%) 0.4% ( -13% - 16%) 0.858
Wildcard 1520.86 (4.0%) 1528.72
(5.3%) 0.5% ( -8% - 10%) 0.729
HighTermMonthSort 3750.85 (8.9%) 3783.58
(7.7%) 0.9% ( -14% - 19%) 0.740
PKLookup 425.26 (15.3%) 429.46
(18.0%) 1.0% ( -28% - 40%) 0.852
OrHighMed 2816.18 (5.7%) 2847.69
(6.5%) 1.1% ( -10% - 14%) 0.564
Fuzzy2 324.51 (4.0%) 329.52
(5.4%) 1.5% ( -7% - 11%) 0.302
HighSpanNear 2014.03 (5.0%) 2045.92
(5.5%) 1.6% ( -8% - 12%) 0.338
HighSloppyPhrase 339.81 (9.1%) 345.36
(8.1%) 1.6% ( -14% - 20%) 0.549
MedIntervalsOrdered 5921.86 (6.3%) 6019.48
(8.3%) 1.6% ( -12% - 17%) 0.477
MedSloppyPhrase 2733.55 (5.8%) 2780.57
(7.9%) 1.7% ( -11% - 16%) 0.432
HighTerm 5973.94 (7.6%) 6084.16
(7.9%) 1.8% ( -12% - 18%) 0.450
LowSpanNear 3859.74 (7.9%) 3951.20
(9.1%) 2.4% ( -13% - 21%) 0.379
LowSloppyPhrase 6304.22 (8.9%) 6460.82
(10.6%) 2.5% ( -15% - 24%) 0.424
OrHighHigh 2347.03 (5.3%) 2410.71
(6.1%) 2.7% ( -8% - 14%) 0.133
AndHighMed 3984.56 (7.4%) 4099.88
(8.7%) 2.9% ( -12% - 20%) 0.257
MedPhrase 4069.20 (8.3%) 4191.45
(7.3%) 3.0% ( -11% - 20%) 0.224
BrowseDateTaxoFacets 6149.31 (4.4%) 6386.12
(5.0%) 3.9% ( -5% - 13%) 0.009
BrowseDayOfYearSSDVFacets 5832.22 (19.1%) 6062.83
(11.8%) 4.0% ( -22% - 43%) 0.431
BrowseRandomLabelSSDVFacets 1635.27 (12.1%) 1704.50
(8.5%) 4.2% ( -14% - 28%) 0.200
BrowseMonthSSDVFacets 5978.68 (17.2%) 6374.07
(17.6%) 6.6% ( -24% - 49%) 0.229
BrowseRandomLabelTaxoFacets 2459.73 (11.9%) 2630.18
(4.2%) 6.9% ( -8% - 26%) 0.014
BrowseDateSSDVFacets 2153.79 (8.1%) 2383.48
(15.7%) 10.7% ( -12% - 37%) 0.007
# round3
Task QPS baseline StdDev QPS bsearch
StdDev Pct diff p-value
HighTermMonthSort 3225.93 (9.2%) 3032.53
(8.5%) -6.0% ( -21% - 12%) 0.032
AndHighHigh 3020.25 (6.9%) 2920.75
(7.4%) -3.3% ( -16% - 11%) 0.145
HighTermDayOfYearSort 2255.60 (7.2%) 2185.62
(11.7%) -3.1% ( -20% - 17%) 0.312
MedPhrase 3924.63 (7.7%) 3810.24
(8.4%) -2.9% ( -17% - 14%) 0.254
HighTerm 4736.30 (6.0%) 4609.63
(6.2%) -2.7% ( -13% - 10%) 0.164
Fuzzy1 1127.25 (4.0%) 1107.10
(4.7%) -1.8% ( -9% - 7%) 0.190
LowPhrase 4488.86 (7.4%) 4414.48
(6.8%) -1.7% ( -14% - 13%) 0.462
OrHighMed 2481.56 (4.5%) 2447.67
(7.1%) -1.4% ( -12% - 10%) 0.468
LowSpanNear 5202.58 (7.1%) 5157.55
(6.4%) -0.9% ( -13% - 13%) 0.685
MedTerm 6901.80 (8.7%) 6852.07
(6.7%) -0.7% ( -14% - 16%) 0.769
Respell 796.66 (6.3%) 791.83
(6.8%) -0.6% ( -12% - 13%) 0.771
LowIntervalsOrdered 3582.00 (6.0%) 3562.96
(6.8%) -0.5% ( -12% - 13%) 0.794
BrowseRandomLabelTaxoFacets 2590.20 (4.1%) 2576.78
(3.1%) -0.5% ( -7% - 7%) 0.655
Fuzzy2 460.42 (3.9%) 459.19
(3.8%) -0.3% ( -7% - 7%) 0.826
BrowseDateTaxoFacets 6234.00 (3.6%) 6217.91
(5.0%) -0.3% ( -8% - 8%) 0.852
MedSpanNear 2694.43 (7.5%) 2687.55
(6.1%) -0.3% ( -12% - 14%) 0.906
OrHighLow 3366.04 (7.5%) 3362.25
(7.4%) -0.1% ( -13% - 15%) 0.962
Wildcard 1597.92 (5.4%) 1596.16
(4.2%) -0.1% ( -9% - 10%) 0.943
LowSloppyPhrase 5180.86 (7.8%) 5183.11
(6.9%) 0.0% ( -13% - 15%) 0.985
BrowseDayOfYearTaxoFacets 5683.56 (4.3%) 5689.03
(3.0%) 0.1% ( -6% - 7%) 0.934
Prefix3 2936.08 (6.2%) 2941.73
(5.1%) 0.2% ( -10% - 12%) 0.914
HighSloppyPhrase 2332.11 (4.9%) 2336.61
(7.6%) 0.2% ( -11% - 13%) 0.924
BrowseMonthTaxoFacets 5845.66 (5.3%) 5871.05
(4.2%) 0.4% ( -8% - 10%) 0.773
IntNRQ 3838.04 (4.2%) 3858.84
(7.0%) 0.5% ( -10% - 12%) 0.767
BrowseDateSSDVFacets 2438.30 (21.7%) 2453.33
(21.7%) 0.6% ( -35% - 56%) 0.928
HighSpanNear 1692.49 (5.4%) 1706.61
(4.4%) 0.8% ( -8% - 11%) 0.592
HighPhrase 2510.17 (6.8%) 2540.12
(5.1%) 1.2% ( -10% - 14%) 0.530
AndHighLow 6390.20 (11.8%) 6475.17
(9.5%) 1.3% ( -17% - 25%) 0.694
MedIntervalsOrdered 2377.72 (5.0%) 2411.77
(9.4%) 1.4% ( -12% - 16%) 0.548
LowTerm 9977.52 (9.0%) 10125.70
(10.3%) 1.5% ( -16% - 22%) 0.627
BrowseRandomLabelSSDVFacets 1783.92 (9.7%) 1811.90
(6.9%) 1.6% ( -13% - 20%) 0.555
OrHighHigh 2221.50 (6.8%) 2257.74
(6.2%) 1.6% ( -10% - 15%) 0.426
HighIntervalsOrdered 2496.96 (6.5%) 2568.80
(9.2%) 2.9% ( -12% - 19%) 0.254
BrowseDayOfYearSSDVFacets 6337.43 (20.1%) 6553.25
(16.4%) 3.4% ( -27% - 49%) 0.557
MedSloppyPhrase 4135.87 (7.5%) 4282.32
(8.5%) 3.5% ( -11% - 21%) 0.162
AndHighMed 4011.41 (6.5%) 4170.15
(8.5%) 4.0% ( -10% - 20%) 0.099
BrowseMonthSSDVFacets 6275.39 (17.9%) 6791.72
(15.5%) 8.2% ( -21% - 50%) 0.120
PKLookup 442.55 (10.7%) 480.49
(10.2%) 8.6% ( -11% - 33%) 0.010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
