kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5261243742
### Benchmarks
Cohere v3 vectors, 100K docs, 10K queries, maxConn=64, beamWidth=250,
topK=100, fanout=100, force merged to 1 segment.
`main`
```
quantized visited index(s) index_docs/s merge(s) force_merge(s)
index_size(MB) filterStrategy recall latency(ms) netCPU avgCpuCount
no 12962 19.93 5016.55 22.12 20.83
403.01 query-time-pre-filter 0.495 4.301 4.300 1.000
no 6198 21.58 4633.71 20.02 35.27
603.03 index-time-filter 0.995 1.941 1.940 1.000
8 bits 12994 19.85 5038.04 23.42 28.17
502.19 query-time-pre-filter 0.495 5.060 5.059 1.000
8 bits 6202 21.93 4560.38 23.92 47.01
751.75 index-time-filter 0.985 2.411 2.410 1.000
4 bits 13136 20.35 4915.21 13.21 20.34
453.40 query-time-pre-filter 0.495 2.946 2.945 1.000
4 bits 6279 22.57 4431.25 12.31 33.11
678.62 index-time-filter 0.931 1.288 1.287 0.999
2 bits 13667 20.51 4875.43 13.71 20.74
429.31 query-time-pre-filter 0.489 2.387 2.386 1.000
2 bits 6587 22.28 4487.73 12.31 32.34
642.48 index-time-filter 0.842 0.960 0.960 0.999
1 bits 15227 19.60 5101.26 11.41 17.39
418.30 query-time-pre-filter 0.452 1.996 1.995 1.000
1 bits 7412 22.28 4488.13 10.21 28.86
625.90 index-time-filter 0.726 0.893 0.892 0.999
```
This PR
```
quantized visited index(s) index_docs/s merge(s) force_merge(s)
index_size(MB) filterStrategy recall latency(ms) netCPU avgCpuCount
no 12956 19.80 5051.53 18.92 20.13
403.38 query-time-pre-filter 0.495 4.481 4.480 1.000
no 6192 24.71 4047.11 25.32 30.99
409.01 index-time-filter 0.995 2.062 2.062 1.000
8 bits 12965 19.87 5031.95 22.82 24.53
502.57 query-time-pre-filter 0.495 5.129 5.128 1.000
8 bits 6206 25.26 3958.51 30.33 37.65
508.19 index-time-filter 0.985 2.372 2.371 0.999
4 bits 13167 20.06 4985.04 11.31 16.64
453.79 query-time-pre-filter 0.495 2.911 2.910 1.000
4 bits 6279 25.18 3971.56 19.82 24.26
459.44 index-time-filter 0.922 1.288 1.286 0.999
2 bits 13765 20.51 4875.91 11.21 17.33
429.54 query-time-pre-filter 0.481 2.325 2.323 1.000
2 bits 6688 25.47 3926.50 18.92 26.52
435.27 index-time-filter 0.809 1.038 1.037 0.999
1 bits 15405 19.88 5029.68 10.41 15.19
418.55 query-time-pre-filter 0.434 2.139 2.138 1.000
1 bits 7590 24.74 4041.87 16.51 24.12
424.78 index-time-filter 0.687 0.952 0.951 0.999
```
Some observations:
- `recall` is lower with the data-blind quantization, and drop increases
with higher levels of quantization as expected (drop is <4% for this
combination).
- `latency` is higher with the de-duplication, and the increase as a
percentage is higher with higher levels of quantization as expected (increase
is <10% for this combination).
- `index_size(MB)` drops with the de-duplicating vector format, and the
index size is approximately the same b/w `query-time-pre-filter` and
`index-time-filter` (differing in HNSW graph size).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
