benwtrent commented on PR #15564: URL: https://github.com/apache/lucene/pull/15564#issuecomment-3745949990
I reran with oversampling. Basically, this is "fake rescoring" where we gather some multiple more `k` and potentially explore the graph more. Oversampling adds exploration cost and likely floating point op cost that are not present here. But interestingly, 2 and 3 bits (representing my hacky names) are always fairly close together with 3 bit always being slightly slower with slightly better recall, becoming eventually indistinguishable at higher recalls. I am not fully convinced we really want 8bit queries here or instead a smarter way to refine against more bits as we explore. I am also not convinced we want both nibble and byte queries for dibit docs. I am leaning towards just having nibble queries for now. Lucene Util code: https://github.com/mikemccand/luceneutil/compare/main...benwtrent:luceneutil:dibit-tests?expand=1 Lucene code that contains this PR and the 8 bit query: https://github.com/apache/lucene/compare/main...benwtrent:two-bit-to-8-bit?expand=1 <img width="1125" height="522" alt="image" src="https://github.com/user-attachments/assets/7ce08c12-7695-4f07-a916-10c1d637f63a"; /> Raw table <details> ``` recall latency(ms) netCPU avgCpuCount fanout quantized visited index(s) index_docs/s force_merge(s) index_size(MB) overSample vec_disk(MB) vec_RAM(MB) 0.693 1.256 1.207 0.961 0 1 bits 5141 362.70 2757.08 297.18 4189.96 1.000 4041.672 135.422 0.813 1.640 1.629 0.993 0 1 bits 7154 362.70 2757.08 297.18 4189.96 1.500 4041.672 135.422 0.874 1.992 1.982 0.995 0 1 bits 9054 362.70 2757.08 297.18 4189.96 2.000 4041.672 135.422 0.933 2.711 2.702 0.997 0 1 bits 12614 362.70 2757.08 297.18 4189.96 3.000 4041.672 135.422 0.694 1.948 1.938 0.995 100 1 bits 9054 362.70 2757.08 297.18 4189.96 1.000 4041.672 135.422 0.814 2.892 2.855 0.987 100 1 bits 12614 362.70 2757.08 297.18 4189.96 1.500 4041.672 135.422 0.875 3.734 3.650 0.978 100 1 bits 15928 362.70 2757.08 297.18 4189.96 2.000 4041.672 135.422 0.934 4.576 4.548 0.994 100 1 bits 22079 362.70 2757.08 297.18 4189.96 3.000 4041.672 135.422 0.694 2.519 2.485 0.987 150 1 bits 10869 362.70 2757.08 297.18 4189.96 1.000 4041.672 135.422 0.814 3.584 3.525 0.984 150 1 bits 15115 362.70 2757.08 297.18 4189.96 1.500 4041.672 135.422 0.876 4.443 4.404 0.991 150 1 bits 19070 362.70 2757.08 297.18 4189.96 2.000 4041.672 135.422 0.934 5.666 5.565 0.982 150 1 bits 26352 362.70 2757.08 297.18 4189.96 3.000 4041.672 135.422 0.694 2.891 2.866 0.991 200 1 bits 12614 362.70 2757.08 297.18 4189.96 1.000 4041.672 135.422 0.814 4.106 4.049 0.986 200 1 bits 17512 362.70 2757.08 297.18 4189.96 1.500 4041.672 135.422 0.876 4.531 4.506 0.994 200 1 bits 22079 362.70 2757.08 297.18 4189.96 2.000 4041.672 135.422 0.934 6.215 6.191 0.996 200 1 bits 30466 362.70 2757.08 297.18 4189.96 3.000 4041.672 135.422 0.788 1.318 1.298 0.985 0 2 bits 4584 321.12 3114.12 306.88 4297.51 1.000 4163.742 257.492 0.908 1.774 1.735 0.978 0 2 bits 6362 321.12 3114.12 306.88 4297.51 1.500 4163.742 257.492 0.954 2.199 2.186 0.994 0 2 bits 8028 321.12 3114.12 306.88 4297.51 2.000 4163.742 257.492 0.984 3.073 3.035 0.988 0 2 bits 11143 321.12 3114.12 306.88 4297.51 3.000 4163.742 257.492 0.793 2.267 2.233 0.985 100 2 bits 8028 321.12 3114.12 306.88 4297.51 1.000 4163.742 257.492 0.913 3.060 3.026 0.989 100 2 bits 11143 321.12 3114.12 306.88 4297.51 1.500 4163.742 257.492 0.958 3.956 3.877 0.980 100 2 bits 14047 321.12 3114.12 306.88 4297.51 2.000 4163.742 257.492 0.987 4.813 4.778 0.993 100 2 bits 19396 321.12 3114.12 306.88 4297.51 3.000 4163.742 257.492 0.794 2.601 2.589 0.995 150 2 bits 9618 321.12 3114.12 306.88 4297.51 1.000 4163.742 257.492 0.914 3.599 3.578 0.994 150 2 bits 13339 321.12 3114.12 306.88 4297.51 1.500 4163.742 257.492 0.959 4.613 4.597 0.997 150 2 bits 16779 321.12 3114.12 306.88 4297.51 2.000 4163.742 257.492 0.987 5.724 5.712 0.998 150 2 bits 23134 321.12 3114.12 306.88 4297.51 3.000 4163.742 257.492 0.794 3.026 3.008 0.994 200 2 bits 11143 321.12 3114.12 306.88 4297.51 1.000 4163.742 257.492 0.914 4.206 4.163 0.990 200 2 bits 15433 321.12 3114.12 306.88 4297.51 1.500 4163.742 257.492 0.959 4.907 4.841 0.987 200 2 bits 19396 321.12 3114.12 306.88 4297.51 2.000 4163.742 257.492 0.988 6.692 6.660 0.995 200 2 bits 26715 321.12 3114.12 306.88 4297.51 3.000 4163.742 257.492 0.793 1.425 1.408 0.988 0 3 bits 4552 331.95 3012.50 327.71 4296.91 1.000 4163.742 257.492 0.913 1.927 1.919 0.996 0 3 bits 6323 331.95 3012.50 327.71 4296.91 1.500 4163.742 257.492 0.957 2.530 2.501 0.989 0 3 bits 7980 331.95 3012.50 327.71 4296.91 2.000 4163.742 257.492 0.985 3.238 3.229 0.997 0 3 bits 11069 331.95 3012.50 327.71 4296.91 3.000 4163.742 257.492 0.798 2.430 2.420 0.996 100 3 bits 7980 331.95 3012.50 327.71 4296.91 1.000 4163.742 257.492 0.919 3.322 3.310 0.996 100 3 bits 11069 331.95 3012.50 327.71 4296.91 1.500 4163.742 257.492 0.962 4.408 4.359 0.989 100 3 bits 13954 331.95 3012.50 327.71 4296.91 2.000 4163.742 257.492 0.989 5.632 5.618 0.998 100 3 bits 19258 331.95 3012.50 327.71 4296.91 3.000 4163.742 257.492 0.799 3.026 2.999 0.991 150 3 bits 9555 331.95 3012.50 327.71 4296.91 1.000 4163.742 257.492 0.919 4.227 4.205 0.995 150 3 bits 13251 331.95 3012.50 327.71 4296.91 1.500 4163.742 257.492 0.963 5.247 5.211 0.993 150 3 bits 16673 331.95 3012.50 327.71 4296.91 2.000 4163.742 257.492 0.989 6.682 6.636 0.993 150 3 bits 22980 331.95 3012.50 327.71 4296.91 3.000 4163.742 257.492 0.799 3.442 3.425 0.995 200 3 bits 11069 331.95 3012.50 327.71 4296.91 1.000 4163.742 257.492 0.919 4.666 4.626 0.991 200 3 bits 15334 331.95 3012.50 327.71 4296.91 1.500 4163.742 257.492 0.963 5.414 5.397 0.997 200 3 bits 19258 331.95 3012.50 327.71 4296.91 2.000 4163.742 257.492 0.989 7.598 7.549 0.994 200 3 bits 26522 331.95 3012.50 327.71 4296.91 3.000 4163.742 257.492 ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
