Re: Upgrade Codec 99 to 101 / 102

[Michael Wechner]

Hi Ben

Thanks very much for the insight!
So IIUC it is correct to use Lucene102HnswBinaryQuantizedVectorsFormat() 
for float32 and int8, right?

Thanks

Michael

Am 10.09.25 um 17:58 schrieb Benjamin Trent:
Hey Michael,

Right now it won't quantize byte vectors that are provided. It just 
passes them and treats them like normal.

In the future, we would like to quantize the bytes as well!

On Wed, Sep 10, 2025, 10:56 AM Michael Wechner 
<michael.wech...@wyona.com> wrote:

    Hi Uwe

    Thanks for your feedback!

    I am using now:

    private final int maxDimensions =16384;

    Codec codec =new Lucene101Codec() {
         @Override public KnnVectorsFormat getKnnVectorsFormatForField(String 
field) {var delegate =new Lucene102HnswBinaryQuantizedVectorsFormat();
             return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
         }
    };
    return codec;

    This seems to work fine, but I do not understand whether there is a 
difference anymore between float32 and int8 vector values,
    or whetherLucene102HnswBinaryQuantizedVectorsFormat always "quantizes" the 
values?

    Thanks

    Michael


    Am 10.09.25 um 10:19 schrieb Uwe Schindler:

    Hi,

    I think the best is to check the source code of default codec of
    the upgrade version and start from there. I agree, there should
    possibly a documentation available that gives the default
    componenets used in the default codec of a given release.

    Uwe

    Am 09.09.2025 um 22:49 schrieb Michael Wechner:

    Hi

    With Lucene 9.12.0 I set my own custom max vector dimension using

    Codec codecInt8 = new Lucene99Codec() { @Override public
    KnnVectorsFormat getKnnVectorsFormatForField(String field) { var
    delegate = new Lucene99HnswScalarQuantizedVectorsFormat();
    log.info <http://log.info>("Vector Value Type: int8, Maximum
    Vector Dimension: " + maxDimensions); return new
    DelegatingKnnVectorsFormat(delegate, maxDimensions); } }; and
    Codec codecFloat32 =new Lucene99Codec() {
         @Override public KnnVectorsFormat getKnnVectorsFormatForField(String 
field) {
             var delegate =new Lucene99HnswVectorsFormat();
             log.info("Vector Value Type: float32, Maximum Vector Dimension: " 
+maxDimensions);
             return new DelegatingKnnVectorsFormat(delegate,maxDimensions);
         }
    };

    I am a little confused re which Codec / Vector Format classes I should use 
when upgrading to Lucene version 10.2.2

    Any hints would be much appreciated!

    Thanks

    Michael


    -- 
    Uwe Schindler
    Achterdiek 19, D-28357 Bremen 
<https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
    https://www.thetaphi.de
    eMail:u...@thetaphi.de