Re: VectorField renamed to KnnVectorField?

[Michael Wechner]

Hi Vigya

Great, thank you very much for these links!

All the best

Michael

Am 02.11.21 um 01:45 schrieb Vigya Sharma:

Hi Michael,

Glad you got it working. There is also a KNN vector search demo that was added not long ago. You might want to check it out. It has references for example, to compute embeddings and build knn vector queries <https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java#L272-L292>, among other things.

  * Search Files:
    
https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java
    
<https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/SearchFiles.java>

  * Index Files:
    
https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java
    
<https://github.com/apache/lucene/blob/main/lucene/demo/src/java/org/apache/lucene/demo/IndexFiles.java>

  * knn demo files:
    
https://github.com/apache/lucene/tree/main/lucene/demo/src/java/org/apache/lucene/demo/knn
    
<https://github.com/apache/lucene/tree/main/lucene/demo/src/java/org/apache/lucene/demo/knn>


- Vigya

On Mon, Nov 1, 2021 at 2:44 PM Michael Wechner <michael.wech...@wyona.com <mailto:michael.wech...@wyona.com>> wrote:

    I was able to update my code

    -        FieldType vectorFieldType =
    VectorField.createHnswType(vector.length,
    VectorValues.SimilarityFunction.DOT_PRODUCT, 16, 500);
    -        VectorField vectorField = new VectorField(VECTOR_FIELD,
    vector, vectorFieldType);
    +        FieldType vectorFieldType =
    KnnVectorField.createFieldType(vector.length,
    VectorSimilarityFunction.DOT_PRODUCT);
    +        KnnVectorField vectorField = new
    KnnVectorField(VECTOR_FIELD, vector, vectorFieldType);

    and

    -            return new TopDocScorer(this,
    context.reader().searchNearestVectors(field, vector, topK, fanout));
    +            return new TopDocScorer(this,
    context.reader().searchNearestVectors(field, vector, topK, null));

    the indexing and searching works again :-)

    Thanks

    Michael

    Am 01.11.21 um 18:53 schrieb Michael Wechner:

    Hi

    In May 2021 I have done a Vector Search implementation based on Lucene 
9.0.0-SNAPSHOT with the following code

    FieldType vectorFieldType = VectorField.createHnswType(vector.length, 
VectorValues.SimilarityFunction.DOT_PRODUCT,16,500);
    VectorField vectorField =new VectorField(VECTOR_FIELD, vector, 
vectorFieldType);
    doc.add(vectorField)

    and

    class KnnWeightextends Weight {

         KnnWeight() {
             super(KnnQuery.this);
         }

         @Override public Scorer scorer(LeafReaderContext context)throws 
IOException {
             log.debug("Get scorer.");
             return new TopDocScorer(this, 
context.reader().searchNearestVectors(field,vector,topK,fanout));
         }

    whereas fanout is of type "int"

    I have now updated Lucene source and rebuilt 9.0.0-SNAPSHOT and get various 
compile errors.

    I assume VectorField got renamed to KnnVectorField, right?

    Does somebody maybe have some sample code how Vector search is being 
implemented with the most recent Lucene code?

    Thanks

    Michael

--
- Vigya