adamjq opened a new pull request, #4698: URL: https://github.com/apache/solr/pull/4698
https://issues.apache.org/jira/browse/SOLR-18267 # Description This PR follows up from https://github.com/apache/solr/pull/4492, which added a new `knnAlgorithm=flat` option to `DenseVectorField`. It adds the same flat index support to `ScalarQuantizedDenseVectorField` so that scalar-quantized fields can also be indexed without building HNSW graphs. The new behavior introduced is: - `ScalarQuantizedDenseVectorField` now supports `knnAlgorithm=flat`. Searching works through the `{!knn}`, `{!knn_text_to_vector}`, and `{!vectorSimilarity}` query parsers, and the `vectorSimilarity()` function query. - `ScalarQuantizedDenseVectorField` with `knnAlgorithm=flat` and `vectorEncoding=BYTE` is rejected explicitly Note: `knnAlgorithm=flat` is not supported for `BinaryQuantizedDenseVectorField` # Solution This PR adds support for `ScalarQuantizedDenseVectorField` with `knnAlgorithm=flat` by delegating to [Lucene104ScalarQuantizedVectorsFormat](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsFormat.java). This implementation differs from https://github.com/apache/solr/pull/4492 in two main ways: 1. The reader in `Lucene104ScalarQuantizedVectorsFormat` implements brute-force KNN search, meaning the `knn`, `knn_text_to_vector` and `vectorSimilarity` query parsers are supported by default 2. `Lucene104ScalarQuantizedVectorsFormat` is registered in SPI, so no wrapper class is needed The table below shows the different segment files generated between the flat vs HNSW configurations: | Ext | Contents | Scalar Quantized (Flat) | Scalar Quantized (HNSW) | |-----|----------|---------|---------| | `.vec` | Raw float vectors | Yes | Yes | | `.vemf` | Raw-float metadata | Yes | Yes | | `.veq` | Quantized vectors | Yes | Yes | | `.vemq` | Quantized metadata | Yes | Yes | | `.vem` | HNSW graph metadata | No | Yes | | `.vex` | HNSW graph | No | Yes | AI Disclosure: Claude was used to assist with this PR. All code has been reviewed and tested by me. # Tests Unit tests for ScalarQuantizedDenseVectorField. # Checklist Please review the following and check all that apply: - [X] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [X] I have created a Jira issue and added the issue ID to my pull request title. - [X] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [X] I have developed this patch against the `main` branch. - [X] I have run `./gradlew check`. - [X] I have added tests for my changes. - [X] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) - [X] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
