Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
benwtrent commented on code in PR #3468: URL: https://github.com/apache/solr/pull/3468#discussion_r2336886498 ## solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc: ## @@ -316,6 +316,70 @@ preserved when `stored` is true. + Accepted values: `BOOLEAN` +=== BinaryBitQuantizedDenseVectorField + +Binary bit quantization is a different quantization technique that is even more aggressive in its compression and is able +to reduce each dimension from a 32 bit float down to a single bit. This is done by normalizing each dimension of a vector +relative to a centroid (mid-point pre-calculated against all vectors in the index) with the stored bit +representing whether the actual value is positive or negative of the centroid's value. A further "corrective factor" is computed +and stored to help compensate accuracy in the estimated distance. For more details, the original white paper can be viewed +https://arxiv.org/pdf/2405.12497[here]. Review Comment: Yeah, its a very large set of combination of techniques and net new techniques. Lucene's various attributions purposefully try to attribute different inspirations to each of the appropriate papers while also trying to call out what makes Lucene's technique unique. Its not a pure copy, but it does evolve on the various techniques listed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3272070580 @alessandrobenedetti would appreciate a review when you have the time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
benwtrent commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3276294915 pretty cool to see Solr pick this up :) Thanks for the ping @alessandrobenedetti -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3276277920 Done! Let me know if the documentation wording requires any further tweaks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
alessandrobenedetti commented on code in PR #3468: URL: https://github.com/apache/solr/pull/3468#discussion_r2336891849 ## solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc: ## @@ -316,6 +316,70 @@ preserved when `stored` is true. + Accepted values: `BOOLEAN` +=== BinaryBitQuantizedDenseVectorField + +Binary bit quantization is a different quantization technique that is even more aggressive in its compression and is able +to reduce each dimension from a 32 bit float down to a single bit. This is done by normalizing each dimension of a vector +relative to a centroid (mid-point pre-calculated against all vectors in the index) with the stored bit +representing whether the actual value is positive or negative of the centroid's value. A further "corrective factor" is computed +and stored to help compensate accuracy in the estimated distance. For more details, the original white paper can be viewed +https://arxiv.org/pdf/2405.12497[here]. Review Comment: Thanks for confirming it, Ben! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
benwtrent commented on code in PR #3468:
URL: https://github.com/apache/solr/pull/3468#discussion_r2336888052
##
solr/core/src/java/org/apache/solr/schema/BinaryBitQuantizedDenseVectorField.java:
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.schema;
+
+import org.apache.lucene.codecs.KnnVectorsFormat;
+import
org.apache.lucene.codecs.lucene102.Lucene102HnswBinaryQuantizedVectorsFormat;
+
+public class BinaryBitQuantizedDenseVectorField extends DenseVectorField {
Review Comment:
I would also argue that "bit" might imply that values are already
quantized...(e.g. bit vectors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
alessandrobenedetti commented on code in PR #3468:
URL: https://github.com/apache/solr/pull/3468#discussion_r2336876769
##
solr/core/src/java/org/apache/solr/schema/BinaryBitQuantizedDenseVectorField.java:
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.schema;
+
+import org.apache.lucene.codecs.KnnVectorsFormat;
+import
org.apache.lucene.codecs.lucene102.Lucene102HnswBinaryQuantizedVectorsFormat;
+
+public class BinaryBitQuantizedDenseVectorField extends DenseVectorField {
Review Comment:
to be consistent with Lucene, what about:
'BinaryQuantizedDenseVectorField'?
I think 'bit' may be considered redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
alessandrobenedetti commented on code in PR #3468:
URL: https://github.com/apache/solr/pull/3468#discussion_r2336867659
##
solr/solr-ref-guide/modules/query-guide/pages/dense-vector-search.adoc:
##
@@ -316,6 +316,70 @@ preserved when `stored` is true.
+
Accepted values: `BOOLEAN`
+=== BinaryBitQuantizedDenseVectorField
+
+Binary bit quantization is a different quantization technique that is even
more aggressive in its compression and is able
+to reduce each dimension from a 32 bit float down to a single bit. This is
done by normalizing each dimension of a vector
+relative to a centroid (mid-point pre-calculated against all vectors in the
index) with the stored bit
+representing whether the actual value is positive or negative of the
centroid's value. A further "corrective factor" is computed
+and stored to help compensate accuracy in the estimated distance. For more
details, the original white paper can be viewed
+https://arxiv.org/pdf/2405.12497[here].
Review Comment:
So, in regards to citing the paper(s), I remember there has been a very long
discussion at the time the contribution ended up in Lucene, between the authors
of RaBitQ and Lucene contributors.
What ended up being accepted was :
> "The binary quantization format used here is a per-vector optimized scalar
quantization. These
> * ideas are evolutions of LVQ proposed in https://arxiv.org/abs/2304.04759";>Similarity
> * search in the blink of an eye with compressed indices by Cecilia
Aguerrebere et al., the
> * previous work on globally optimized scalar quantization in Apache
Lucene, and * href="https://arxiv.org/abs/1908.10396";>Accelerating Large-Scale
Inference with Anisotropic
> * Vector Quantization by Ruiqi Guo et. al. Also see {@link
> * org.apache.lucene.util.quantization.OptimizedScalarQuantizer}. Some of
key features are:
> *
> *
> * Estimating the distance between two vectors using their centroid
centered distance. This
> * requires some additional corrective factors, but allows for
centroid centering to occur.
> * Optimized scalar quantization to single bit level of centroid
centered vectors.
> * Asymmetric quantization of vectors, where query vectors are
quantized to half-byte (4 bits)
> * precision (normalized to the centroid) and then compared directly
against the single bit
> * quantized vectors in the index.
> * Transforming the half-byte quantized query vectors in such a way
that the comparison with
> * single bit vectors can be done with bit arithmetic.
> *
> *
> * A previous work related to improvements over regular LVQ is * href="https://arxiv.org/abs/2409.09913";>Practical and Asymptotically
Optimal Quantization of
> * High-Dimensional Vectors in Euclidean Space for Approximate Nearest
Neighbor Search by
> * Jianyang Gao, et. al."
(you can find it in the Javadocs of
org.apache.lucene.codecs.lucene102.Lucene102BinaryQuantizedVectorsFormat).
So given how sensible and long that discussion was, I would encourage
replacing the short citation of RaBitQ with what was added in the official
Lucene JavaDoc (as that should be a better attribution of merit of all the
players involved).
@benwtrent, am I correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3258640252 > Task :solr:core:renderSiteJavadoc [14:23:07.931] FATAL (antora): Failed to download UI bundle: https://nightlies.apache.org/solr/solr-reference-guide-ui-bundle/ui-bundle.zip Pipeline failure seems to have no relation to the code. Am able to run `:solr:solr-ref-guide:buildLocalAntoraSite` locally and the .adoc page compiles correctly. Pushed an empty commit to re-trigger pipeline -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3177925549 @alessandrobenedetti the interface for this new type is actually a subset of the args for DenseVectorField. Would welcome any insight if there are other ways of testing this I should code up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3177920565 This change requires some of the refactoring done in [SOLR-17780](https://github.com/apache/solr/pull/3385) so it's best to view the diff against that branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen commented on PR #3468: URL: https://github.com/apache/solr/pull/3468#issuecomment-3177902151 Going to rebase this branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen closed pull request #3468: [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField URL: https://github.com/apache/solr/pull/3468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[PR] [SOLR-17812] Add support for BinaryBitQuantizedDenseVectorField [solr]
liangkaiwen opened a new pull request, #3468: URL: https://github.com/apache/solr/pull/3468 https://issues.apache.org/jira/browse/SOLR-17812 # Description Incorporate codec for binary bit quantization of dense vectors # Solution Add schema class for BinaryBitQuantizedDenseVectorField # Tests Test schema for new class type loads correctly # Checklist Please review the following and check all that apply: - [X] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [X] I have created a Jira issue and added the issue ID to my pull request title. - [X] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [X] I have developed this patch against the `main` branch. - [X] I have run `./gradlew check`. - [X] I have added tests for my changes. - [X] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
