Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti merged PR #3316: URL: https://github.com/apache/solr/pull/3316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3637815809 A first revised draft is fully green after the refactor following @hossman suggestions. I'll keep it open for a few days, refine it a bit and then merge it unless critical observations! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2549191103
##
solr/solr-ref-guide/modules/query-guide/pages/searching-nested-documents.adoc:
##
@@ -184,6 +184,22 @@ $ curl 'http://localhost:8983/solr/gettingstarted/select'
-d 'omitHeader=true' -
+[#vector-search-child]
+[CAUTION]
+.Vector search - children are nested documents with a vector field
+
+It is quite common to encode the original text of a document into multiple
nested vectors.
+
+This may happen, among other use cases, because you chunked the original text
into paragraphs, each of them modeled as a nested document with the paragraph
text and the vector representation.
+
+Solr doesn't need to have denormalised nested documents, you can still
retrieve the children paragraphs by knn vector search and prefilter them using
parent level metadata.
+
+[source,text]
+
+$ curl 'http://localhost:8983/solr/gettingstarted/select' -d 'omitHeader=true'
--data-urlencode 'fq={!child of=$block_mask filters=$parentsFilter}&q={!knn
f=childVectorField topK=5}[1.0,2.5,3.0...]' --data-urlencode 'block_mask=(*:*
-{!prefix f="_nest_path_" v="/skus/" parentsFilter="name_s:pen"})'
Review Comment:
+1 I'll sort this out and update the PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316: URL: https://github.com/apache/solr/pull/3316#discussion_r2549189739 ## solr/solr-ref-guide/modules/query-guide/pages/searching-nested-documents.adoc: ## @@ -184,6 +184,22 @@ $ curl 'http://localhost:8983/solr/gettingstarted/select' -d 'omitHeader=true' - +[#vector-search-child] +[CAUTION] +.Vector search - children are nested documents with a vector field + +It is quite common to encode the original text of a document into multiple nested vectors. + +This may happen, among other use cases, because you chunked the original text into paragraphs, each of them modeled as a nested document with the paragraph text and the vector representation. + +Solr doesn't need to have denormalised nested documents, you can still retrieve the children paragraphs by knn vector search and prefilter them using parent level metadata. Review Comment: I agree 100%, I wrote it a few months ago but I don't understand it now, a clear sign that a rephrase is necessary. Possibly I meant to do it as the work in this PR needed to be finished, anyway, I'll spend some time on it very soon, and update the PR (and comments) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3558697935 I'm resuming the work on this briefly tomorrow, and then at full capacity the week of the 1st of December. Thanks, David, for the pointers. I'll carefully read them and respond soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
dsmiley commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2188787351
##
solr/solr-ref-guide/modules/query-guide/pages/searching-nested-documents.adoc:
##
@@ -184,6 +184,22 @@ $ curl 'http://localhost:8983/solr/gettingstarted/select'
-d 'omitHeader=true' -
+[#vector-search-child]
+[CAUTION]
+.Vector search - children are nested documents with a vector field
+
+It is quite common to encode the original text of a document into multiple
nested vectors.
+
+This may happen, among other use cases, because you chunked the original text
into paragraphs, each of them modeled as a nested document with the paragraph
text and the vector representation.
+
+Solr doesn't need to have denormalised nested documents, you can still
retrieve the children paragraphs by knn vector search and prefilter them using
parent level metadata.
+
+[source,text]
+
+$ curl 'http://localhost:8983/solr/gettingstarted/select' -d 'omitHeader=true'
--data-urlencode 'fq={!child of=$block_mask filters=$parentsFilter}&q={!knn
f=childVectorField topK=5}[1.0,2.5,3.0...]' --data-urlencode 'block_mask=(*:*
-{!prefix f="_nest_path_" v="/skus/" parentsFilter="name_s:pen"})'
Review Comment:
A new line per param (thus end the line with `\`) would be helpful to make
this more readable. But mostly I think this complicated query needs an english
description preceding it.
##
solr/solr-ref-guide/modules/query-guide/pages/searching-nested-documents.adoc:
##
@@ -184,6 +184,22 @@ $ curl 'http://localhost:8983/solr/gettingstarted/select'
-d 'omitHeader=true' -
+[#vector-search-child]
+[CAUTION]
+.Vector search - children are nested documents with a vector field
+
+It is quite common to encode the original text of a document into multiple
nested vectors.
+
+This may happen, among other use cases, because you chunked the original text
into paragraphs, each of them modeled as a nested document with the paragraph
text and the vector representation.
+
+Solr doesn't need to have denormalised nested documents, you can still
retrieve the children paragraphs by knn vector search and prefilter them using
parent level metadata.
Review Comment:
> Solr doesn't need to have denormalised nested documents
That sentence fragment confuses me. What does "denomalized nested
documents" mean? Do you mean simply that Solr doesn't support multiValued
vector fields? If so, just say that please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3437979953 I'll resume shortly the work on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
github-actions[bot] closed pull request #3316: SOLR-17736: introducing support for KNN search on nested vectors (block join) URL: https://github.com/apache/solr/pull/3316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
github-actions[bot] commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3434622235 This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
github-actions[bot] commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3217487153 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-3000816569 I'll resume this soon, some sponsorship should arrive and my agenda looks more relaxed starting from July -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
github-actions[bot] commented on PR #3316: URL: https://github.com/apache/solr/pull/3316#issuecomment-2974772199 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2047340978
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +93,127 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+ try {
+ List childrenClauses = childrenQuery.clauses();
+ if (isByteKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
Review Comment:
agreed! Coming in the next commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2047337900
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +93,127 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+ try {
+ List childrenClauses = childrenQuery.clauses();
+ if (isByteKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+byte[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query acceptedChildren =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenByteKnnVectorQuery(
+vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else if (isFloatKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnFloatVectorQuery knnChildrenQuery =
+(KnnFloatVectorQuery) childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+float[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query childrenFilter =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenFloatKnnVectorQuery(
+vectorField, queryVector, childrenFilter, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else {
+return new AllParentsAware(
+childrenQuery,
+getBitSetProducer(allParents),
+ScoreModeParser.parse(scoreMode),
+allParents);
+ }
+ } catch (IOException e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
+ }
+ }
+
+ private void setAppropriateChildrenListingTransformer(SolrQueryRequest
request, Query knnOnVectorField) throws IOException {
+QueryLimits currentLimits = QueryLimits.getCurrentLimits();
+ReturnFields returnFields = currentLimits.getRsp().getReturnFields();
+DocTransformer originalTransformer = returnFields.getTransformer();
+
+if (originalTransformer instanceof DocTransformers) {
+ DocTransformers transformers = (DocTransformers) originalTransformer;
+ boolean noChildTransformer = true;
+ for (int i = 0; i < transformers.size() && noChildTransformer; i++) {
+DocTransformer t = transformers.getTransformer(i);
+if (t instanceof ChildDocTransformer) {
+ ChildDocTransformer childTransformer = (ChildDocTransformer) t;
+ if (childTransformer.getChildDocSet() == null) {
+
childTransformer.setChildDocSet(request.getSearcher().getDocSet(knnOnVectorField));
+ }
+ noChildTransformer = false;
+}
+ }
+} else {
+ if ((originalTransformer instanceof ChildDocTransformer)) {
+ChildDocTransformer childTransformer = (ChildDocTransformer)
originalTransformer;
+if (childTransformer.getChildDocSet() == null) {
+
childTransformer.setChildDocSet(request.getSearcher().getDocSet(knnOnVectorField));
+}
+ }
+}
+ }
+
+ private boolean isFloatKnnQuery(List childrenClauses) {
+return childrenClauses.size() == 1
+&&
childrenClauses.get(0).getQuery().getClass().equals(KnnFloatVectorQuery.class);
+ }
+
+ private boolean is
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2047275605
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +93,127 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+ try {
+ List childrenClauses = childrenQuery.clauses();
+ if (isByteKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+byte[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query acceptedChildren =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenByteKnnVectorQuery(
+vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else if (isFloatKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnFloatVectorQuery knnChildrenQuery =
+(KnnFloatVectorQuery) childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+float[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query childrenFilter =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenFloatKnnVectorQuery(
+vectorField, queryVector, childrenFilter, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else {
+return new AllParentsAware(
+childrenQuery,
+getBitSetProducer(allParents),
+ScoreModeParser.parse(scoreMode),
+allParents);
+ }
+ } catch (IOException e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
+ }
Review Comment:
I just double checked and it's related to the searcher.rewrite and the
searcher.getDocSet, they throw IOException, not sure how much we can comment
here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
cpoerschke commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2039495987
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +93,127 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+ try {
+ List childrenClauses = childrenQuery.clauses();
+ if (isByteKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+byte[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query acceptedChildren =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenByteKnnVectorQuery(
+vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else if (isFloatKnnQuery(childrenClauses)) {
+BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+BooleanQuery parentsFilter = getParentsFilter();
+
+KnnFloatVectorQuery knnChildrenQuery =
+(KnnFloatVectorQuery) childrenClauses.get(0).getQuery();
+String vectorField = knnChildrenQuery.getField();
+float[] queryVector = knnChildrenQuery.getTargetCopy();
+int topK = knnChildrenQuery.getK();
+
+Query childrenFilter =
+getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+Query knnChildren =
+new DiversifyingChildrenFloatKnnVectorQuery(
+vectorField, queryVector, childrenFilter, topK,
allParentsBitSet);
+knnChildren = knnChildren.rewrite(req.getSearcher());
+this.setAppropriateChildrenListingTransformer(req,knnChildren);
+
+return new ToParentBlockJoinQuery(
+knnChildren, allParentsBitSet,
ScoreModeParser.parse(scoreMode));
+ } else {
+return new AllParentsAware(
+childrenQuery,
+getBitSetProducer(allParents),
+ScoreModeParser.parse(scoreMode),
+allParents);
+ }
+ } catch (IOException e) {
+throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
+ }
+ }
+
+ private void setAppropriateChildrenListingTransformer(SolrQueryRequest
request, Query knnOnVectorField) throws IOException {
+QueryLimits currentLimits = QueryLimits.getCurrentLimits();
+ReturnFields returnFields = currentLimits.getRsp().getReturnFields();
+DocTransformer originalTransformer = returnFields.getTransformer();
+
+if (originalTransformer instanceof DocTransformers) {
+ DocTransformers transformers = (DocTransformers) originalTransformer;
+ boolean noChildTransformer = true;
+ for (int i = 0; i < transformers.size() && noChildTransformer; i++) {
+DocTransformer t = transformers.getTransformer(i);
+if (t instanceof ChildDocTransformer) {
+ ChildDocTransformer childTransformer = (ChildDocTransformer) t;
+ if (childTransformer.getChildDocSet() == null) {
+
childTransformer.setChildDocSet(request.getSearcher().getDocSet(knnOnVectorField));
+ }
+ noChildTransformer = false;
+}
+ }
+} else {
+ if ((originalTransformer instanceof ChildDocTransformer)) {
+ChildDocTransformer childTransformer = (ChildDocTransformer)
originalTransformer;
+if (childTransformer.getChildDocSet() == null) {
+
childTransformer.setChildDocSet(request.getSearcher().getDocSet(knnOnVectorField));
+}
+ }
+}
+ }
+
+ private boolean isFloatKnnQuery(List childrenClauses) {
+return childrenClauses.size() == 1
+&&
childrenClauses.get(0).getQuery().getClass().equals(KnnFloatVectorQuery.class);
+ }
+
+ private boolean isByteKnnQu
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
liangkaiwen commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2035547303
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +87,89 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+List childrenClauses = childrenQuery.clauses();
+if (isByteKnnQuery(childrenClauses)) {
+ BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+ BooleanQuery parentsFilter = getParentsFilter();
+
+ KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+ String vectorField = knnChildrenQuery.getField();
+ byte[] queryVector = knnChildrenQuery.getTargetCopy();
+ int topK = knnChildrenQuery.getK();
+
+ Query acceptedChildren =
+ getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+ Query knnChildren =
+ new DiversifyingChildrenByteKnnVectorQuery(
+ vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+ return new ToParentBlockJoinQuery(
+ knnChildren, allParentsBitSet, ScoreModeParser.parse(scoreMode));
+} else if (isFloatKnnQuery(childrenClauses)) {
Review Comment:
Makes sense. I did not realize the parent class was package private. Maybe
there's an argument to be made to make it public so as not to introduce this
kind of branching everywhere, but that's a separate lucene question. This is
fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
alessandrobenedetti commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2033539345
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +87,89 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+List childrenClauses = childrenQuery.clauses();
+if (isByteKnnQuery(childrenClauses)) {
+ BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+ BooleanQuery parentsFilter = getParentsFilter();
+
+ KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+ String vectorField = knnChildrenQuery.getField();
+ byte[] queryVector = knnChildrenQuery.getTargetCopy();
+ int topK = knnChildrenQuery.getK();
+
+ Query acceptedChildren =
+ getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+ Query knnChildren =
+ new DiversifyingChildrenByteKnnVectorQuery(
+ vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+ return new ToParentBlockJoinQuery(
+ knnChildren, allParentsBitSet, ScoreModeParser.parse(scoreMode));
+} else if (isFloatKnnQuery(childrenClauses)) {
Review Comment:
Hi @liangkaiwen, the parent class
org.apache.lucene.search.AbstractKnnVectorQuery is not public and also when
building the
DiversifyingChildrenFloatKnnVectorQuery/DiversifyingChildrenByteKnnVectorQuery
I am afraid the if branching can not be avoided?
I agree there's boring repeated code, not sure we can avoid it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17736: introducing support for KNN search on nested vectors (block join) [solr]
liangkaiwen commented on code in PR #3316:
URL: https://github.com/apache/solr/pull/3316#discussion_r2033397396
##
solr/core/src/java/org/apache/solr/search/join/BlockJoinParentQParser.java:
##
@@ -78,10 +87,89 @@ protected Query noClausesQuery() throws SyntaxError {
return new BitSetProducerQuery(getBitSetProducer(parseParentFilter()));
}
- protected Query createQuery(final Query parentList, Query query, String
scoreMode)
+ protected Query createQuery(final Query allParents, BooleanQuery
childrenQuery, String scoreMode)
throws SyntaxError {
-return new AllParentsAware(
-query, getBitSetProducer(parentList),
ScoreModeParser.parse(scoreMode), parentList);
+List childrenClauses = childrenQuery.clauses();
+if (isByteKnnQuery(childrenClauses)) {
+ BitSetProducer allParentsBitSet = getBitSetProducer(allParents);
+ BooleanQuery parentsFilter = getParentsFilter();
+
+ KnnByteVectorQuery knnChildrenQuery = (KnnByteVectorQuery)
childrenClauses.get(0).getQuery();
+ String vectorField = knnChildrenQuery.getField();
+ byte[] queryVector = knnChildrenQuery.getTargetCopy();
+ int topK = knnChildrenQuery.getK();
+
+ Query acceptedChildren =
+ getChildrenFilter(knnChildrenQuery.getFilter(), parentsFilter,
allParentsBitSet);
+
+ Query knnChildren =
+ new DiversifyingChildrenByteKnnVectorQuery(
+ vectorField, queryVector, acceptedChildren, topK,
allParentsBitSet);
+ return new ToParentBlockJoinQuery(
+ knnChildren, allParentsBitSet, ScoreModeParser.parse(scoreMode));
+} else if (isFloatKnnQuery(childrenClauses)) {
Review Comment:
is the branching on type of Knn query necessary? As far as I can see, the
majority of the code blocks are equivalent and the different instance variables
of KnnByteVectorQuery/KnnFloatVectorQuery are being used like its generic
parent class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
