[jira] [Commented] (SOLR-17615) facets exclusion does not exclude dense vector search prefilters
[
https://issues.apache.org/jira/browse/SOLR-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911641#comment-17911641
]
Yue Yu commented on SOLR-17615:
---
[~abenedetti] [~hossman] Thanks for the respond.
Yes I'm aware the {{excludeTags}} support was added to the {{knn}} Qparser. And
currently it's the workaround to get facet exclusion to work with vector search
query.
However I'd say it's a tradeoff where we'll lose all the benefit of adding fqs
as knn query's prefilter, especially for the vector search recall when topk is
small and several fqs are added as post-filters.
Would it be fair to say that "as of now, there is a limitation that for facet
exclusion to work with knn query, ALL fqs to be excluded by the facet request
MUST ALSO be excluded from the knn query's pre-filters" ? I'm wondering could
we fix the facet exclusion logic in
[SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
so that we can remove this limitation?
Looking at the computeDocSet function in
[SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
, the logic was working perfectly fine until some special Query (knn query)
comes with pre-filters. It feels natural to update this function's logic to
accommodate Query with pre-filters.
The fix seems not trivial. We can use the similar idea in computeDocSet
function to reconstruct and update knn query's pre-filters, but the challenging
part is how to find all the knn queries in the base query {*}rb.getQuery(){*}.
Anyway if you guys think this is worth fixing, we can discuss further.
> facets exclusion does not exclude dense vector search prefilters
>
>
> Key: SOLR-17615
> URL: https://issues.apache.org/jira/browse/SOLR-17615
> Project: Solr
> Issue Type: Bug
> Components: faceting
>Reporter: Yue Yu
>Priority: Major
>
> Vector search adds all fqs as implicit pre-filters by default:
> [https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#implicit-pre-filtering]
> for example:
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true {code}
> where both "category" and "instock" fq are added as the main knn query's
> pre-filters.
>
> However, for the facet multi-select use case, we want to exclude these fqs in
> the facet request so that the facet shows other values
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true
> &facet.field={!ex="category" key="category"}category
> &facet.field={!ex="instock" key="instock"}inStock{code}
> This is done by
> [SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
> who recalculates the DocSet using the main query and all fqs except the
> excluded one:
> {code:java}
>for (String excludeTag : excludeTagList) {
> Object olst = tagMap.get(excludeTag);
> // tagMap has entries of List>, but subject to
> change in the future
> if (!(olst instanceof Collection)) continue;
> for (Object o : (Collection) olst) {
> if (!(o instanceof QParser qp)) continue;
> excludeSet.put(qp.getQuery(), Boolean.TRUE);
> }
> }
> if (excludeSet.size() == 0) return baseDocSet;
>
> List qlist = new ArrayList<>();
> // add the base query
> if (!excludeSet.containsKey(rb.getQuery())) {
> qlist.add(rb.getQuery());
> }
> // add the filters
> if (rb.getFilters() != null) {
> for (Query q : rb.getFilters()) {
> if (!excludeSet.containsKey(q)) {
> qlist.add(q);
> }
> }
> }
> // get the new base docset for this facet
> DocSet base = searcher.getDocSet(qlist); {code}
> This works fine for non-knn main queries as they don't have any pre-filters.
> For knn main queries, the base query *rb.getQuery()* added to the *qlist*
> contains all the fqs as pre-filters, so this facet exclusion logic has no
> effect.
>
> In the example above, the facet values for "category" will only have AAA and
> "instock" will only have inStock.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-17615) facets exclusion does not exclude dense vector search prefilters
[
https://issues.apache.org/jira/browse/SOLR-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911626#comment-17911626
]
Alessandro Benedetti commented on SOLR-17615:
-
I linked the specific documentation in the original solr-user mail thread that
precedes this issue creation.
[~yy147379138] can you tell us what's missing in the feature/documentation?
We'll be happy to think more about this!
> facets exclusion does not exclude dense vector search prefilters
>
>
> Key: SOLR-17615
> URL: https://issues.apache.org/jira/browse/SOLR-17615
> Project: Solr
> Issue Type: Bug
> Components: faceting
>Reporter: Yue Yu
>Priority: Major
>
> Vector search adds all fqs as implicit pre-filters by default:
> [https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#implicit-pre-filtering]
> for example:
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true {code}
> where both "category" and "instock" fq are added as the main knn query's
> pre-filters.
>
> However, for the facet multi-select use case, we want to exclude these fqs in
> the facet request so that the facet shows other values
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true
> &facet.field={!ex="category" key="category"}category
> &facet.field={!ex="instock" key="instock"}inStock{code}
> This is done by
> [SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
> who recalculates the DocSet using the main query and all fqs except the
> excluded one:
> {code:java}
>for (String excludeTag : excludeTagList) {
> Object olst = tagMap.get(excludeTag);
> // tagMap has entries of List>, but subject to
> change in the future
> if (!(olst instanceof Collection)) continue;
> for (Object o : (Collection) olst) {
> if (!(o instanceof QParser qp)) continue;
> excludeSet.put(qp.getQuery(), Boolean.TRUE);
> }
> }
> if (excludeSet.size() == 0) return baseDocSet;
>
> List qlist = new ArrayList<>();
> // add the base query
> if (!excludeSet.containsKey(rb.getQuery())) {
> qlist.add(rb.getQuery());
> }
> // add the filters
> if (rb.getFilters() != null) {
> for (Query q : rb.getFilters()) {
> if (!excludeSet.containsKey(q)) {
> qlist.add(q);
> }
> }
> }
> // get the new base docset for this facet
> DocSet base = searcher.getDocSet(qlist); {code}
> This works fine for non-knn main queries as they don't have any pre-filters.
> For knn main queries, the base query *rb.getQuery()* added to the *qlist*
> contains all the fqs as pre-filters, so this facet exclusion logic has no
> effect.
>
> In the example above, the facet values for "category" will only have AAA and
> "instock" will only have inStock.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-17615) facets exclusion does not exclude dense vector search prefilters
[
https://issues.apache.org/jira/browse/SOLR-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911620#comment-17911620
]
Chris M. Hostetter commented on SOLR-17615:
---
Yup, facet drill downs are exactly why {{excludeTags}} support was added to the
{{knn}} Qparser
We even have a nice test specifically of multi-select facet usecases...
[https://github.com/apache/solr/blob/80dfe11fe4c8f123f56ec2f85870d12b7e086e9e/solr/core/src/test/org/apache/solr/search/neural/KnnQParserTest.java#L831]
(perhaps the bug reporter just wasn't aware of it in recent solr versions?)
> facets exclusion does not exclude dense vector search prefilters
>
>
> Key: SOLR-17615
> URL: https://issues.apache.org/jira/browse/SOLR-17615
> Project: Solr
> Issue Type: Bug
> Components: faceting
>Reporter: Yue Yu
>Priority: Major
>
> Vector search adds all fqs as implicit pre-filters by default:
> [https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#implicit-pre-filtering]
> for example:
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true {code}
> where both "category" and "instock" fq are added as the main knn query's
> pre-filters.
>
> However, for the facet multi-select use case, we want to exclude these fqs in
> the facet request so that the facet shows other values
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true
> &facet.field={!ex="category" key="category"}category
> &facet.field={!ex="instock" key="instock"}inStock{code}
> This is done by
> [SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
> who recalculates the DocSet using the main query and all fqs except the
> excluded one:
> {code:java}
>for (String excludeTag : excludeTagList) {
> Object olst = tagMap.get(excludeTag);
> // tagMap has entries of List>, but subject to
> change in the future
> if (!(olst instanceof Collection)) continue;
> for (Object o : (Collection) olst) {
> if (!(o instanceof QParser qp)) continue;
> excludeSet.put(qp.getQuery(), Boolean.TRUE);
> }
> }
> if (excludeSet.size() == 0) return baseDocSet;
>
> List qlist = new ArrayList<>();
> // add the base query
> if (!excludeSet.containsKey(rb.getQuery())) {
> qlist.add(rb.getQuery());
> }
> // add the filters
> if (rb.getFilters() != null) {
> for (Query q : rb.getFilters()) {
> if (!excludeSet.containsKey(q)) {
> qlist.add(q);
> }
> }
> }
> // get the new base docset for this facet
> DocSet base = searcher.getDocSet(qlist); {code}
> This works fine for non-knn main queries as they don't have any pre-filters.
> For knn main queries, the base query *rb.getQuery()* added to the *qlist*
> contains all the fqs as pre-filters, so this facet exclusion logic has no
> effect.
>
> In the example above, the facet values for "category" will only have AAA and
> "instock" will only have inStock.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-17615) facets exclusion does not exclude dense vector search prefilters
[
https://issues.apache.org/jira/browse/SOLR-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911394#comment-17911394
]
Alessandro Benedetti commented on SOLR-17615:
-
I was thinking more on this issue, can't you just use the excludeTags parameter
of knn query parser?
I think this is exactly the use case it was designed for (isn't this right
[~hossman])
I'm probably missing your point on why that's not useful.
https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#implicit-pre-filtering
excludeTags
Optional
Default: none
Indicates that fq filters with the specified tag should be excluded from
consideration for implicit Pre-Filtering. Must not be combined with preFilter.
> facets exclusion does not exclude dense vector search prefilters
>
>
> Key: SOLR-17615
> URL: https://issues.apache.org/jira/browse/SOLR-17615
> Project: Solr
> Issue Type: Bug
> Components: faceting
>Reporter: Yue Yu
>Priority: Major
>
> Vector search adds all fqs as implicit pre-filters by default:
> [https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#implicit-pre-filtering]
> for example:
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true {code}
> where both "category" and "instock" fq are added as the main knn query's
> pre-filters.
>
> However, for the facet multi-select use case, we want to exclude these fqs in
> the facet request so that the facet shows other values
> {code:java}
> ?q={!knn f=vector topK=10}[1.0, 2.0, 3.0, 4.0]
> &fq={!tag="category"}category:AAA
> &fq={!tag="instock"}inStock:true
> &facet.field={!ex="category" key="category"}category
> &facet.field={!ex="instock" key="instock"}inStock{code}
> This is done by
> [SimpleFacets.java|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/request/SimpleFacets.java#L223]
> who recalculates the DocSet using the main query and all fqs except the
> excluded one:
> {code:java}
>for (String excludeTag : excludeTagList) {
> Object olst = tagMap.get(excludeTag);
> // tagMap has entries of List>, but subject to
> change in the future
> if (!(olst instanceof Collection)) continue;
> for (Object o : (Collection) olst) {
> if (!(o instanceof QParser qp)) continue;
> excludeSet.put(qp.getQuery(), Boolean.TRUE);
> }
> }
> if (excludeSet.size() == 0) return baseDocSet;
>
> List qlist = new ArrayList<>();
> // add the base query
> if (!excludeSet.containsKey(rb.getQuery())) {
> qlist.add(rb.getQuery());
> }
> // add the filters
> if (rb.getFilters() != null) {
> for (Query q : rb.getFilters()) {
> if (!excludeSet.containsKey(q)) {
> qlist.add(q);
> }
> }
> }
> // get the new base docset for this facet
> DocSet base = searcher.getDocSet(qlist); {code}
> This works fine for non-knn main queries as they don't have any pre-filters.
> For knn main queries, the base query *rb.getQuery()* added to the *qlist*
> contains all the fqs as pre-filters, so this facet exclusion logic has no
> effect.
>
> In the example above, the facet values for "category" will only have AAA and
> "instock" will only have inStock.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
