[
https://issues.apache.org/jira/browse/SOLR-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643913#comment-17643913
]
Gabriel Magno commented on SOLR-16567:
--------------------------------------
I've taken a quick look at the Solr code, and noticed that the problem is
related to the new pre-filtering feature introduced in Solr 9.1.
Apparently what happens is that when parsing the KNN query, this is the flow:
1. The `KnnQParser.parse` function is called to parse the KNN query;
2. The `KnnQParser.getFilterQuery` function is called, to identify all the
filters in the query
3. Among the filters of the query there is the `frange` filter, which is
applied to the result of the KNN sub-query
4. Then the parser for the KNN query is called again, and we enter in an
infinite loop of function calls, causing the Stack Overflow.
The thing is that in the case of the example I've given, one of the filters is
actually a post-filter (since it actually filters based on the value of the
similarity of the KNN).
Regardless of the frange filter itself, I think the problem is related to the
scope of filters considering queries and sub-queries of KNN.
Brainstorm of possible solutions:
- Apply the pre-filtering of KNN only if it is the actual main query
(`q=\{!knn ...}`), but not apply when it is a sub-query (`my_subquery=\{!knn
...}`).
- Apply the pre-filtering only to pre-filters, and isolate "post-filters" to
be applied later.
- ?
I don't have knowledge of the internal Solr code workflow, so I'm not sure of
the best approach and rather these really solves the problem. Maybe there are
other solutions that I'm not seeing here.
> java.lang.StackOverflowError when combining KnnQParser and
> FunctionRangeQParser
> -------------------------------------------------------------------------------
>
> Key: SOLR-16567
> URL: https://issues.apache.org/jira/browse/SOLR-16567
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query
> Affects Versions: 9.1
> Environment: Solr Cloud with `solr:9.1` Docker image
> Reporter: Gabriel Magno
> Priority: Major
> Attachments: create_example-solr_9_0.sh, create_example-solr_9_1.sh,
> error_full.txt, response-error.json, run_query.sh
>
>
> Hello there!
> I had a Solr 9.0 cluster running, using the new Dense Vector feature.
> Recently I have migrated to Solr 9.1. Most of the things are working fine,
> except for a special case I have here.
> *Error Description*
> The problem happens when I try making an Edismax query with a KNN sub-query
> and a Function Range filter. For example, I try making this query.
> * defType=edismax
> * df=name
> * q=the
> * similarity_vector=\{!knn f=vector topK=10}[1.1,2.2,3.3,4.4]
> * {!frange l=0.99}$similarity_vector
> In other words, I want all the documents matching the term "the" in the
> "name" field, and I filter to return only documents having a vector
> similarity of at least 0.99. This query was working fine on Solr 9.0, but on
> Solr 9.1, I get his error:
>
> {code:java}
> java.lang.RuntimeException: java.lang.StackOverflowError at
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:840) at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:641) at
> org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:250)
> at
> org.apache.solr.servlet.SolrDispatchFilter.lambda/usr/bin/zsh(SolrDispatchFilter.java:218)
> at
> org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257)
> at
> org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
> at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
> ... (manually supressed for brevity) at
> java.base/java.lang.Thread.run(Unknown Source)Caused by:
> java.lang.StackOverflowError at
> org.apache.solr.search.StrParser.getId(StrParser.java:172) at
> org.apache.solr.search.StrParser.getId(StrParser.java:168) at
> org.apache.solr.search.QueryParsing.parseLocalParams(QueryParsing.java:100)
> at
> org.apache.solr.search.QueryParsing.parseLocalParams(QueryParsing.java:65)
> at org.apache.solr.search.QParser.getParser(QParser.java:364) at
> org.apache.solr.search.QParser.getParser(QParser.java:334) at
> org.apache.solr.search.QParser.getParser(QParser.java:321) at
> org.apache.solr.search.QueryUtils.parseFilterQueries(QueryUtils.java:244)
> at
> org.apache.solr.search.neural.KnnQParser.getFilterQuery(KnnQParser.java:93)
> at org.apache.solr.search.neural.KnnQParser.parse(KnnQParser.java:83) at
> org.apache.solr.search.QParser.getQuery(QParser.java:188) at
> org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:384)
> at org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:94)
> at org.apache.solr.search.QParser.getQuery(QParser.java:188) at
> org.apache.solr.search.FunctionRangeQParserPlugin.parse(FunctionRangeQParserPlugin.java:53)
> at org.apache.solr.search.QParser.getQuery(QParser.java:188) at
> org.apache.solr.search.QueryUtils.parseFilterQueries(QueryUtils.java:246)
> at
> org.apache.solr.search.neural.KnnQParser.getFilterQuery(KnnQParser.java:93)
> at org.apache.solr.search.neural.KnnQParser.parse(KnnQParser.java:83) at
> org.apache.solr.search.QParser.getQuery(QParser.java:188) at
> org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:384)
> at org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:94)
> at org.apache.solr.search.QParser.getQuery(QParser.java:188) at
> org.apache.solr.search.FunctionRangeQParserPlugin.parse(FunctionRangeQParserPlugin.java:53)
> at org.apache.solr.search.QParser.getQuery(QParser.java:188) ...
> (manually supressed for brevity){code}
>
> The backtrace is much bigger, I'm attaching the raw Solr response in JSON
> (which is truncated) in {_}*response-error.json*{_}, and the full error
> message in _*error_full.txt*_ . Apparently there is an infinite loop on the
> error message that same 9 backtrace entries are repeated ad-infinitum, so I
> believe this is the reason the error message Solr gives is truncated.
> *How to Reproduce the Error*
> I was able to create an small reproducible example to replicate the error. It
> creates a collection with two fields, and index 2 example documents. I
> provide the bash scripts to create the collections, and also the script to
> make the query. These are the scripts:
> * {*}create_example-solr_9_0.sh{*}: creates the collection, the fields,
> schema and index example documents for Solr 9.0.
> * {*}create_example-solr_9_1.sh{*}: creates the collection, the fields,
> schema and index example documents for Solr 9.1.
> * {*}run_query.sh{*}: executes the query that will responde correctly on
> Solr 9.0, and will return a Java Stack Overflow error on Solr 9.1.
> I have created separated scripts for Solr 9.0 and Solr 9.1 because of the
> change on the parameters of the Dense Vector field, which when from
> codecFormat to knnAlgorithm. So the two scripts differ only when defyning the
> vector field type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
