Varun Thacker created SOLR-17642:
------------------------------------
Summary: Knn queries use multi threaded search unintentionally
Key: SOLR-17642
URL: https://issues.apache.org/jira/browse/SOLR-17642
Project: Solr
Issue Type: Bug
Reporter: Varun Thacker
First reported on
[https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh|https://lists.apache.org/thread/xn1xg5szxhrs0sbcy4gmx4cvohy6flvh,],
Dr. Andreas Moll reported a failing query.
I was able to reproduce is consistently locally on 9.7 and 9.8 using Dr.
Andreas Moll's repro steps.
There are two bugs in play here
*The Solr side bug:*
* Solr 9.7 added https://issues.apache.org/jira/browse/SOLR-13350 but disabled
multi-threaded search by default. I can confirm the failing query follows this
code path
{code:java}
if (!MultiThreadedSearcher.allowMT(pf.postFilter, cmd)) {
log.trace("SINGLE THREADED search, skipping collector manager in
getDocListNC");{code}
* However when following that code path the query goes to
AbstractKnnVectorQuery and this is where executor thread pool gets used
{code:java}
TimeLimitingKnnCollectorManager knnCollectorManager =
new TimeLimitingKnnCollectorManager(
getKnnCollectorManager(k, indexSearcher), indexSearcher.getTimeout());
TaskExecutor taskExecutor = indexSearcher.getTaskExecutor();
{code}
* This executor was added in SOLR-13350 in CoreContainer(
indexSearcherExecutor) . If you follow the SolrIndexSearcher code that calls
{code:java}
In SolrIndexSearcher.java
super(wrapReader(core, r),
core.getCoreContainer().getIndexSearcherExecutor());{code}
{code:java}
In IndexSearcher.java
this.executor = executor;
this.taskExecutor =
executor == null ? new TaskExecutor(Runnable::run) : new
TaskExecutor(executor);{code}
I think the idea in SOLR-13350 was create an indexSearcherExector, but only use
it when multiThreaded=true as a query param (default=false). The fact we
initialised indexSearcherExector means all KNN queries use that thread pool and
there is no way to disable that.
*Now the Lucene Bug:*
[~benwtrent] noted on
[https://lists.apache.org/thread/ftm6f32f2nsoys6noxx8p2ygpwnlfjc9] that
Multi-threaded vector search over multiple segments can lead to inconsistent
results
So the fact Solr by default always uses multi threaded search 9.7+ triggers the
lucene bug. But I think in Solr we should also have a way to not use
multi-threaded KNN like we do for other queries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
