[
https://issues.apache.org/jira/browse/LUCENE-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845843#comment-16845843
]
Atri Sharma commented on LUCENE-8808:
-------------------------------------
[~jpountz] Let me elaborate by taking an example:
Consider a scenario where the underlying hardware is a 16 vCPU machine. There
is a single large index (approx 450 GB).
Now, N concurrent IndexSearchers sharing the same executor are firing different
queries. This could mean that some queries could have X number of slices, where
X >> 16. Other queries could need number of slices << 16.
If we had a hard cap of the number of slices a single query can have, that will
mean that the large query will be bound by the N.
I agree that if there is a sizeable percentage of the overall query set which
will ideally need slices >> than the number of cores present, then eventually
the executor's queue will build up. However, that seems more like a problem of
hardware underprovisioning than a limitation in Lucene?
The essential idea is to have a best-effort way to limit the amount of
resources a single query can use. As you highlighted, this will not cover all
edge cases, but should be better than nothing. WDYT?
> Introduce Optional Cap on Number Of Threads Per Query
> -----------------------------------------------------
>
> Key: LUCENE-8808
> URL: https://issues.apache.org/jira/browse/LUCENE-8808
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Reporter: Atri Sharma
> Priority: Major
>
> With the presence of https://issues.apache.org/jira/browse/LUCENE-8757 , a
> natural progression is to allow advanced users to specify a cap on the number
> of threads a query can use. This is especially useful for long polled
> IndexSearcher instances, where the same instance is used to fire queries for
> multiple runs and there are multiple concurrent IndexSearcher instances on
> the same node.
>
> This will be an optional parameter and local only to the IndexSearcher
> instance being configured during construction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
