[
https://issues.apache.org/jira/browse/LUCENE-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837003#comment-16837003
]
Simon Willnauer commented on LUCENE-8757:
-----------------------------------------
{quote}
I think there is an important justification for the 2nd criteria (number of
segments in each work unit / slice), which is if you have an index with some
large segments, and then with a long tail of small segments (easily happens if
your machine has substantially CPU concurrency and you use multiple threads),
since there is a fixed cost for visiting each segment, if you put too many
small segments into one work unit, those fixed costs multiply and that one work
unit can become too slow even though it's not actually going to visit too many
documents.
I think we should keep it?
{quote}
fair enough. lets add it back
> Better Segment To Thread Mapping Algorithm
> ------------------------------------------
>
> Key: LUCENE-8757
> URL: https://issues.apache.org/jira/browse/LUCENE-8757
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Atri Sharma
> Priority: Major
> Attachments: LUCENE-8757.patch, LUCENE-8757.patch, LUCENE-8757.patch
>
>
> The current segments to threads allocation algorithm always allocates one
> thread per segment. This is detrimental to performance in case of skew in
> segment sizes since small segments also get their dedicated thread. This can
> lead to performance degradation due to context switching overheads.
>
> A better algorithm which is cognizant of size skew would have better
> performance for realistic scenarios
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
