[
https://issues.apache.org/jira/browse/LUCENE-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988614#action_12988614
]
Michael McCandless commented on LUCENE-2898:
--------------------------------------------
bq. If however MP returns N + 1 merges, then CMS will register N + 1 merges,
and then stall?
Right, now I allow N+1 merges to be running, and then all incoming threads will
stall.
We could change that (remove the +1) but I didn't like that because then we
stall app threads when exactly the max allowed number of merges are running.
bq. Before your changes, at most maxMergeCount merges could be registered
No, before my change, maxMergeCount *plus* the number of app threads you send
into IW, could be registered. (This was the bug -- that our "last line of
defense" was vulnerable to how many app threads you send in).
bq. And my question I think is stil valid - what if CMS waits for nothing -
meaning after wait() finishes, IW.getNextMerge() returns null? The user thread
will be blocked until then, even though no more merges are 'pending'?
That's exactly the case I'd expect to happen. It's not "for nothing" -- it's
very much to stall the thread that's responsible for causing new segments to be
added. The fact that after the stall there are likely no new merges to do is
fine... the app thread runs free and then produces more segments again.
Really, if we could somehow cutover to a blocking queue... where MergePolicy is
the producer and MergeScheduler consumes, that'd be cleaner.
> CMS merge throttling is not aggressive enough
> ---------------------------------------------
>
> Key: LUCENE-2898
> URL: https://issues.apache.org/jira/browse/LUCENE-2898
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.1, 4.0
>
>
> I hit this crab while working on the NRT benchmarker (in luceneutil).
> CMS today forcefully idles any incoming threads, when there are too many
> merges pending.
> This is the last line of defense that it has, since it also juggles thread
> priorities (and forcefully idles the biggest merges) to try to reduce the
> outstanding merge count.
> But when it cannot keep up it has no choice but to stall those threads
> responsible for making new segments.
> However, the logic is in the wrong place now -- the stalling happens after
> pulling the next merge from IW. This is poor because it means if you have N
> indexing threads, you allow max + N outstanding merges.
> I have a simple fix, which is to just move the stall logic to before we pull
> the next merge from IW.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
