[
https://issues.apache.org/jira/browse/LUCENE-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931796#action_12931796
]
Shai Erera commented on LUCENE-2755:
------------------------------------
I've looked into integrating an ExecutorService and I think it can really
simplify things, as long as we can let go of CMS sorting merges by their size.
And I think - why should it? What if we make it to MP's decision? Namely, if
you care about which merges run first, have your MP sort them the way you want,
before you return them. If we make OneMerge comparable, that should be a very
trivial extension one has to make to MP (extend MP, override the methods, call
super.method() and then sort accordingly).
If we do that, then SMS and CMS will work the same - execute the merges in the
order returned by MP, only CMS will do so in parallel and SMS will do so
synchronously. By reading that you can already see that later (in a separate
issue), we can let go of SMS entirely - it will be a single-threaded
ExecutorService, w/ an implementation to wait until the work competes (unlike
CMS which returns immediately). But that's for another day.
What do you think?
> Some improvements to CMS
> ------------------------
>
> Key: LUCENE-2755
> URL: https://issues.apache.org/jira/browse/LUCENE-2755
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
>
> While running optimize on a large index, I've noticed several things that got
> me to read CMS code more carefully, and find these issues:
> * CMS may hold onto a merge if maxMergeCount is hit. That results in the
> MergeThreads taking merges from the IndexWriter until they are exhausted, and
> only then that blocked merge will run. I think it's unnecessary that that
> merge will be blocked.
> * CMS sorts merges by segments size, doc-based and not bytes-based. Since the
> default MP is LogByteSizeMP, and I hardly believe people care about doc-based
> size segments anymore, I think we should switch the default impl. There are
> two ways to make it extensible, if we want:
> ** Have an overridable member/method in CMS that you can extend and override
> - easy.
> ** Have OneMerge be comparable and let the MP determine the order (e.g. by
> bytes, docs, calibrate deletes etc.). Better, but will need to tap into
> several places in the code, so more risky and complicated.
> On the go, I'd like to add some documentation to CMS - it's not very easy to
> read and follow.
> I'll work on a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
