[
https://issues.apache.org/jira/browse/LUCENE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841078#action_12841078
]
Shai Erera commented on LUCENE-1879:
------------------------------------
(Warning, this post is long, and is easier to read in JIRA)
I've investigated the attached code a lot and I'd like to propose a different
design and approach to this whole Parallel Index solution. I'll start by
describing the limitations of the current design (whether its the approach or
the code is debatable):
* Lucene is not built/designed properly to a Master/Slave architecture, where
different indexes share important files with others (such as segments_N,
segments.gen and .del).
** I've realized this when I found that if tests (in this patch) are run with
"-ea", there are many assert exceptions that are printed from
IndexWriter.startCommit. The reason is the Master just updated one of the
segments .del generation (and deleted the previous one), but the Slave is not
aware of that yet and looks for the wrong .del file. While this does not run on
production (e.g. "-ea" is usually not activated), it does affect the tests
because the assertion stops operations abruptly.
** Though someone can claim we can fix that, I think it points at a problem in
the design, and makes the whole solution fragile.
* I think it'd be really neat to introduce a ParallelWriter, equivalent to
ParallelReader. The latter does not have a Master/Slave notion and so I don't
think PW should have.
* When I inspected the code carefully, I realized there are lots of hoola hoops
done in order to make the Master and Slave in sync. Such hoola hoops may be
resolved if Lucene's IW API would be more extensible, but still:
** The MergePolicy used is one that for the Slaves never checks the segments
for which merges should actually be done. Rather, it relies on the Master
policy to set the proper merges. Which is a must in this design because only
the master needs to decide when to merge.
** However, and here I think it's because of lack of API on IW, the way the
merge is done is that the master first calls mergeInit(merge), then on all
slaves .maybeMerge() and then it merges that merge. maybeMerge() on the slaves
consume all the merges that were decided to be run by the master, while when
that finished, the master didn't finish even one merge ...
** That works though because the MergeScheduler used is a Serial one (not SMS
but still Serial) and blocking. However that leads to inconsistencies - slaves'
segments view is different at one point in time from the master's.
* The current approach does not support multi-threaded indexing, but I think
that's a limitation that could be solved by exposing some API on IW or DW.
* Only SMS is supported on the slaves.
* Optimize, expungeDeletes are unsupported. Though the could and perhaps just
not implemented.
* The current approach prevents having an architecture on which some of the
parallels reside on different machines, because they share the .del and
segments file with the master. It's not the worse limitation in the world, but
still a limitation (of having any chance to do it efficiently) I'd like to
avoid.
* And I'm sure there are more disadvantages that I don't remember now.
I'd like to point out that even if the above limitations can be worked around,
I still think the Master and Slave notion is not the best approach. At least,
I'd like to propose a different approach:
* Introduce a ParallelWriter which serves as a manager/orchestrator on top of
other IWs. It is not a pure decorator because it drives everything that happens
on the IWs, but it does not contain any actual indexing logic (e.g.
add/delete/update documents).
** The IWs PW will manage will be named hereinafter Slices.
* IW will expose enough API to perform two-phase operations, like the two-phase
commit one can achieve today. Example operations (and I don't cover all for the
interest of space):
** addDocument - first obtain a doc ID, then proceed w/ addDocument on all
Slices
** optimize - already exists
** merge - do the merge on all Slices and stamp it after all finished.
** deleteDocuments - here we would need to expose some API on IW for DW to get
an IndexReader so that IW can still return its readerPool.getReader but PW will
return a ParallelSegmentReader or something, to perform the deletes across all
Slices.
** The idea is that we should do enough on the Slices so that if one fails we
can still rollback, and the final 'stamp' process will be very fast and less
likely to fail.
* For correctness and protectiveness, PW will only accept a Directory and not
IW. Reason is:
** We want all sorts of settings like MP, MS, RAM buffer usage to be controlled
by PW and not on the Slices. If we allow to pass an IW instance, one could
override whatever we set, which is wrong.
** Even though one could claim that someone 'can shoot himself in the leg
freely', I think that we should be resilient enough to protect stupid users
from themselves.
** We will need to allow to pass in an IW Configuration object, so that we can
still account for settings such as Analyzer, MaxFieldLength etc., but discard
other settings which PW will control directly
*** Such Configuration was proposed in the past already and will eliminate lots
of methods on IW and ctors.
** On a side note, ParallelReader accepts IR today, which can lead the problems
such as one passes two IRs, one read-only and one not, and then deletes
documents by the writable IR, with PR not knowing about it. But it's a
different issue, and I'll open a separate one for that.
* A special MergeScheduler and MergePolicy will be introduced to allow PW to
drive merges across the Slices. The idea is to support whatever MS/MP the
application wants (SMS, CMS, custom), and ensuring that when MP decides a merge
should be performed, that merge is executed by MS across all Slices. Few things:
** I think that a special MP is not needed, only MS. But need to validate that.
If that's true, then apps could use their own custom MPs freely.
** I think custom MS may be supported ... all that's required is for the MS to
run on PW and whenever it calls its merge(), let PW run the merges across all
Slices? But I still need to validate that code.
** CMS can introduce two-level concurrency. One like today which executes
different merges decided by MP concurrently. The other would control the
concurrency level those merges are executed on the Slices.
*** Hmm ... even SMS can benefit from that ...
I realize that accepting only Directory on PW might limit applications who want
to pass in their own IW extension, for whatever reason. But other than saying
"if you pass in IW and configure it afterwards, it's on your head", I don't
think there is any other option ... Well maybe except if we expose a
package-private API for PW to turn off configuration on an IW after it set it,
so successive calls to the underlying IW's setters will throw an exception ...
hmm might be doable. I'll look into that. If that will work, we might want to
do the same for the ParallelReader as well.
Michael mentioned a scenario above where one would want to rebuild an index
Slice. That's still achievable by this design - one should build the IW on the
outside and then replace the Directory instance on PW. We'll need to expose
such API as well.
BTW, some of the things I've mentioned can be taken care of in different
issues, as follow on improvements, such as two-level concurrency, supporting
custom MS etc. I've detailed them here just so we all see the bigger picture
that's going on in my head.
I think I wrote all (or most) of the high-level details. I'd like to start
implementing this soon. In my head it's all chewed and digested, so I feel I
can start implementing today. If possible, I'd like to get this out in 3.1.
I'll try to break this issue down to as many issues as I can, to make the
contributions containable. We should just keep in mind for each such issue the
larger picture it solves.
I'd appreciate your comments.
> Parallel incremental indexing
> -----------------------------
>
> Key: LUCENE-1879
> URL: https://issues.apache.org/jira/browse/LUCENE-1879
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Fix For: 3.1
>
> Attachments: parallel_incremental_indexing.tar
>
>
> A new feature that allows building parallel indexes and keeping them in sync
> on a docID level, independent of the choice of the MergePolicy/MergeScheduler.
> Find details on the wiki page for this feature:
> http://wiki.apache.org/lucene-java/ParallelIncrementalIndexing
> Discussion on java-dev:
> http://markmail.org/thread/ql3oxzkob7aqf3jd
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
