Marvin Humphrey wrote:
These processes are forbidden from merging any files that pre-date
the
establishment of "consolidate.lock".
Why? It seems like it needs to merge segments created before it
acquired
that lock (that's why it was launched).
I was unclear. By "these processes", I meant write processes
launched while
the consolidation process is active.
OK makes sense. You basically split the index segments into 2 sets;
the first set, only consolidator can change; the 2nd set, only your
primary writer
can change.
It finishes that task, commits, releases "write.lock", releases
"consolidate.lock",then exits.
That, and update the master "segments" file to actually record the
merge,
and incRef/decRef to delete files.
By "commits", I was referring to updating the master file; I
probably should
have used more precise language. You're right that I'd left off file
deletion.
OK
But, what if while a large merge is happening, and enough segments
have
been written to warrant a small merge to kick off & finish?
That should work OK. Write processes launched during consolidation
will be
allowed to performing any mods or merges they like on segments that
were
written *after* the the consolidator grabbed consolidate.lock. They
just
can't touch stuff that the consolidator might be operating on.
OK, so writer can choose to do merging if it wants (I was under the
impression
only consolidator could).
The only thing I'm concerned about is the time that it takes to
carry recent
deletions against merged segments forward so that they apply against
the new,
consolidated segment. The consolidator process has to obtain the
write.lock
for that, blocking any new mods to the index. Hopefully that
doesn't cause
too big a blip.
Ahh, right. Lucene also has a blip, but it's different because Lucene
will still accept added/deleted documents; but, one cannot reopen
a new realtime (LUCENE-1516) reader during the blip.
Hang on: does your writer process hold onto the write lock the whole
time it's open? Or it only grabs it when it needs to commit a change?
Mike
