[
https://issues.apache.org/jira/browse/LUCENE-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790988#action_12790988
]
Michael McCandless commented on LUCENE-2026:
--------------------------------------------
{quote}
bq. I think that's a poor default (trades safety for performance), unless Lucy
eg uses a transaction log so you can concretely bound what's lost on
crash/power loss. Or, if you go back to autocommitting I guess...
Search indexes should not be used for canonical data storage - they should be
built on top of canonical data storage.
{quote}
I agree with that, in theory, but I think in practice it's too
idealistic to force/expect apps to meet that ideal.
I expect for many apps it's a major cost to unexpectedly lose the
search index on power loss / OS crash.
{quote}
Users have many options for dealing with the potential for such corruption.
You can go back to your canonical data store and rebuild your index from
scratch when it happens. In a search cluster environment, you can rsync a
known-good copy from another node. Potentially, you might enable
fsync-before-commit and keep your own transaction log. However, if the time
it takes to rebuild or recover an index from scratch would have caused you
unacceptable downtime, you can't possibly be operating in a
single-point-of-failure environment where a power failure could take you down
anyway - so other recovery options are available to you.
Turning on fsync is only one step towards ensuring index integrity; others
steps involve making decisions about hard drives, RAID arrays, failover
strategies, network and off-site backups, etc, and are outside of our domain
as library authors. We cannot meet the needs of users who need guaranteed
index integrity on our own.
{quote}
Yes, high availability apps will already take their measures to
protect the search index / recovery process, going beyond fsync.
EG, making a hot backup of Lucene index is now straightforwarded.
{quote}
For everybody else, what turning on fsync by default achieves is to make an
exceedingly rare event rarer. That's valuable, but not essential. My
argument is that since the search indexes should not be used for canonical
storage, and since fsync is not testably reliable and not sufficient on its
own, it's a good engineering compromise to prioritize performance.
{quote}
Losing power to the machine, or OS crash, or the user doing a hard
power down because OS isn't responding, I think are not actually
*that* uncommon in an end user setting. Think of a desktop app
embedding Lucene/Lucy...
{quote}
bq. If we did this in Lucene, you can have unbounded corruption. It's not just
the last few minutes of updates...
Wasn't that a possibility under autocommit as well? All it takes is for the
OS to finish flushing the new snapshot file to persistent storage before it
finishes flushing a segment data file needed by that snapshot, and for the
power failure to squeeze in between.
{quote}
Not after LUCENE-1044... autoCommit simply called commit() at certain
opportune times (after finish big merges), which does the right thing
(I hope!). The segments file is not written until all files it
references are sync'd.
{quote}
In practice, locality of reference is going to make the window very very
small, since those two pieces of data will usually get written very close to
each other on the persistent media.
{quote}
Not sure about that -- it depends on how effectively the OS's write cache
"preserves" that locality.
{quote}
I've seen a lot more messages to our user lists over the years about data
corruption caused by bugs and misconfigurations than by power failures.
{quote}
I would agree, though, I think it may be a sampling problem... ie
people whose machines crashed and they lost the search index would
often not raise it on the list (vs say a persistent config issue that keeps
leading to corruption).
{quote}
But really, that's as it should be. Ensuring data integrity to the degree
required by a database is costly - it requires far more rigorous testing, and
far more conservative development practices. If we accept that our indexes
must never go corrupt, it will retard innovation.
{quote}
It's not really that costly, with NRT -- you can get a searcher on the
index without paying the commit cost. And now you can call commit
however frequently you need to. Quickly turning around a new
searcher, and how frequently you commit, are now independent.
Also, having the app explicitly decouple these two notions keeps the
door open for future improvements. If we force absolutely all sharing
to go through the filesystem then that limits the improvements we can
make to NRT.
{quote}
Of course we should work very hard to prevent index corruption. However, I'm
much more concerned about stuff like silent omission of search results due to
overzealous, overly complex optimizations than I am about problems arising
from power failures. When a power failure occurs, you know it - so you get
the opportunity to fsck the disk, run checkIndex(), perform data integrity
reconciliation tests against canonical storage, and if anything fails, take
whatever recovery actions you deem necessary.
{quote}
Well... I think search performance is important, and we should pursue it
even if we risk bugs.
{quote}
bq. You don't need to turn off sync for NRT - that's the whole point. It gives
you a reader without syncing the files.
I suppose this is where Lucy and Lucene differ. Thanks to mmap and the
near-instantaneous reader opens it has enabled, we don't need to keep a
special reader alive. Since there's no special reader, the only way to get
data to a search process is to go through a commit. But if we fsync on every
commit, we'll drag down indexing responsiveness. Fishishing the commit and
returning control to client code as quickly as possible is a high priority for
us.
{quote}
NRT reader isn't that special -- the only things different is 1) it
loaded the segments_N "file" from IW instead of the filesystem, and 2)
it uses a reader pool to "share" the underlying SegmentReaders with
other places that have loaded them. I guess, if Lucy won't allow
this, then, yes, forcing a commit in order to reopen is very costly,
and so sacrificing safety is a tradeoff you have to make.
Alternatively, you could keep the notion "flush" (an unsafe commit)
alive? You write the segments file, but make no effort to ensure it's
durability (and also preserve the last "true" commit). Then a normal
IR.reopen suffices...
{quote}
Furthermore, I don't want us to have to write the code to support a
near-real-time reader hanging off of IndexWriter a la Lucene. The
architectural discussions have made for very interesting reading, but the
design seems to be tricky to pull off, and implementation simplicity in core
search code is a high priority for Lucy. It's better for Lucy to kill two
birds with one stone and concentrate on making all index opens fast.
{quote}
But shouldn't you at least give an option for index durability? Even
if we disagree about the default?
{quote}
bq. Really, this is your safety tradeoff - it means you can commit less
frequently, since the NRT reader can search the latest updates. But, your app
has complete control over how it wants to to trade safety for performance.
So long as fsync is an option, the app always has complete control,
regardless of whether the default setting is fsync or no fsync.
{quote}
Well it is an "option" in Lucene -- "it's just software" ;) I don't
want to make it easy to be unsafe. Lucene shouldn't sacrifice safety
of the index... and with NRT there's no need to make that tradeoff.
{quote}
If a Lucene app wanted to increase NRT responsiveness and throughput, and if
absolute index integrity wasn't a concern because it had been addressed
through other means (e.g. multi-node search cluster), would turning off fsync
speed things up under any of the proposed designs?
{quote}
Yes, turning off fsync would speed things up -- you could fall back to
simple reopen and get good performance (NRT should still be faster
since the readers are pooled). The "use RAMDir on top of Lucene"
designs would be helped less since fsync is a noop in RAMDir.
> Refactoring of IndexWriter
> --------------------------
>
> Key: LUCENE-2026
> URL: https://issues.apache.org/jira/browse/LUCENE-2026
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
>
> I've been thinking for a while about refactoring the IndexWriter into
> two main components.
> One could be called a SegmentWriter and as the
> name says its job would be to write one particular index segment. The
> default one just as today will provide methods to add documents and
> flushes when its buffer is full.
> Other SegmentWriter implementations would do things like e.g. appending or
> copying external segments [what addIndexes*() currently does].
> The second component's job would it be to manage writing the segments
> file and merging/deleting segments. It would know about
> DeletionPolicy, MergePolicy and MergeScheduler. Ideally it would
> provide hooks that allow users to manage external data structures and
> keep them in sync with Lucene's data during segment merges.
> API wise there are things we have to figure out, such as where the
> updateDocument() method would fit in, because its deletion part
> affects all segments, whereas the new document is only being added to
> the new segment.
> Of course these should be lower level APIs for things like parallel
> indexing and related use cases. That's why we should still provide
> easy to use APIs like today for people who don't need to care about
> per-segment ops during indexing. So the current IndexWriter could
> probably keeps most of its APIs and delegate to the new classes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
