On 2/26/09 1:50 PM, Michael McCandless wrote:
Michael Busch wrote:
On 2/24/09 4:05 AM, Michael McCandless wrote:
I believe we still need this, for remote filesystems (like NFS) that
have inconsistent client-side caching.
The fsync() ensures the local IO system has moved the bytes/file
metadata to stable storage, but I'd expect remote caches would still
potentially be stale.
We could have an expert API to turn using the .gen file on/off? And
then default it to off in 3.0.
We could do that, though I'd default to leaving it on?
I said off, because I think most people don't use NFS (and we don't even
recommend using it for performance reasons). But defaulting to on would
be ok too, to be on the safe side.
I think the .gen file and the CompoundFileWriter are the only places
left where we overwrite (parts of) files? To change the latter we
could move the cfs header to the segments file.
TermInfosWriter also seeks & writes a header; see here (enabling
Lucene to write directly to HDFS):
http://issues.apache.org/jira/browse/LUCENE-532
What situation are you seeing that requires absolute "write once"?
Actually I personally don't need Lucene to be "write once". The reason
why I started this thread about the segments.gen file was that in our
project we sometimes need to rollback to a previous commit-point (using
Lucene 2.4.0) that we keep around with the SnapshotDeletionPolicy. To
get rid of the newest commit-point we simply delete the most recent
segments file. But then we also have to delete the segments.gen file,
otherwise Lucene will read the generation from it and try to find the
segments file we deleted. Then Lucene will recreate the segments.gen
file. This just made me think that this is not very clean (deleting and
recreating the segments.gen) especially because we use a local FS and
don't even need the .gen file.
I mentioned the "write once" issue because I think with disabling the
.gen file and all your recent changes (especially lockless commits)
we've almost accomplished "write once", so we might as well change the
last two places that don't comply (CFSWriter and TermInfosWriter), to
resolve LUCENE-532. Not sure how many user there are out there who
really need it though.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
