Uwe, Thanks for the detailed information. Are you aware of an existing implementation of the IndexDeletionPolicy interface that is "known" to work reliably with NFS? /Jong On Tue, Oct 2, 2012 at 9:01 AM, Uwe Schindler <u...@thetaphi.de> wrote:
> There are no real issues with NFS regarding safety of the data. The > problem with NFS is the following (maybe it is fixed in NFS4, I have no > idea): > Lucene deletes index files while they are in use, which is perfectly fine > for local file systems (because the inode is still alive, although it is no > longer appearing in directory listing). Unfortunately the deletes of those > index files are not visible to the directory listing asap when using NFS; > also newly added files are not always showing up in the directory listing > once created. This causes problems with Lucene like file not found > exceptions. Also the index directory locking does not work (it times out, > because NativeFSLockFactory does not work with NFS - which is a somehow a > bug in NFS). > > To use it with NFS make sure: > - Use a custom deletion policy on IndexWriter, so unused files are not > deleted asap ( > https://lucene.apache.org/core/3_6_1/api/all/org/apache/lucene/index/IndexDeletionPolicy.html > ) > - Use SimpleFSLockFactory > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Paul Libbrecht [mailto:p...@hoplahup.net] > > Sent: Tuesday, October 02, 2012 2:45 PM > > To: java-user@lucene.apache.org > > Subject: Re: Lucene index on NFS > > > > I doubt NFS is an unreliable file-system. > > Lucene uses normal random access to files and this has no reason to be > > unreliable unless bad things such as network drops happen (in which case > you'd > > get direct failures or timeouts rather than corruption). I've seen > fairly large > > infrastructures being based on NFS and corruption is something I've never > > heard about. > > > > Note: no concurrent access to a lucene index, right? > > > > Paul > > > > > > Le 2 oct. 2012 à 14:01, Jong Kim a écrit : > > > > > Thank you all for reply. > > > > > > So it soudns like it is a known fact that the performance would suffer > > > rather significantly when the index files are accessed over NFS. But > > > how about reliability and robustness (which seems even more > > > important)? Isn't there any increased possibility for intermittent > > > errors such as index file corruption (due to cache inconsistency, > > > difference in delete semantics, > > > etc.) when using NFS? Has anyone run into such trouble? Or is it > > > strictly just a performance issue? > > > > > > /Jong > > > On Tue, Oct 2, 2012 at 5:17 AM, Paul Libbrecht <p...@hoplahup.net> > wrote: > > > > > >> My experience in the Lucene 1.x times were a factor of at least four > > >> in writing to NFS and about two when reading from there. I'd > > >> discourage this as much as possible! > > >> > > >> (rsync is way more your friend for transporting and replication à la > > >> solr should also be considered) > > >> > > >> paul > > >> > > >> > > >> Le 2 oct. 2012 à 11:10, Ian Lea a écrit : > > >> > > >>> You'll certainly need to factor in the performance of NFS versus > > >>> local > > >> disks. > > >>> > > >>> My experience is that smallish low activity indexes work just fine > > >>> on NFS, but large high activity indexes are not so good, > > >>> particularly if you have a lot of modifications to the index. > > >>> > > >>> You may want to install a custom IndexDeletionPolicy. See the > > >>> javadocs for details with specific reference to NFS. > > >>> > > >>> > > >>> -- > > >>> Ian. > > >>> > > >>> On Tue, Oct 2, 2012 at 3:21 AM, Vitaly Funstein > > >>> <vfunst...@gmail.com> > > >> wrote: > > >>>> How tolerant is your project of decreased search and indexing > > >> performance? > > >>>> You could probably write a simple test that compares search and > > >>>> write speeds of local and NFS-mounted indexes and make the decision > > >>>> based on > > >> the > > >>>> results. > > >>>> > > >>>> On Mon, Oct 1, 2012 at 3:06 PM, Jong Kim <jong.luc...@gmail.com> > > wrote: > > >>>> > > >>>>> Hi, > > >>>>> > > >>>>> According to the Lucene In Action (Second Edition), the section > > >>>>> 2.11.2 "Accessing an index over a remote file system" explains > > >>>>> that there are issues related to accessing a Lucene index across > > >>>>> remote file system including NFS. > > >>>>> > > >>>>> I'm particuarly interested in NFS compatibility, and wondering if > > >> there has > > >>>>> been any work done to solve or mitigate this problem. Has this > > >>>>> issue > > >> been > > >>>>> addressed? If not, are there some reliable work-arounds that make > > >>>>> this possible at the expense of some sacrifice in other areas? > > >>>>> > > >>>>> Any information would be greatly appreciated, since my project > > >>>>> heavily depends on the feasibility of this. > > >>>>> > > >>>>> Thanks > > >>>>> /Jong > > >>>>> > > >>> > > >>> -------------------------------------------------------------------- > > >>> - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > >>> For additional commands, e-mail: java-user-h...@lucene.apache.org > > >>> > > >> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > >> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
