Re: FSDirectory patch for file renaming

[Scott Ganyo]

I'm ok with this, given the lack of a better option.  Please ensure that 
the input and output streams are closed in a finally block, though.

Scott

Otis Gospodnetic wrote:

I hate having to make the suggested change for the reasons Matt
mentioned as disadvantages below....but what is one to do?
I'll change the code shortly.
If any other developers are against this change, speak up.

Otis

--- Matt Tucker <[EMAIL PROTECTED]> wrote:
 

Hello all,

For quite some time, we've been experiencing intermittent indexing
failures
that look similar to the following:
java.io.IOException: couldn't rename segments.new to segments 
    at

   

org.apache.lucene.store.FSDirectory.renameFile(Ljava.lang.String;Ljava.lang.
 

String)V(Unknown Source) 
    at

   

org.apache.lucene.index.SegmentInfos.write(Lorg.apache.lucene.store.Director
 

y)V(Unknown Source)

This is caused by File.renameTo failing. From previous run-ins with
that
method, it's actually fairly flaky, especially under various Windows
JVMs.
The fix is pretty easy -- if File.renameTo fails, manually copy the
file
from the old to new destination.
Old code (FSDirectory.renameFile): 

------------------------------
   if (!old.renameTo(nu))
     throw new IOException("couldn't rename " + from + " to " + to);
------------------------------
New code ("buf" variable is declared at top of file as null):

------------------------------
   // Rename the old file to the new one. Unfortunately, the
renameTo()
method doesn't
   // work reliably under some JVMs. Therefore, if the rename fails,
we
manually rename by
   // copying the old file to the new one.
   if (!old.renameTo(nu)) {
     try {
       java.io.InputStream in = new FileInputStream(old);
       java.io.OutputStream out = new FileOutputStream(nu);
       int len;
       // See if the buf needs to be initialized. Initialization is
only
done on-demand
       // since many VM's will never run into the renameTo bug and
hence
shouldn't waste
       // 1K of mem for no reason.
       if (buf == null) {
         buf = new byte[1024];
       }
       while ((len = in.read(buf)) >= 0) {
          out.write(buf, 0, len);
       }
       in.close();
       out.close();
       // Delete the old file.
       old.delete();
     }
     catch (IOException ioe) {
       throw new IOException("couldn't rename " + from + " to " +
to);
     }
   }
------------------------------
Advantages to this fix:

* Makes indexing more reliable, especially under VM's where the
renameTo
method doesn't always work.
* Uses no additional memory or resources when File.renameTo works
normally.
Disadvantages:

* It always sucks to add workarounds to bugs in other libraries, but
what
is one to do?
* We've run into File.renameTo issues on multiple machines and VM
versions,
but it would be good if others could confirm they've had problems as
well. 

I'd suggest this fix as a good candidate for Lucene 1.3 unless others
see
problems with it.
Regards,
Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
   



__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
 

--
"Perfection in design is not achieved when there is nothing more to add, but when there is 
nothing more to remove." - Antoine de Saint-Exupéry


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]