12 concurrent search threads in a stress environment. (with about 5M doc index)
-John On Sun, May 2, 2010 at 3:13 PM, Mark Miller <[email protected]> wrote: > Perfectly safe to use SimpleFSDirectory. When you measure the performance, > are you measuring a lot of concurrent requests? That's where you should see > the win. > > - Mark > > > On 5/1/10 6:38 PM, John Wang wrote: > >> We are seeing this issue as well in your production. (using Zoie on top >> of lucene 2.9.1) >> After some performance comparisons, we do NOT see performance gain with >> NIO, rather these nasty ClosedChannelExceptions. >> >> I think the performance gains ppl are seeing with 2.9.1 can be due to >> many different things. From what we seen, they are not related to >> NIOFSDirectory. >> >> Our solution is to avoid calling FSDirectory.open(), instead just call >> new SimpleFSDirectory(). Is this safe? >> >> -John >> >> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <[email protected] >> <mailto:[email protected]>> wrote: >> >> Perhaps - one of the things they are supposed to be addressing is >> extendability. >> >> nio2 does have FileSystemProvider, which would actually allow you to >> create a custom channel ! >> >> I have not dug in enough to know much more than that though. >> >> *But*, another really interesting thing is that in Java 7, >> FileDescriptors are ref counted ! (though users can't inc/dec). >> >> But, FileInputStream and OutputStream have a new constructor that takes >> a FileDescriptor. >> >> So possibly, you could just make one that sits around to keep the >> FileDescriptor valid, and get your channel off >> FileInputStream/FileOutputStream? >> >> And then if it goes down, make a new one using the FileDescriptor which >> was not actually closed because there was a still a ref to it. >> >> Possibly .... ;) >> >> Michael McCandless wrote: >> > Does anyone know if nio2 has improved this...? >> > >> > Mike >> > >> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen >> > <[email protected] <mailto:[email protected]>> >> >> wrote: >> > >> >> Defaulting NIOFSDir could account for some of the recent speed >> >> improvements users have been reporting in Lucene 2.9. So >> removing it >> >> as a default could reverse those and people could then report >> Lucene >> >> 3.X has slowed... >> >> >> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless >> >> <[email protected] <mailto:[email protected]>> >> >> wrote: >> >> >> >>> Bummer. >> >>> >> >>> So the only viable workarounds are 1) don't use >> Thread.interrupt (nor, >> >>> things like Future.cancel, which in turn use Thread.interrupt) >> with >> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the >> app must >> >>> make a deletion policy that keeps a commit alive if any reader is >> >>> using it. Or, 3) don't use NIOFSDir! >> >>> >> >>> Mike >> >>> >> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer >> >>> <[email protected] >> <mailto:[email protected]>> wrote: >> >>> >> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless >> >>>> <[email protected] <mailto:[email protected]>> >> >> wrote: >> >>>> >> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler >> <[email protected] <mailto:[email protected]>> wrote: >> >>>>> >> >>>>> >> >>>>>> So I checked the code of NIOFSIndexInput, my last comment >> was not really correct: >> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens >> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps open >> until the file is closed (and also the channel). >> >>>>>> >> >>>>>> So it's really simple to fix in my opinion, just call >> getChannel() again on this exception. Because the RAF should still >> be open? >> >>>>>> >> >>>> Short answer: >> >>>> public final FileChannel getChannel() { >> >>>> synchronized (this) { >> >>>> if (channel == null) >> >>>> channel = FileChannelImpl.open(fd, true, rw, >> this); >> >>>> return channel; >> >>>> } >> >>>> } >> >>>> >> >>>> this is not gonna work I tried it before. The RandomAccessFile >> buffers >> >>>> the channel!! >> >>>> >> >>>> simon >> >>>> >> >>>>> I think we need a definitive answer on what happens to the >> RAF when >> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can >> you test >> >>>>> this? >> >>>>> >> >>>>> Mike >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: >> [email protected] >> <mailto:[email protected]> >> >> >>>>> For additional commands, e-mail: >> [email protected] <mailto: >> [email protected]> >> >> >>>>> >> >>>>> >> >>>>> >> >>>> >> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> >> >>>> For additional commands, e-mail: >> [email protected] <mailto: >> [email protected]> >> >> >>>> >> >>>> >> >>>> >> >>> >> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> >> >>> For additional commands, e-mail: >> [email protected] <mailto: >> [email protected]> >> >> >>> >> >>> >> >>> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> >> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> >> >> >> >> >> >> > >> > >> --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> >> > For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> > >> > >> >> >> -- >> - Mark >> >> http://www.lucidimagination.com >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> >> > > -- > - Mark > > http://www.lucidimagination.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
