Pretty much pure searching. Not reopening readers. Keeping the reader open for the test.
-John On Mon, May 3, 2010 at 2:47 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > This is strange -- it's completely opposite of what others have seen! > > Other have seen sizable concurrency gains when switching from > SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing > JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734) > prevents concurrency gains. > > There must be something different about the testing. > > John are you testing pure searching? Are you reopening readers as > well? How much HW concurrency do your machines have? > > Mike > > On Sun, May 2, 2010 at 11:53 PM, John Wang <john.w...@gmail.com> wrote: > > Solaris and MacOs > > > > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <markrmil...@gmail.com> > wrote: > >> > >> Interesting ... whats the OS? > >> > >> On Sun, May 2, 2010 at 7:12 PM, John Wang <john.w...@gmail.com> wrote: > >>> > >>> 12 concurrent search threads in a stress environment. (with about 5M > doc > >>> index) > >>> -John > >>> > >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmil...@gmail.com> > >>> wrote: > >>>> > >>>> Perfectly safe to use SimpleFSDirectory. When you measure the > >>>> performance, are you measuring a lot of concurrent requests? That's > where > >>>> you should see the win. > >>>> > >>>> - Mark > >>>> > >>>> On 5/1/10 6:38 PM, John Wang wrote: > >>>>> > >>>>> We are seeing this issue as well in your production. (using Zoie on > top > >>>>> of lucene 2.9.1) > >>>>> After some performance comparisons, we do NOT see performance gain > with > >>>>> NIO, rather these nasty ClosedChannelExceptions. > >>>>> > >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due to > >>>>> many different things. From what we seen, they are not related to > >>>>> NIOFSDirectory. > >>>>> > >>>>> Our solution is to avoid calling FSDirectory.open(), instead just > call > >>>>> new SimpleFSDirectory(). Is this safe? > >>>>> > >>>>> -John > >>>>> > >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller <markrmil...@gmail.com > >>>>> <mailto:markrmil...@gmail.com>> wrote: > >>>>> > >>>>> Perhaps - one of the things they are supposed to be addressing is > >>>>> extendability. > >>>>> > >>>>> nio2 does have FileSystemProvider, which would actually allow you > to > >>>>> create a custom channel ! > >>>>> > >>>>> I have not dug in enough to know much more than that though. > >>>>> > >>>>> *But*, another really interesting thing is that in Java 7, > >>>>> FileDescriptors are ref counted ! (though users can't inc/dec). > >>>>> > >>>>> But, FileInputStream and OutputStream have a new constructor that > >>>>> takes > >>>>> a FileDescriptor. > >>>>> > >>>>> So possibly, you could just make one that sits around to keep the > >>>>> FileDescriptor valid, and get your channel off > >>>>> FileInputStream/FileOutputStream? > >>>>> > >>>>> And then if it goes down, make a new one using the FileDescriptor > >>>>> which > >>>>> was not actually closed because there was a still a ref to it. > >>>>> > >>>>> Possibly .... ;) > >>>>> > >>>>> Michael McCandless wrote: > >>>>> > Does anyone know if nio2 has improved this...? > >>>>> > > >>>>> > Mike > >>>>> > > >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen > >>>>> > <jason.rutherg...@gmail.com <mailto:jason.rutherg...@gmail.com > >> > >>>>> wrote: > >>>>> > > >>>>> >> Defaulting NIOFSDir could account for some of the recent speed > >>>>> >> improvements users have been reporting in Lucene 2.9. So > >>>>> removing it > >>>>> >> as a default could reverse those and people could then report > >>>>> Lucene > >>>>> >> 3.X has slowed... > >>>>> >> > >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless > >>>>> >> <luc...@mikemccandless.com <mailto:luc...@mikemccandless.com > >> > >>>>> wrote: > >>>>> >> > >>>>> >>> Bummer. > >>>>> >>> > >>>>> >>> So the only viable workarounds are 1) don't use > >>>>> Thread.interrupt (nor, > >>>>> >>> things like Future.cancel, which in turn use > Thread.interrupt) > >>>>> with > >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND the > >>>>> app must > >>>>> >>> make a deletion policy that keeps a commit alive if any > reader > >>>>> is > >>>>> >>> using it. Or, 3) don't use NIOFSDir! > >>>>> >>> > >>>>> >>> Mike > >>>>> >>> > >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer > >>>>> >>> <simon.willna...@googlemail.com > >>>>> <mailto:simon.willna...@googlemail.com>> wrote: > >>>>> >>> > >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless > >>>>> >>>> <luc...@mikemccandless.com <mailto: > luc...@mikemccandless.com>> > >>>>> wrote: > >>>>> >>>> > >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler > >>>>> <u...@thetaphi.de <mailto:u...@thetaphi.de>> wrote: > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment > >>>>> was not really correct: > >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens > >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps > open > >>>>> until the file is closed (and also the channel). > >>>>> >>>>>> > >>>>> >>>>>> So it's really simple to fix in my opinion, just call > >>>>> getChannel() again on this exception. Because the RAF should still > >>>>> be open? > >>>>> >>>>>> > >>>>> >>>> Short answer: > >>>>> >>>> public final FileChannel getChannel() { > >>>>> >>>> synchronized (this) { > >>>>> >>>> if (channel == null) > >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw, > >>>>> this); > >>>>> >>>> return channel; > >>>>> >>>> } > >>>>> >>>> } > >>>>> >>>> > >>>>> >>>> this is not gonna work I tried it before. The > RandomAccessFile > >>>>> buffers > >>>>> >>>> the channel!! > >>>>> >>>> > >>>>> >>>> simon > >>>>> >>>> > >>>>> >>>>> I think we need a definitive answer on what happens to the > >>>>> RAF when > >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can > >>>>> you test > >>>>> >>>>> this? > >>>>> >>>>> > >>>>> >>>>> Mike > >>>>> >>>>> > >>>>> >>>>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> >>>>> To unsubscribe, e-mail: > >>>>> java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> >>>>> For additional commands, e-mail: > >>>>> java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> >>>> To unsubscribe, e-mail: > java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> >>>> For additional commands, e-mail: > >>>>> java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> >>> To unsubscribe, e-mail: > java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> >>> For additional commands, e-mail: > >>>>> java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> >> To unsubscribe, e-mail: > java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> >> For additional commands, e-mail: > java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> > > >>>>> > > >>>>> --------------------------------------------------------------------- > >>>>> > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> > For additional commands, e-mail: > java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> > > >>>>> > > >>>>> > >>>>> > >>>>> -- > >>>>> - Mark > >>>>> > >>>>> http://www.lucidimagination.com > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> > >>>>> For additional commands, e-mail: java-dev-h...@lucene.apache.org > >>>>> <mailto:java-dev-h...@lucene.apache.org> > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> - Mark > >>>> > >>>> http://www.lucidimagination.com > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>>> > >>> > >> > >> > >> > >> -- > >> - Mark > >> > >> http://www.lucidimagination.com > >> > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
