Re : HW concurrency. 8 core machine, avg latency around 30ms, so with 12 threads, more than enough room.
-John On Mon, May 3, 2010 at 10:44 AM, John Wang <john.w...@gmail.com> wrote: > Pretty much pure searching. Not reopening readers. Keeping the reader open > for the test. > > -John > > > On Mon, May 3, 2010 at 2:47 AM, Michael McCandless < > luc...@mikemccandless.com> wrote: > >> This is strange -- it's completely opposite of what others have seen! >> >> Other have seen sizable concurrency gains when switching from >> SimpleFSDir -> NIOFSDir, except on Windows where Sun's long standing >> JVM bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734) >> prevents concurrency gains. >> >> There must be something different about the testing. >> >> John are you testing pure searching? Are you reopening readers as >> well? How much HW concurrency do your machines have? >> >> Mike >> >> On Sun, May 2, 2010 at 11:53 PM, John Wang <john.w...@gmail.com> wrote: >> > Solaris and MacOs >> > >> > On Sun, May 2, 2010 at 4:47 PM, Mark Miller <markrmil...@gmail.com> >> wrote: >> >> >> >> Interesting ... whats the OS? >> >> >> >> On Sun, May 2, 2010 at 7:12 PM, John Wang <john.w...@gmail.com> wrote: >> >>> >> >>> 12 concurrent search threads in a stress environment. (with about 5M >> doc >> >>> index) >> >>> -John >> >>> >> >>> On Sun, May 2, 2010 at 3:13 PM, Mark Miller <markrmil...@gmail.com> >> >>> wrote: >> >>>> >> >>>> Perfectly safe to use SimpleFSDirectory. When you measure the >> >>>> performance, are you measuring a lot of concurrent requests? That's >> where >> >>>> you should see the win. >> >>>> >> >>>> - Mark >> >>>> >> >>>> On 5/1/10 6:38 PM, John Wang wrote: >> >>>>> >> >>>>> We are seeing this issue as well in your production. (using Zoie on >> top >> >>>>> of lucene 2.9.1) >> >>>>> After some performance comparisons, we do NOT see performance gain >> with >> >>>>> NIO, rather these nasty ClosedChannelExceptions. >> >>>>> >> >>>>> I think the performance gains ppl are seeing with 2.9.1 can be due >> to >> >>>>> many different things. From what we seen, they are not related to >> >>>>> NIOFSDirectory. >> >>>>> >> >>>>> Our solution is to avoid calling FSDirectory.open(), instead just >> call >> >>>>> new SimpleFSDirectory(). Is this safe? >> >>>>> >> >>>>> -John >> >>>>> >> >>>>> On Fri, Jan 29, 2010 at 12:32 PM, Mark Miller < >> markrmil...@gmail.com >> >>>>> <mailto:markrmil...@gmail.com>> wrote: >> >>>>> >> >>>>> Perhaps - one of the things they are supposed to be addressing is >> >>>>> extendability. >> >>>>> >> >>>>> nio2 does have FileSystemProvider, which would actually allow you >> to >> >>>>> create a custom channel ! >> >>>>> >> >>>>> I have not dug in enough to know much more than that though. >> >>>>> >> >>>>> *But*, another really interesting thing is that in Java 7, >> >>>>> FileDescriptors are ref counted ! (though users can't inc/dec). >> >>>>> >> >>>>> But, FileInputStream and OutputStream have a new constructor that >> >>>>> takes >> >>>>> a FileDescriptor. >> >>>>> >> >>>>> So possibly, you could just make one that sits around to keep the >> >>>>> FileDescriptor valid, and get your channel off >> >>>>> FileInputStream/FileOutputStream? >> >>>>> >> >>>>> And then if it goes down, make a new one using the FileDescriptor >> >>>>> which >> >>>>> was not actually closed because there was a still a ref to it. >> >>>>> >> >>>>> Possibly .... ;) >> >>>>> >> >>>>> Michael McCandless wrote: >> >>>>> > Does anyone know if nio2 has improved this...? >> >>>>> > >> >>>>> > Mike >> >>>>> > >> >>>>> > On Fri, Jan 29, 2010 at 2:00 PM, Jason Rutherglen >> >>>>> > <jason.rutherg...@gmail.com <mailto: >> jason.rutherg...@gmail.com>> >> >>>>> wrote: >> >>>>> > >> >>>>> >> Defaulting NIOFSDir could account for some of the recent >> speed >> >>>>> >> improvements users have been reporting in Lucene 2.9. So >> >>>>> removing it >> >>>>> >> as a default could reverse those and people could then report >> >>>>> Lucene >> >>>>> >> 3.X has slowed... >> >>>>> >> >> >>>>> >> On Thu, Jan 28, 2010 at 5:24 AM, Michael McCandless >> >>>>> >> <luc...@mikemccandless.com <mailto:luc...@mikemccandless.com >> >> >> >>>>> wrote: >> >>>>> >> >> >>>>> >>> Bummer. >> >>>>> >>> >> >>>>> >>> So the only viable workarounds are 1) don't use >> >>>>> Thread.interrupt (nor, >> >>>>> >>> things like Future.cancel, which in turn use >> Thread.interrupt) >> >>>>> with >> >>>>> >>> NIOFSDir, or 2) we fix NIOFSDir to reopen the channel AND >> the >> >>>>> app must >> >>>>> >>> make a deletion policy that keeps a commit alive if any >> reader >> >>>>> is >> >>>>> >>> using it. Or, 3) don't use NIOFSDir! >> >>>>> >>> >> >>>>> >>> Mike >> >>>>> >>> >> >>>>> >>> On Thu, Jan 28, 2010 at 7:29 AM, Simon Willnauer >> >>>>> >>> <simon.willna...@googlemail.com >> >>>>> <mailto:simon.willna...@googlemail.com>> wrote: >> >>>>> >>> >> >>>>> >>>> On Thu, Jan 28, 2010 at 12:43 PM, Michael McCandless >> >>>>> >>>> <luc...@mikemccandless.com <mailto: >> luc...@mikemccandless.com>> >> >>>>> wrote: >> >>>>> >>>> >> >>>>> >>>>> On Thu, Jan 28, 2010 at 6:38 AM, Uwe Schindler >> >>>>> <u...@thetaphi.de <mailto:u...@thetaphi.de>> wrote: >> >>>>> >>>>> >> >>>>> >>>>> >> >>>>> >>>>>> So I checked the code of NIOFSIndexInput, my last comment >> >>>>> was not really correct: >> >>>>> >>>>>> NIOFSIndexInput extends SimpleFSIndexInput and that opens >> >>>>> the RAF. In the ctor RAF.getChannel() is called. The RAF keeps >> open >> >>>>> until the file is closed (and also the channel). >> >>>>> >>>>>> >> >>>>> >>>>>> So it's really simple to fix in my opinion, just call >> >>>>> getChannel() again on this exception. Because the RAF should >> still >> >>>>> be open? >> >>>>> >>>>>> >> >>>>> >>>> Short answer: >> >>>>> >>>> public final FileChannel getChannel() { >> >>>>> >>>> synchronized (this) { >> >>>>> >>>> if (channel == null) >> >>>>> >>>> channel = FileChannelImpl.open(fd, true, rw, >> >>>>> this); >> >>>>> >>>> return channel; >> >>>>> >>>> } >> >>>>> >>>> } >> >>>>> >>>> >> >>>>> >>>> this is not gonna work I tried it before. The >> RandomAccessFile >> >>>>> buffers >> >>>>> >>>> the channel!! >> >>>>> >>>> >> >>>>> >>>> simon >> >>>>> >>>> >> >>>>> >>>>> I think we need a definitive answer on what happens to the >> >>>>> RAF when >> >>>>> >>>>> the FileChannel was closed by Thread.Interrupt. Simon can >> >>>>> you test >> >>>>> >>>>> this? >> >>>>> >>>>> >> >>>>> >>>>> Mike >> >>>>> >>>>> >> >>>>> >>>>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> >>>>> To unsubscribe, e-mail: >> >>>>> java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> >>>>> For additional commands, e-mail: >> >>>>> java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> >>>>> >> >>>>> >>>>> >> >>>>> >>>>> >> >>>>> >>>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> >>>> To unsubscribe, e-mail: >> java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> >>>> For additional commands, e-mail: >> >>>>> java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> >>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> >>> To unsubscribe, e-mail: >> java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> >>> For additional commands, e-mail: >> >>>>> java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> >>> >> >>>>> >>> >> >>>>> >>> >> >>>>> >> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> >> To unsubscribe, e-mail: >> java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> >> For additional commands, e-mail: >> java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> > >> >>>>> > >> >>>>> >> --------------------------------------------------------------------- >> >>>>> > To unsubscribe, e-mail: >> java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> > For additional commands, e-mail: >> java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> > >> >>>>> > >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> - Mark >> >>>>> >> >>>>> http://www.lucidimagination.com >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> >>>>> <mailto:java-dev-unsubscr...@lucene.apache.org> >> >>>>> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> >>>>> <mailto:java-dev-h...@lucene.apache.org> >> >>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> - Mark >> >>>> >> >>>> http://www.lucidimagination.com >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>>> >> >>> >> >> >> >> >> >> >> >> -- >> >> - Mark >> >> >> >> http://www.lucidimagination.com >> >> >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >
