Indeed - I just ran the FileReaderTest on a Linux tmpfs ramdisk - with SeparateFile all 4 of my cores are immediately pinned and remain so. With ChannelFile, all 4 cores hover 20-30%.
It would appear it may not be a good idea to use NIOFSDirectory on ramdisks. Even still though - it looks like you have a further issue - your Lucene 2.9 old-api results don't use it, and are still not good. -- - Mark http://www.lucidimagination.com Uwe Schindler wrote: > Maybe Linux has some problems with NIO on tmpfs/other ramdisks. What Linux > do you use, 64bit or 32bit JVM and kernel, ram fs type? > > If you have 64 bit and you stored your index in Linux tmpfs (not the old RAM > fs), the fastest would be MMapDirectory, as the tmpfs RAM can be directly > used when mapped into the JVM address space. > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: Mark Miller [mailto:markrmil...@gmail.com] >> Sent: Tuesday, September 15, 2009 5:30 PM >> To: java-user@lucene.apache.org >> Subject: Re: lucene 2.9.0RC4 slower than 2.4.1? >> >> Thomas Becker wrote: >> >>> Hey Mark, >>> >>> yes. I'm running the app on unix. You see the difference between 2.9 and >>> >> 2.4 here: >> >>> http://ankeschwarzer.de/tmp/graph.jpg >>> >>> >> Right - I know your measurements showed a difference (and will keep that >> in mind) - but the profiling results then seem >> oddly similar. >> >> >>> 2.4 responds much quicker thus increasing throughput severly. I'm having >>> >> a >> >>> single segment only: >>> >>> -rw-r--r-- 1 asuser asgroup 20 Sep 9 16:40 segments.gen >>> -rw-r--r-- 1 asuser asgroup 294 Sep 9 16:44 segments_1o >>> -rw-r--r-- 1 asuser asgroup 2810603184 Sep 9 16:44 _7c.cfs >>> >>> The FileChannel.read hotspot is indeed strange. Especially if taking >>> >> into >> >>> account that the index is lying on a tmpfs (in memory). And 2.4 should >>> >> be using >> >>> NIOFSDirectory as well?! Will check that. >>> >>> >> 2.4 did not use NIOFSDirectory by default - you would have had to >> manually specified it. Now its used by default if its detects a non >> Windows OS. >> >> Any particular reason your profiling output does not show invocations? >> Its not essential most likely, but I've found it to be helpful in >> comparisons. >> >> We are about to release 2.9, and its been such a long haul, I'd hate to >> see a release with an unknown performance trap. >> >> -- >> - Mark >> >> http://www.lucidimagination.com >> >> >> >>> Thanks a lot for your support! >>> >>> Cheers, >>> Thomas >>> >>> Mark Miller wrote: >>> >>> >>>> A few quick notes - >>>> >>>> Lucene 2.9 old api doesn't appear much worse than Lucene 2.4? >>>> >>>> You save a lot with the new Intern impl, because thats not a hotspot >>>> anymore. But then, >>>> RandomAccessFile seeks end up being a lot more of the pie. They look >>>> fairly similar in speed overall? >>>> >>>> It looks like the major bottleneck with 2.9 new api is that its using >>>> NIOFSDirectory (your on unix I guess, and it now >>>> defaults to that on non Windows os's), and that appears to be a real >>>> killer for you. Its taking half the time for its >>>> reads. ??? >>>> >>>> No conclusions yet, but I'm looking it over. Some other guys will come >>>> in with some ideas as well. >>>> >>>> Do confirm that those profiling results are on a single segment though. >>>> >>>> - Mark >>>> >>>> >>>> Mark Miller wrote: >>>> >>>> >>>>> Thomas Becker wrote: >>>>> >>>>> >>>>> >>>>>> Here's the results of profiling 10 different search requests: >>>>>> >>>>>> http://ankeschwarzer.de/tmp/lucene_24_oldapi.png >>>>>> http://ankeschwarzer.de/tmp/lucene_29_oldapi.png >>>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi.png >>>>>> >>>>>> But you already gave me a good hint. The index being used is an old >>>>>> >> one build >> >>>>>> with lucene 2.4. I will now try a freshly build 2.9 index and see if >>>>>> >> performance >> >>>>>> improves. Maybe that already solves the issue...stupid me... >>>>>> >>>>>> >>>>>> >>>>>> >>>>> That shouldn't be an issue unless there is some odd bug. >>>>> >>>>> >>>>> >>>>> >>>>>> We're updating the index every 30 min. at the moment and it gets >>>>>> >> optimized after >> >>>>>> each update. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> So this profiling is on an optimized index (eg a single segment) ? >>>>> That would be odd indeed, and possibly point to some of the scoring >>>>> >> changes. >> >>>>> >>>>> >>>>>> Mark Miller wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Thomas Becker wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hey Mark, >>>>>>>> >>>>>>>> thanks for your reply. Will do. Results will follow in a couple of >>>>>>>> >> minutes. >> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Thanks, awesome. >>>>>>> >>>>>>> Also, how many segments (approx) are in your index? If there are a >>>>>>> >> lot, >> >>>>>>> have you/can you try the same tests on an optimized index? Don't >>>>>>> >> want to >> >>>>>>> get ahead of the profiling results, but just to continue the info >>>>>>> >> loop. >> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
