Kevin, Those results are awsome. Could you please give those of us that were following but not quite understanding everything some pseudo code or some more explaination?
Thanks, andrew -----Original Message----- From: Kevin Burton <[EMAIL PROTECTED]> Sent: Jun 7, 2005 7:18 PM To: java-user@lucene.apache.org Subject: Re: Fastest way to fetch N documents with unique keys within large numbers of indexes.. Paul Elschot wrote: >For a large number of indexes, it may be necessary to do this over >multiple indexes by first getting the doc numbers for all indexes, >then sorting these per index, then retrieving them >from all indexes, and repeating the whole thing using terms determined >from the retrieved docs. > > Well this was a BIG win. Just benchmarking it out shows a 10x -> 50x performance increase. Times in milliseconds: Before: duration: 1127 duration: 449 duration: 394 duration: 564 After: duration: 182 duration: 39 duration: 12 duration: 11 The values of 2-4 I'm sure are due to the filesystem buffer cache but I can't imagine why they'd be faster in the second round. It might be that Linux is deciding not to buffer the document blocks. Kevin -- Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. See irc.freenode.net #rojo if you want to chat. Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html Kevin A. Burton, Location - San Francisco, CA AIM/YIM - sfburtonator, Web - http://peerfear.org/ GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
