I switched the Comparators and FieldCache classes to use java.util.HashMap instead of java.util.WeakHashMap, and got the performance boost I was looking for (test index of 100K documents; initial search took 991 ms, all subsequent searchs took < 90ms. Before, I was seeing initial query of ~1sec, subsequent queries between 500 and 700 ms, with comparator and field lookup table computed each time).
I guess the question is why use a WeakHashMap here as opposed to a HashMap? Greg --- Greg Gershman <[EMAIL PROTECTED]> wrote: > I've done a bit more snooping around; it seems that > in > FieldSortedHitQueue.getCachedComparator(line 153), > calls to lookup a stored comparator in the cache > always return null. This occurs even for the > built-in > sort types (I tested it on integers and my code for > longs). The comparators don't even appear to be > being > stored in the HashMap to begin with. > > Any ideas? > > Greg > > > > --- Aviran <[EMAIL PROTECTED]> wrote: > > Since I had to implement sorting in lucene 1.2 I > had > > to write my own sorting > > using something similar to a lucene's contribution > > called SortField. > > Yesterday I did some tests, trying to use lucene > 1.4 > > Sort objects and I > > realized that my old implementation works 40% > faster > > then Lucene's > > implementation. My guess is that you are right and > > there is a problem with > > the cache although I couldn't find what that is > yet. > > > > Aviran > > > > -----Original Message----- > > From: Greg Gershman [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, July 21, 2004 9:22 AM > > To: [EMAIL PROTECTED] > > Subject: Sort: 1.4-rc3 vs. 1.4-final > > > > > > When rc3 came out, I modified the classes used for > > Sorting to, in addition to Integer, Float and > > String-based sort keys, use Long values. All I > did > > was add extra statements in 2 classes (SortField > and > > FieldSortedHitQueue) that made a special case for > > longs, and created a LongSortedHitQueue identical > to > > the IntegerSortedHitQueue, only using longs. > > > > This worked as expected; Long values converted to > > strings and stored in Field.Keyword type fields > > would > > be sorted according to Long order. The initial > > query > > would take a while, to build the sorted array, but > > subsequent queries would take little to no time at > > all. > > > > I went back to look at 1.4 final, and noticed the > > Sort implementation has > > changed quite a bit. I tried the same type of > > modifications to the existing > > source files, but was unable to achieve similiar > > results. > > Each subsequent query seems to take a significant > > amount of time, as if the Sorted array is being > > rebuilt each time. Also, I tried sorting on an > > Integer fields and got similar results, which > leads > > me > > to believe there might be a caching problem > > somewhere. > > > > Has anyone else seen this in 1.4-final? Also, I > > would > > like it if Long sorted fields could become a part > of > > the API; it makes sorting by date a breeze. > > > > Thanks! > > > > Greg Gershman > > > > > > > > __________________________________ > > Do you Yahoo!? > > New and Improved Yahoo! Mail - Send 10MB messages! > > http://promotions.yahoo.com/new_mail > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > > > > __________________________________ > Do you Yahoo!? > Vote for the stars of Yahoo!'s next ad campaign! > http://advision.webevents.yahoo.com/yahoo/votelifeengine/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > __________________________________ Do you Yahoo!? Vote for the stars of Yahoo!'s next ad campaign! http://advision.webevents.yahoo.com/yahoo/votelifeengine/ --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
