Chris, Thanks for all your invaluable comments. The killer was the fact that the timestamp for each document was unique. For a search with millions of results, this resulted in allocation of millions of strings during the sorting step (FieldCacheImpl.getStrings). With some loss of precision, I rounded down the timestamp to DateTools.MINUTE resolution in the indexing phase (but I still store it as a string). The cool thing about this is that it limits the maximum number of strings for this field (525000 in a year). So getStrings never allocates more than 0.5 million strings. With this, the sort is suitably fast (amazingly fast in fact, 13,000,000 documents sorted in < 10 seconds on a middling hardware configuration). I don't need a 64-bit JVM either :). Beautiful.
Of course, I can improve this by storing the modified timestamp as an integer...that's the next step. ----- Original Message ---- From: Chris Hostetter <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Friday, January 26, 2007 11:15:05 AM Subject: Re: Extending scoring to eliminate sorting on timestamp : I used String because the timestamp is a Long and there wasn't any : SortField.LONG (I guess I should have used SortField.CUSTOM). In this : case, what should the indexing call look like? Currently, I have: : doc.add(new Field("timestamp",Long.toString(timestamp),Field.Store.NO,Field.Index.UN_TOKENIZED)); for the record, what you've got there won't sort correctly as a string anyway, the number 923456 will sort after the number 12345678 because as a string it's lexigraphicaly larger ... you'd need to 0 pad the value, or use something like NumberTools. i never really noticed there was an int field cache, but no long field cache ... either way, if you are tryingto avoid hte FieldCachebecause of hte time ittakes to initialize and not hte memory it takes up, then long support in FieldCachewouldnt' help you. : The other thing I was considering is to automatically limit the number : of results (there is no way a user can grok 3 million results anyway) by : breaking down the range filter into a series of range filters and : executing multiple searches in series until the max number of results i'm not sure how that would help you unlessy ou did all of the sorting external to Lucene (or used something like the patch i mentioned) : case (when the number of results is reasonable). One way around this is : to execute an initial search just to figure out the number of hits : (without sorting, without scoring) and then apply different strategies, if you try this use Sort.INDEXORDER .. short of writing your own HitCollector i believe it's the best way to eliminate any extra work Lucene might do for you to sort results. : The patch you pointed out looks very very promising. There's been some disagreement as to how usefull it is -- i don't know if anyone has had time to do a thorough performnce analysys of it in differnet sitautions. if you try it out please comment in the issue with your experiences 9good or bad) ----- Original Message ---- From: Chris Hostetter <[EMAIL PROTECTED]> To: java-user@lucene.apache.org Sent: Thursday, January 25, 2007 9:09:12 PM Subject: Re: Extending scoring to eliminate sorting on timestamp : For various reasons, we'd like to eliminate the sort step. can you elaborate on what those reasons are? FunctionQuery (in the solr code base, you'll find lots of discussing in the archives of this list) can let you use a numeric field value in the score calculation, but it still uses the FieldCache so if you are trying to avoid that for space/time reasons it won't help. you may also be interested in this patch... https://issues.apache.org/jira/browse/LUCENE-769 in the general case it should be slower then standard sorting, but if you are dealing with an extremely large index and your result sets all tend to be small, it may be faster (and it won't pay the initial FieldCache setup time on frequently modified indexes) : new SortField("timestamp",SortField.STRING,true)})); why are you sorting timestamps as strings? ... if you sort them as ints, your FieldCache will be a whole hell of a lot smaller (i'm guessing very few documents have identicle timestamps, so your FieldCache should be at least half as big if you sort on ints (and probably a lot more). -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] : : --------------------------------------------------------------------- : To unsubscribe, e-mail: [EMAIL PROTECTED] : For additional commands, e-mail: [EMAIL PROTECTED] : -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]