I agree. this falls into the area where technical limit is reached. Time to modify the spec.
I thought about this issue over this couple of days, there is really NO silver bullet. If the field is multi-value field and the distinct field values are not too many, you might reduce memory usage by storing the field as bitset. Each bit corresponding to a distinct value. But either way, you have to load the whole thing into memory for good performance. Jian On 4/10/07, Chris Hostetter <[EMAIL PROTECTED]> wrote:
: I'm wondering then if the Sorting infrastructure could be refactored : to allow with some sort of policy/strategy where one can choose a : point where one is not willing to use memory for sorting, but willing ... : To accomplish this would require a substantial change to the : FieldSortHitQueue et al, and I realize that the use of NIO I don't follow ... why could this be implemented entirely via a new SortComparatorSource? (you would also need something to create your file, but that could probably be done as a decorator or subclass of IndexWRiter couldn't it?) : immediately pins Lucene to Java 1.4, so I'm sure this is : controversial. But, if we wish Lucene to go beyond where it is now, Java 1.5 is controversial, Lucene already has 1.4 dependencies. : I think we need to start thinking about this particular problem : sooner rather than later. it depends on your timeline, Lucene's gotten pretty far with what it's got. Personally i'm banking on RAM getting cheaper fast enough that I won't ever need to worry about this. If i needed to support sorting on lots of fields with lots of differnet locales, and my index was big enough that i couldn't feasibly keep all of the FieldCaches in memory on one box, i wouldn't partition the index across multiple boxes and merge results with a MultiSearcher ... i'd clone the index across multiple boxes and partition the traffic based on the field/locale it's searching on. it's a question of cache management, if i know i have two very differnet use cases for a Solr index, i partition those use case to seperate tiers of machines to get better cache utilization, FieldCache is just another type of cache. -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]