I think you're jumping into the conversation too late. What you have said here does not address the problem at hand. That is, in TermInfosReader, all terms in the segment get loaded into three very large arrays. If your index is massive and has many fields indexed (dates for example), you need huge amounts of memory to accommodate for it. The only way around this is to change lucene in such a way that it does not store all these terms in memory. It is possible that an implementation can be made that will be fast for regular fields, but go to disk for fields that are "special" (i.e. dates) or some such thing.
Tony Schwartz [EMAIL PROTECTED] "What we need is more cowbell." > You can still have the complete date as a separate field, and sort or > filter by it, just don't use this field in your query. > > Aviran > http://www.aviransplace.com > > -----Original Message----- > From: Tony Schwartz [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 18, 2005 8:32 AM > To: java-user@lucene.apache.org > Subject: Re: OutOfMemoryError on addIndexes() > > Is this a viable solution? > Doesn't this make sorting and filtering much more complex and much more > expensive as well? > > Tony Schwartz > [EMAIL PROTECTED] > > >> On Wednesday 17 August 2005 22:49, Paul Elschot wrote: >>> > the index could potentially be huge. >>> > >>> > So if this is indeed the case, it is a potential scalability >>> > bottleneck in lucene index size. >>> >>> Splitting the date field into century, year in century, month, day, >>> hour, seconds, and milliseconds will reduce the total number of > indexed terms to 2300 or so. >> >> 1300 is closer, I forgot that I had split off the centuries. >> >> Regards, >> Paul Elschot >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
