__
From: Lebiram
To: java-user@lucene.apache.org
Sent: Wednesday, December 24, 2008 2:43:12 PM
Subject: Re: Optimize and Out Of Memory Errors
Hello Mark,
As of the moment the index could not be rebuilt to remove norms.
Right now, I'm trying to figure out what luke is doing by going throug
To: java-user@lucene.apache.org
Sent: Wednesday, December 24, 2008 2:20:23 PM
Subject: Re: Optimize and Out Of Memory Errors
We don't know those norms are "the" problem. Luke is loading norms if its
searching that index. But what else is Luke doing? What else is your App doing?
I
he.org
Sent: Tuesday, December 23, 2008 5:25:30 PM
Subject: Re: Optimize and Out Of Memory Errors
Mark Miller wrote:
Lebiram wrote:
Also, what are norms
Norms are a byte value per field stored in the index that is factored into the
score. Its used for length normalization (shorter docume
r
To: java-user@lucene.apache.org
Sent: Tuesday, December 23, 2008 5:25:30 PM
Subject: Re: Optimize and Out Of Memory Errors
Mark Miller wrote:
> Lebiram wrote:
>> Also, what are norms
> Norms are a byte value per field stored in the index that is factored into
> the score.
Mark Miller wrote:
Lebiram wrote:
Also, what are norms
Norms are a byte value per field stored in the index that is factored
into the score. Its used for length normalization (shorter documents =
more important) and index time boosting. If you want either of those,
you need norms. When norms
Lebiram wrote:
Also, what are norms
Norms are a byte value per field stored in the index that is factored
into the score. Its used for length normalization (shorter documents =
more important) and index time boosting. If you want either of those,
you need norms. When norms are loaded up into a
2008 17:03:07
Subject: Re: Optimize and Out Of Memory Errors
Hi All,
Thanks for the replies,
I've just managed to reproduced the error on my test machine.
What we did was, generate about 100,000,000 documents with about 7 fields in
it, with terms from 1 to 10.
After the index of about 20GB
"groupId" + i, Field.Store.YES,
Field.Index.UN_TOKENIZED));
writer.addDocument(doc);
____
From: mark harwood
To: java-user@lucene.apache.org
Sent: Tuesday, December 23, 2008 2:42:25 PM
Subject: Re: Optimize and Out Of Memory Errors
I've had reports of OOM e
ifferent environments
with large indexes make me think there is still something to be uncovered
here...
- Original Message
From: Michael McCandless
To: java-user@lucene.apache.org
Cc: Utan Bisaya
Sent: Tuesday, 23 December, 2008 14:08:26
Subject: Re: Optimize and Out Of Memory Errors
How many indexed fields do you have, overall, in the index?
If you have a very large number of fields that are "sparse" (meaning
any given document would only have a small subset of the fields), then
norms could explain what you are seeing.
Norms are not stored sparsely, so when segments g
Recently, our lucene index version was upgraded to 2.3.1 and the index had to
be rebuilt for several weeks which made the entire index a total of 20 GB or
so.
After the the rebuild, a weekly sunday task was executed for optimization.
During that time, the optimization failed several times comp
11 matches
Mail list logo
