Hi,
we are having tremendous problems building a large lucene index and querying
it.
The programmers are telling me that when the index file reaches 3.5 gb or 5
million docs the index file can no longer grow any larger.
To rectify this they have built index files in multiple directories. Now
Yes. We have the same problem. It is mainly because TermInforReader.java
that takes memory space to keep *.tii.
Eugene
-Original Message-
From: Leon Chaddock [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 14, 2006 8:43 AM
To: java-user@lucene.apache.org
Subject: Size + memory
]>
To:
Sent: Tuesday, February 14, 2006 6:38 PM
Subject: RE: Size + memory restrictions
Yes. We have the same problem. It is mainly because TermInforReader.java
that takes memory space to keep *.tii.
Eugene
-Original Message-
From: Leon Chaddock [mailto:[EMAIL PROTECTED]
Sent: Tues
On Dienstag 14 Februar 2006 19:38, Eugene Tuan wrote:
> Yes. We have the same problem. It is mainly because TermInforReader.java
> that takes memory space to keep *.tii.
In Lucene 1.9 you can change that using IndexWriter.setTermIndexInterval().
Regards
Daniel
--
http://www.danielnaber.de
--
You may consider incrementally adding documents to
your index; I'm not sure why there would be problems
adding to an existing index, but you can always add
additional documents. You can optimize later to get
everything back into a single segment.
Querying is a different story; if you are using th
double the
amount of docs we can handle, or would this provide an exponential increase?
Thanks
Leon
- Original Message -
From: "Greg Gershman" <[EMAIL PROTECTED]>
To:
Sent: Wednesday, February 15, 2006 12:41 AM
Subject: Re: Size + memory restrictions
You may consid
;
To:
Sent: Wednesday, February 15, 2006 9:28 AM
Subject: Re: Size + memory restrictions
Hi Greg,
Thanks. We are actually running against 4 segments of 4gb so about 20
million docs. We cant merge the segments as their seems to be problems
with out linux box , with having files over about 4gb.
] srs = new IndexSearcher[SearchersDir.size()];
: int maxI = 2;
:// Searcher[] srs = new IndexSearcher[maxI];
: int i = 0;
: for (Iterator iter = SearchersDir.iterator(); iter.hasNext() && i
: To:
: Sent: Wednesday, February 15, 2006 9:28 AM
: Subject: Re: Size + me
OTECTED]>
To:
Sent: Wednesday, February 15, 2006 6:40 PM
Subject: Re: Size + memory restrictions
: We may have many different segments of our index, and it seems below we
are
: using one
: IndexSearcher per segment. Could this explain why we run out of memory
when
: using more than 2/3 segment
es).
Otis
P.S.
You asked about Lucene in Action... :)
- Original Message
From: Chris Hostetter <[EMAIL PROTECTED]>
To: java-user@lucene.apache.org
Sent: Wednesday, February 15, 2006 1:40:01 PM
Subject: Re: Size + memory restrictions
: We may have many different segments of our ind
10 matches
Mail list logo
