Ning Li wrote:
I was away so I'm catching up.
If this (occasional large documents consume too much memory) happens
to a few applications, should it be solved in IndexWriter?
A possible design could be:
First, in addDocument(), compute the byte size of a ram segment after
the ram segment is created. In the synchronized block, when the newly
created segment is added to ramSegmentInfos, also add its byte size to
the total byte size of ram segments.
Then, in maybeFlushRamSegments(), either one of two conditions can
trigger a flush: number of ram segments reaching maxBufferedDocs, and
total byte size of ram segments exceeding a threshold.
The overhead is very small in this design. Of course, IndexWriter
would have another configurable parameter. :-) But it's nice if an
application could set a limit on the memory it uses to buffer docs.
Ning
This is exactly how I implemented it in my private version a couple of
weeks ago. It works good and I don't see performance problems with this
design. I named the new parameter in IndexWriter: setMaxBufferSize(long).
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
