Doug Cutting wrote:
Kevin A. Burton wrote:
Is there any way to reduce this footprint? The index is fully
optimized... I'm willing to take a performance hit if necessary. Is
this documented anywhere?
You can increase TermInfosWriter.indexInterval. You'll need to
re-write the .tii file for
Kevin A. Burton wrote:
1. Do I have to do this with a NEW directory? Our nightly index merger
uses an existing target index which I assume will re-use the same
settings as before? I did this last night and it still seems to use the
same amount of memory. Above you assert that I should use a
Kevin A. Burton wrote:
Is there any way to reduce this footprint? The index is fully
optimized... I'm willing to take a performance hit if necessary. Is
this documented anywhere?
You can increase TermInfosWriter.indexInterval. You'll need to re-write
the .tii file for this to take effect.
Sounds interesting. (Is there a btree seralization impl in java?)
.V
jian chen wrote:
Hi,
If it is really the case that every 128th term is loaded into memory.
Could you use a relational database or b-tree to index to do the work
of indexing of the terms instead?
Even if you create another level
On Jan 24, 2005, at 00:10, Vic wrote:
(Is there a btree seralization impl in java?)
http://jdbm.sourceforge.net/
Cheers
--
PA
http://alt.textdrive.com/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
On Saturday 22 January 2005 01:39, Kevin A. Burton wrote:
Kevin A. Burton wrote:
We have one large index right now... its about 60G ... When I open it
the Java VM used 940M of memory. The VM does nothing else besides
open this index.
After thinking about it I guess 1.5% of memory
It would be interesting to know _what_exactly_ uses your memory.
Running under an optimizer should tell you that.
The only thing that comes to mind is... can't remember the details now,
but when the index is opened, I believe every 128th term is read into
memory. This, I believe, helps with
There Kevin, that's what I was referring to, the .tii file.
Otis
--- Paul Elschot [EMAIL PROTECTED] wrote:
On Saturday 22 January 2005 01:39, Kevin A. Burton wrote:
Kevin A. Burton wrote:
We have one large index right now... its about 60G ... When I
open it
the Java VM used 940M
Hi,
If it is really the case that every 128th term is loaded into memory.
Could you use a relational database or b-tree to index to do the work
of indexing of the terms instead?
Even if you create another level of indexing on top of the .tii fle,
it is just a hack and would not scale well.
I
Paul Elschot wrote:
This would be similar to the way the MySQL index cache works...
It would be possible to add another level of indexing to the terms.
No one has done this yet, so I guess it's prefered to buy RAM instead...
The problem I think for everyone right now is that 32bits just
Chris Hostetter wrote:
: We have one large index right now... its about 60G ... When I open it
: the Java VM used 940M of memory. The VM does nothing else besides open
Just out of curiosity, have you tried turning on the verbose gc log, and
putting in some thread sleeps after you open the reader,
Otis Gospodnetic wrote:
It would be interesting to know _what_exactly_ uses your memory.
Running under an optimizer should tell you that.
The only thing that comes to mind is... can't remember the details now,
but when the index is opened, I believe every 128th term is read into
memory. This, I
On Jan 22, 2005, at 23:50, Kevin A. Burton wrote:
The problem I think for everyone right now is that 32bits just doesn't
cut it in production systems... 2G of memory per process and you
really start to feel it.
Hmmm... no... no pain at all... or perhaps you are implying that your
entire
Yes, I remember your email about the large number of Terms. If it can
be avoided and you figure out how to do it, I'd love to patch
something. :)
Otis
--- Kevin A. Burton [EMAIL PROTECTED] wrote:
Otis Gospodnetic wrote:
It would be interesting to know _what_exactly_ uses your memory.
We have one large index right now... its about 60G ... When I open it
the Java VM used 940M of memory. The VM does nothing else besides open
this index.
Here's the code:
System.out.println( opening... );
long before = System.currentTimeMillis();
Directory dir =
Kevin A. Burton wrote:
We have one large index right now... its about 60G ... When I open it
the Java VM used 940M of memory. The VM does nothing else besides
open this index.
After thinking about it I guess 1.5% of memory per index really isn't
THAT bad. What would be nice if there was a way
: We have one large index right now... its about 60G ... When I open it
: the Java VM used 940M of memory. The VM does nothing else besides open
Just out of curiosity, have you tried turning on the verbose gc log, and
putting in some thread sleeps after you open the reader, to see if the
memory
17 matches
Mail list logo
