NRT Caching Dir to allow for exact memory usage, better buffer allocation and
"global" cross indices control
------------------------------------------------------------------------------------------------------------
Key: LUCENE-3425
URL: https://issues.apache.org/jira/browse/LUCENE-3425
Project: Lucene - Java
Issue Type: Improvement
Components: core/index
Reporter: Shay Banon
A discussion on IRC raised several improvements that can be made to NRT caching
dir. Some of the problems it currently has are:
1. Not explicitly controlling the memory usage, which can result in overusing
memory (for example, large new segments being committed because refreshing is
too far behind).
2. Heap fragmentation because of constant allocation of (probably promoted to
old gen) byte buffers.
3. Not being able to control the memory usage across indices for multi index
usage within a single JVM.
A suggested solution (which still needs to be ironed out) is to have a
BufferAllocator that controls allocation of byte[], and allow to return unused
byte[] to it. It will have a cap on the size of memory it allows to be
allocated.
The NRT caching dir will use the allocator, which can either be provided (for
usage across several indices) or created internally. The caching dir will also
create a wrapped IndexOutput, that will flush to the main dir if the allocator
can no longer provide byte[] (exhausted).
When a file is "flushed" from the cache to the main directory, it will return
all the currently allocated byte[] to the BufferAllocator to be reused by other
"files".
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
