[ http://issues.apache.org/jira/browse/LUCENE-709?page=all ]
Chuck Williams updated LUCENE-709:
----------------------------------
Attachment: ramDirSizeManagement.patch
I've just attached my version of this patch. It includes a multi-threaded test
case. I believe it is sound.
A few notes:
1. Re. Yonik's comment about my synchronization scenario. Synhronizing as
described does resolve the issue. No higher level synchronization is requried.
It doesn't matter how concurent operations on the directory are ordered or
intereleaved, so long as any computation that does a loop sees some instance of
the directory that corresponds to its actual content at any polnt in time. The
result of the loop will then be accurate for that instant.
2. Lucene has this same syncrhonization bug today in RAMDIrectory.list(). It
can return a list of files that never comprised the contents of the directory.
This is fixed in the attached.
3. Also, the long synchronization bug exists in RAMDirectory.fileModified() as
well as RAMDIrectory.fileLength() since both are public. These are fixed in
the attached.
4. I moved the synchronization off of the Hashtable (replacing it with a
HashMap) up to the RAMDirectory as there are some operations that require
synchronization at the directory level. Using just one lock seems better. As
all Hashtable operations were already synchonized, I don't believe any material
additional synchronization is added.
5. Lucene currently make the assumption that if a file is being written by a
stream then no other streams are simultaneously reading or writing it. I've
maintained this assumption as an optimization, allowing the streams to access
fields directly without syncrhonization. This is documented in the comments,
as is the locking order.
5. sizeInBytes is now maintained incrementally, efficiently.
6. Yonik, your version (which I just now saw) has a bug in
RAMDIrectory.renameFile(). The to file may already exist, in which case it is
overwritten and it's size must be subtracted. I actually hit this in my test
case for my implementation and fixed it (since Lucene renames a new version of
the segments file).
All Lucene tests, including the new test, pass. Some contrib tests fail, I
believe none of these failures are in any way related to this patch.
> [PATCH] Enable application-level management of IndexWriter.ramDirectory size
> ----------------------------------------------------------------------------
>
> Key: LUCENE-709
> URL: http://issues.apache.org/jira/browse/LUCENE-709
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.0.1
> Environment: All
> Reporter: Chuck Williams
> Attachments: ramdir.patch, ramdir.patch, ramDirSizeManagement.patch,
> ramDirSizeManagement.patch, ramDirSizeManagement.patch
>
>
> IndexWriter currently only supports bounding of in the in-memory index cache
> using maxBufferedDocs, which limits it to a fixed number of documents. When
> document sizes vary substantially, especially when documents cannot be
> truncated, this leads either to inefficiencies from a too-small value or
> OutOfMemoryErrors from a too large value.
> This simple patch exposes IndexWriter.flushRamSegments(), and provides access
> to size information about IndexWriter.ramDirectory so that an application can
> manage this based on total number of bytes consumed by the in-memory cache,
> thereby allow a larger number of smaller documents or a smaller number of
> larger documents. This can lead to much better performance while elimianting
> the possibility of OutOfMemoryErrors.
> The actual job of managing to a size constraint, or any other constraint, is
> left up the applicatation.
> The addition of synchronized to flushRamSegments() is only for safety of an
> external call. It has no significant effect on internal calls since they all
> come from a sychronized caller.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
