[
https://issues.apache.org/jira/browse/LUCENE-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846835#action_12846835
]
Earwin Burrfoot commented on LUCENE-2328:
-----------------------------------------
A shot in the sky (didn't delve deep into the problem, could definetly miss
stuff):
What about tracking 'syncidness' from within Directory?
There shouldn't be more than one writer anyway (unless your locking is broken),
so that's a single set of 'files-to-be-synced' for each given moment of time.
Might as well keep track of it inside the directory, and have a
syncAllUnsyncedGuys() on it.
This will also remove the need to transfer that list around when transferring
write lock (IR hell).
And all-round that sounds quite logical, as the need/method of syncing depends
solely on directory. If you're working with RAMDirectory, you don't need to
keep track of these files at all.
Probably same for some of DB impls.
Also some filesystems sync everything, when you ask to sync a single file, so
if you're syncing a batch of them in a row, that's some overhead that you can
theoretically work around with a special flag to FSDir.
> IndexWriter.synced field accumulates data leading to a Memory Leak
> -------------------------------------------------------------------
>
> Key: LUCENE-2328
> URL: https://issues.apache.org/jira/browse/LUCENE-2328
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9.1, 2.9.2, 3.0, 3.0.1
> Environment: all
> Reporter: Gregor Kaczor
> Priority: Minor
> Fix For: 3.1
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I am running into a strange OutOfMemoryError. My small test application does
> index and delete some few files. This is repeated for 60k times. Optimization
> is run from every 2k times a file is indexed. Index size is 50KB. I did
> analyze
> the HeapDumpFile and realized that IndexWriter.synced field occupied more than
> half of the heap. That field is a private HashSet without a getter. Its task
> is
> to hold files which have been synced already.
> There are two calls to addAll and one call to add on synced but no remove or
> clear throughout the lifecycle of the IndexWriter instance.
> According to the Eclipse Memory Analyzer synced contains 32618 entries which
> look like file names "_e065_1.del" or "_e067.cfs"
> The index directory contains 10 files only.
> I guess synced is holding obsolete data
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
