[ https://issues.apache.org/jira/browse/LUCENE-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846872#action_12846872 ]
Michael McCandless commented on LUCENE-2328: -------------------------------------------- I like this idea! But, we don't want to simply sync all new files. When IW commits, it's possibly a subset of all new files. EG running merges (or any still-open files) should not be sync'd. Not necessarily all closed files should be sync'd either -- eg any files that were opened & closed while we were syncing (since syncing can take some time) should not then be sync'd. Maybe we change Dir.sync to take a Collection<String>? Then dir would be the one place that keeps track of what's already been sync'd and what hasn't. Or... I wonder if calling sync on a file that's already been sync'd is really that wasteful... I mean it's technically a no-op, so it's just the overhead of a no-op system call from way up in javaland. > IndexWriter.synced field accumulates data leading to a Memory Leak > ------------------------------------------------------------------- > > Key: LUCENE-2328 > URL: https://issues.apache.org/jira/browse/LUCENE-2328 > Project: Lucene - Java > Issue Type: Bug > Components: Index > Affects Versions: 2.9.1, 2.9.2, 3.0, 3.0.1 > Environment: all > Reporter: Gregor Kaczor > Priority: Minor > Fix For: 3.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > I am running into a strange OutOfMemoryError. My small test application does > index and delete some few files. This is repeated for 60k times. Optimization > is run from every 2k times a file is indexed. Index size is 50KB. I did > analyze > the HeapDumpFile and realized that IndexWriter.synced field occupied more than > half of the heap. That field is a private HashSet without a getter. Its task > is > to hold files which have been synced already. > There are two calls to addAll and one call to add on synced but no remove or > clear throughout the lifecycle of the IndexWriter instance. > According to the Eclipse Memory Analyzer synced contains 32618 entries which > look like file names "_e065_1.del" or "_e067.cfs" > The index directory contains 10 files only. > I guess synced is holding obsolete data -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org