Hi Otis,
Thanks very much for your reply.
I've removed the LuceneIndexAccessor code, and still have the same
problem, so that at least rules out LuceneIndexAccessor as the source.
maxBufferedDocs is just set to the default, which I believe is 10.
I've tried jconsole, + jmap/jhat for looking at the contents of the
heap. One interesting thing is that although the memory allocated as
reported by the processes tab of Windows Task Manager goes up and up,
and the JVM eventually crashes with an OutOfMemory error, the total size
of heap + non-heap as reported by jconsole is constant and much lower
than the Windows-reported allocated memory. I've also tried Netbeans
profiler, which suggests that the variables in the heap that are
continually surviving garbage collection do not all originate from one
class.
I can't definitely rule out Tomcat. Clearly something is interacting
with a change in JVM 1.6 and causing the problem. The fact that it only
occurs during indexing not searching suggested that it might be related
to the indexing code rather than Tomcat. It's much more likely that it's
my code than Lucene, but I can't see anything in my code though I'm
definitely no expert on memory leaks. All the variables created during
indexing except IndexReader and Searcher instances are local to my
addDocument function so should be garbage collected after each document
is added. I did wonder if it might be related to SnowballAnalyzer as
quite a few long lived variables in the heap were created by this - but
then the heap is not increasing.
Regards,
Steve
Otis Gospodnetic wrote:
Hi Steve,
You said the OOM happens only when you are indexing. You don't need
LuceneIndexAccess for that, so get rid of that to avoid one suspect that is not
part of Lucene core. What is your maxBufferedDocs set to? And since you are using
JVM 1.6, check out jmap, jconsole & friends, they'll provide insight into where
your OOM is coming from. I see your app is a webapp. How do you know it's Lucene
and its indexing that are the source of OOM and not something else, such as a bug
in Tomcat?
Otis
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/ - Tag - Search - Share
----- Original Message ----
From: Stephen Gray <[EMAIL PROTECTED]>
To: java-user@lucene.apache.org
Sent: Tuesday, May 15, 2007 2:31:05 AM
Subject: Memory leak (JVM 1.6 only)
Hi everyone,
I have an application that indexes/searches xml documents using Lucene.
I'm having a problem with what looks like a memory leak, which occurs
when indexing a large number of documents, but only when the application
is running under JVM 1.6. Under JVM 1.5 there is no problem. What
happens is that the memory allocated consistently rises during indexing
until the JVM crashes with an OutOfMemory exception.
I'm using Lucene 2.1, and am using Maik Schreiber's LuceneIndexAccess
API, which hands out references to cached IndexWriter/Reader/Searchers
to objects that need to use them, and handles closing and re-opening
IndexSearchers after documents are added to the index. The application
is running under Tomcat 6.
I'm a bit out of my depth determining the source of the leak - I've
tried using Netbeans profiler, which shows a large number of HashMap
instances that survive a long time, but these are created by many
different classes so it's difficult to pinpoint one source.
Has anyone found similar problems with Lucene indexing operations
running under JVM 1.6? Does anyone have any suggestions re how to deal
with this?
Any help much appreciated.
Thanks,
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Stephen Gray
Archive IT Officer
Australian Social Science Data Archive
18 Balmain Crescent (Building #66)
The Australian National University
Canberra ACT 0200
Phone +61 2 6125 2185
Fax +61 2 6125 0627
Web http://assda.anu.edu.au/
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
