Hi Narendra and all,
Here is what I have so far.
The leak is coming from two classes: Lucene.Net.Store.RAMInputStream and
Lucene.Net.Store.RAMOutputStream. Both of those classes are leaking because
the instances of Lucene.Net.Store.RAMFile (which both create) are not being
released.
In the class Lucene.Net.Store.RAMInputStream, if I modify the function
Close() from:
public override void Close()
{
}
To:
public override void Close()
{
file.buffers.Clear(); // I added this line
}
It will cut down the memory leak by up to 40%.
A similar code change in RAMOutputStream will result with a crash.
It's clear to me that instances of RAMInputStream and RAMOutputStream are
not being released (because of RAMFile) thus, the leak. I don't believe the
line that I added is the fix.
So, I am still digging but any help anyone can offer would be great.
Regards,
-- George Aroush
-----Original Message-----
From: Narendra Tumkur [mailto:[EMAIL PROTECTED]
Sent: Sunday, September 24, 2006 7:05 PM
To: [email protected]
Subject: RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Hi George,
Any news on this memory leak ?
Narendra Tumkur
Web Developer
drive.com.au
-----Original Message-----
From: George Aroush [mailto:[EMAIL PROTECTED]
Sent: Wednesday, 13 September 2006 8:01 AM
To: [email protected];
[email protected]
Subject: RE: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
Hi Folks,
Some more update on the memory leak.
If I compile Lucene.Net using VC8 (i.e.: .NET 2.0) the leak won't occur. In
addition, with .NET 2.0, the destructor/finalize of TermInfosReader is
getting called, but not when compiled with .NET 1.1.
I checked the Java version, and there too, TermInfosReader.finalize() is
getting called.
No, the answer is not to simply switch over to .NET 2.0, I have a need to
run Lucene.Net on .NET 1.1 and I am sure many others do too.
Regards,
-- George
-----Original Message-----
From: George Aroush [mailto:[EMAIL PROTECTED]
Sent: Monday, September 11, 2006 9:34 PM
To: [email protected]
Subject: Memory leak (was: RE: Lucene.Net Indexing Large Databases)
No, adding directory.Close() will not do the trick. However, I do believe
we are missing a Close() somewhere, most likely to SegmentReader or
TermInfosReader.
All indication are that one or both of those classes is the source of the
leak. Those were my suspects and the MS CLR Profiler is pointing in this
direction.
The Java version lands you into the Finalize of those two classes, but with
the C# version, the Finalize of those two classes is never getting called!
I hope to find out more in the next few days, but if anyone got the cycles
and got a nice tool to help with this defect, please give this a try and let
me know what you find.
Regards,
-- George
-----Original Message-----
From: Eyal Post [mailto:[EMAIL PROTECTED]
Sent: Monday, September 11, 2006 6:50 PM
To: [email protected]
Subject: RE: Lucene.Net Indexing Large Databases
Adding:
directory.Close();
At the end of the method should help. Do you see still see leaks after that?
Eyal
> -----Original Message-----
> From: George Aroush [mailto:[EMAIL PROTECTED]
> Sent: Monday, September 11, 2006 3:50 AM
> To: [email protected];
> [email protected]
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Folks,
>
> Since last weekend, I have been trying to narrow down the problem to
> this memory leak without much of a luck.
>
> Does anyone have a tool (or could recommend one, without costing me
> $$) which hopefully show the source of the leak?
>
> Unlike C++ code, the leak here, obviously, is due to not releasing
> references to temporary or real objects. The trick is finding the
> object.
>
> This leak can be created with this simple code:
>
> public static void Main(System.String[] args)
> {
> IndexWriter diskIndex;
> Directory directory;
> Analyzer analyzer;
> Document doc;
> int count;
> string indexDirectory;
> System.IO.FileInfo fi;
>
> indexDirectory = "C:\\Index.Bad";
>
> fi = new System.IO.FileInfo(indexDirectory);
> directory =
> Lucene.Net.Store.FSDirectory.GetDirectory(fi,
> true);
>
> analyzer = new SimpleAnalyzer();
> diskIndex = new IndexWriter(directory, analyzer, true);
>
> count = 0;
> while (count < 10000)
> {
> doc = new Document();
> diskIndex.AddDocument(doc);
> count++;
> }
>
> diskIndex.Close();
> }
>
> This code will show a leak in 1.9, 1.9.1 and 2.0 but not 1.4.3. I
> also verified and it doesn't leak under the Java version of Lucene
> (2.0 is where I tested.)
>
> Regards,
>
> -- George
>
>
> -----Original Message-----
> From: George Aroush [mailto:[EMAIL PROTECTED]
> Sent: Friday, September 01, 2006 9:21 PM
> To: [email protected]
> Subject: RE: Lucene.Net Indexing Large Databases
>
> Hi Chris,
>
> I am using 1.9.1 in production and I am not having this problem.
> Sorry, I don't have enough cycles to try your code on 1.9.
>
> This problem was reported on 1.4.x and was fixed. I am sure I carried
> it over to 1.9.x and 2.0 -- or maybe this is a new issue. I will
> double check when I get the cycles.
>
> You can get 1.4.3's source code as ZIP from the download site of
> Lucene.Net which is here:
> https://svn.apache.org/repos/asf/incubator/lucene.net/site/dow
nload/ or you can SVN the source code from here:
https://svn.apache.org/repos/asf/incubator/lucene.net/tags/
Regards,
-- George Aroush
-----Original Message-----
From: Chris David [mailto:[EMAIL PROTECTED]
Sent: Friday, September 01, 2006 1:46 PM
To: [email protected]
Subject: RE: Lucene.Net Indexing Large Databases
Thanks René, so its not just me with this problem. Now where can I get a
hold of this wonderful 1.4 Build of Lucene. Its not listed directly on
Apache's Lucene.NET Page. I am anxious to see if my code actually does work.
Thanks again for all your help, I really do appreciate it.
Chris
Snapstream Media
-----Original Message-----
From: René de Vries [mailto:[EMAIL PROTECTED]
Sent: Friday, September 01, 2006 7:32 AM
To: [email protected]
Subject: RE: Lucene.Net Indexing Large Databases
Update: I didn't realize my earlier code example ran against 1.4.
If I run this with 1.9final-005 build, I am experiencing the exact same
problems as Chris mentions. Memory consumtion keeps growing, I had to kill
it at 1.5Gb. Exact same code, but with a 1.4 version of the lucene.net DLL,
and it runs along at 50Mb
René
