Hi,
I attached code containing some tests for memoryleaks. Unlike Lucene.NET 1.4.3,
the current 2.0 codebase from SVN passes these tests without problems.
cheers,
Erich
________________________________
Von: Erich Eichinger [mailto:[EMAIL PROTECTED]
Gesendet: Mi 2007-03-07 21:53
An: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Betreff: RE: [VOTE] Release of Apache Lucene.Net 2.0.0
Hi all,
I attached the patch for 1.4.3-final-004. It fixes a couple of memory-/resource
leaks and some bugs in GermanStemmer. I have this version in use for over 1
year now in the "wilderness" and never had any problems (at least with Lucene)
since then.
Unfortunately I didn't have time to look into 2.0 more detailed yet.
hope this helps,
Erich
> -----Original Message-----
> From: George Aroush [mailto:[EMAIL PROTECTED]
> Sent: Thursday, March 01, 2007 4:00 PM
> To: [EMAIL PROTECTED];
> [EMAIL PROTECTED]
> Subject: RE: [VOTE] Release of Apache Lucene.Net 2.0.0
>
> Hi Erich,
>
> Now that you mentioned the version you are using, I believe
> those are fixed in 2.0 -- not necessarily in the same way as
> your fix. Can you:
>
> 1) Try Lucene.Net 1.9.1 and see if you are still seeing the
> leaks. I rather have you try 2.0 but some API's have changed in 2.0.
>
> 2) Share with us the patch you used for 1.4.3 -- others may
> find it useful.
>
> Regards,
>
> -- George Aroush
>
> -----Original Message-----
> From: Erich Eichinger [mailto:[EMAIL PROTECTED]
> Sent: Thursday, March 01, 2007 4:12 AM
> To: [EMAIL PROTECTED];
> [EMAIL PROTECTED]
> Subject: RE: [VOTE] Release of Apache Lucene.Net 2.0.0
>
>
> Hi George,
>
> > The memory and resource leaks that you mentioned, have you observed
> > them or do you think they exist?
>
> Yes, I experienced them in a couple of high loaded
> web-applications and the
> fixes I mentioned took me 2 weeks to find them. TBH I
> experienced them using
> Lucene.NET 1.4.3 but checking the codebase showed, that the
> respective code
> didn't change to 2.0.
>
> > Do you have sample code that demonstrates them?
> > I ask this because you said "possible".
>
> This is misleading. I said "possible" because the underlying
> IndexInput
> might be a RAMDirectory. In case of FileDirectory (or the
> DBDirectory I
> mentioned in my private email) there is almost certainly a
> leak because of
> unclosed file-handles or db-connections.
> I can write some sample code to show up the leaks if you
> like. Btw: I found
> and fixed the leaks in v1.4.3 using SciTech's ".NET Memory
> Profiler" (don't
> want to ad here, but it has been the only useful tool I found for this
> purpose)
>
> > As for your WeakHashtable suggestion and the use of
> NHibernate, this
> > isn't possible.
>
> Another misunderstanding ;-). I didn't want you to link Lucene.NET to
> NHibernate. I just wanted to point you to an implementation I
> am aware of.
>
>
> > In my opinion, none of those issues are critical to stop
> this release.
>
> Since I have to deal almost 100% with web-applications, for
> me these leaks
> are definitely a showstopper. Depending on the application's traffic I
> experienced OutOfMemoryExceptions every few hours. After
> applying the fixes
> I described, everything is running smooth.
>
> cheers,
> Erich
>
> > -----Original Message-----
> > From: George Aroush [mailto:[EMAIL PROTECTED]
> > Sent: Thursday, March 01, 2007 2:19 AM
> > To: [EMAIL PROTECTED];
> > [EMAIL PROTECTED]
> > Subject: RE: [VOTE] Release of Apache Lucene.Net 2.0.0
> >
> > Hi Erich,
> >
> > In my opinion, none of those issues are critical to stop
> this release.
> >
> > The memory and resource leaks that you mentioned, have you observed
> > them or do you think they exist? Do you have sample code that
> > demonstrates them? I ask this because you said "possible".
> Yes, I am
> > aware of a leak issue during sorting when Lucene.Net is
> compiled using
> > .NET 1.1, but with .NET 2.0, it disappears.
> >
> > As for your WeakHashtable suggestion and the use of
> NHibernate, this
> > isn't possible. This release is supporting .NET 1.1 so we are
> > somewhat limited with what we have. No, I can't use NHibernate as
> > it's a 3rd party artifact and none ASF.
> >
> > Regards,
> >
> > -- George
> >
> >
> > -----Original Message-----
> > From: Erich Eichinger [mailto:[EMAIL PROTECTED]
> > Sent: Tuesday, February 27, 2007 9:04 AM
> > To: [EMAIL PROTECTED];
> > [EMAIL PROTECTED]
> > Subject: RE: [VOTE] Release of Apache Lucene.Net 2.0.0
> >
> > Hi,
> >
> > there are some resource leaks that lead to really nasty problems in
> > highloaded webapplications. I summed up the most required changes:
> >
> > *)
> > Index/CompoundFileReader.cs:260:
> > Lucene.Net.Index.Compound.FileReader.CSIndexInput doesn't close
> > underlying IndexInput (possible resource leak)
> >
> >
> > *)
> > Index/TermInfosReader.cs
> > Index/SegmentReader.cs
> >
> > Usage of System.Threading.Thread.GetData()/.SetData() may result in
> > memory leaks in web-applications. Using
> > System.Runtime.Remoting.Messaging.CallContext is a much
> better choice.
> >
> >
> > *)
> > Search/FieldCacheImpl.cs
> >
> > cache should be a "WeakHashtable" instead of Hashtable.
> > Otherwise cached readers can't ever be collected and cause
> > memory/resource leaks.
> >
> > *)
> > FieldSortedHitQueue.cs
> >
> > the same problem as in FieldCacheImpl but with "Comparators" table.
> >
> > Asfaik a possible implementation of a "WeakHashtable" can
> be found in
> > the NHibernate project.
> >
> >
> > cheers,
> > Erich
> >
> >
> > > -----Original Message-----
> > > From: George Aroush [mailto:[EMAIL PROTECTED]
> > > Sent: Tuesday, February 27, 2007 4:06 AM
> > > To: [EMAIL PROTECTED];
> > > [EMAIL PROTECTED]
> > > Subject: [VOTE] Release of Apache Lucene.Net 2.0.0
> > >
> > > Hi folks,
> > >
> > > To follow the proper release process of ASF, please take a
> > moment to
> > > cast your vote to release Lucene.Net 2.0.0. I have
> placed both the
> > > release candidate of source code and binary
> > > here: http://people.apache.org/~aroush/
> > >
> > > The change history for this release can be found here:
> > > https://svn.apache.org/repos/asf/incubator/lucene.net/trunk/C%
> > > 23/src/HISTORY
> > > .txt
> > >
> > > In addition to Lucene.Net release, this release also
> > contains ported
> > > code in "contrib".
> > >
> > > Please cast your vote on releasing Apache Lucene.Net
> 2.0.0 no later
> > > then this coming Saturday.
> > >
> > > [ ] +1 Approve release
> > > [x] -1 Veto release (please give reason)
> > >
> > > Regards,
> > >
> > > -- George Aroush
> > >
> > >
> >
> >
>
>
/*
/Search/FieldSortedHitQueue
*/
using System;
using System.Collections;
using System.Reflection;
using System.Threading;
using Lucene.Net.Analysis;
using Lucene.Net.Documents;
using Lucene.Net.Index;
using Lucene.Net.Search;
using Lucene.Net.Store;
using NUnit.Framework;
using FieldInfo=System.Reflection.FieldInfo;
namespace Lucene.Net.Test
{
using Document = Documents.Document;
/// <summary>
/// Summary description for MemoryLeakage.
/// </summary>
[TestFixture]
public class MemoryLeakage
{
private void Reset()
{
FieldSortedHitQueue_ComparatorsCache = new Hashtable();
FieldCache_Cache = new Hashtable();
object[] threadDataTable = GetThreadLocalDataSlots();
for(int i=0;i<threadDataTable.Length;i++)
{
threadDataTable[i] = null;
}
}
[Test]
public void TestComparatorCache()
{
Reset();
DoSomeIndexWork();
GC.Collect();
Assert.AreEqual(0,
FieldSortedHitQueue_ComparatorsCache.Count);
}
[Test]
public void TestFieldCache()
{
Reset();
DoSomeIndexWork();
GC.Collect();
Assert.AreEqual(0, FieldCache_Cache.Count);
}
[Test]
public void TestTermInfosReader()
{
Reset();
DoSomeIndexWork();
GC.Collect();
// Ensure nothing is left on ThreadLocal storage
object[] threadDataTable = GetThreadLocalDataSlots();
for(int i=0;i<threadDataTable.Length;i++)
{
Assert.AreEqual(null, threadDataTable[i]);
}
}
private object[] GetThreadLocalDataSlots()
{
// ensure ThreadLocalStorage is initialized
LocalDataStoreSlot slot = Thread.AllocateDataSlot();
Thread.SetData(slot, null);
object localDataStore =
typeof(Thread).InvokeMember("GetDomainLocalStore"
, BindingFlags.Static | BindingFlags.NonPublic
| BindingFlags.InvokeMethod
, null
, null
, null);
object[] dataTable = (object[])
localDataStore.GetType().GetField("m_DataTable", BindingFlags.Instance |
BindingFlags.NonPublic).GetValue(localDataStore);
return dataTable;
}
private IDictionary FieldCache_Cache
{
get
{
FieldCache fieldCache =
FieldCache_Fields.DEFAULT;
IDictionary cache = (IDictionary)
fieldCache.GetType().GetField("cache", BindingFlags.Instance |
BindingFlags.NonPublic).GetValue(fieldCache);
return cache;
}
set
{
FieldCache fieldCache =
FieldCache_Fields.DEFAULT;
fieldCache.GetType().GetField("cache",
BindingFlags.Instance | BindingFlags.NonPublic).SetValue(fieldCache, value);
}
}
private IDictionary FieldSortedHitQueue_ComparatorsCache
{
get
{
Type tQueue =
typeof(IndexSearcher).Assembly.GetType("Lucene.Net.Search.FieldSortedHitQueue");
FieldInfo fComparators =
tQueue.GetField("Comparators", BindingFlags.Static | BindingFlags.NonPublic);
return (IDictionary)
fComparators.GetValue(null);
}
set
{
Type tQueue =
typeof(IndexSearcher).Assembly.GetType("Lucene.Net.Search.FieldSortedHitQueue");
FieldInfo fComparators =
tQueue.GetField("Comparators", BindingFlags.Static | BindingFlags.NonPublic);
fComparators.SetValue(null, value);
}
}
#region Test Index Directory Generation
private void DoSomeIndexWork()
{
Searcher searcher = this.GetIndex(true, true);
Hits docs = searcher.Search(new TermQuery(new
Term("contents", "b")), new Sort("string", true));
for(int i=0;i<docs.Length();i++)
{
docs.Doc(i).SetBoost(1.0f);
}
searcher.Close();
}
// document data:
// the tracer field is used to determine which document was hit
// the contents field is used to search and sort by relevance
// the int field to sort by int
// the float field to sort by float
// the string field to sort by string
// the i18n field includes accented characters for testing
locale-specific sorting
private System.String[][] data = new System.String[][]{
// tracer contents
int float string custom i18n
new System.String[]{ "A",
"x a", "5", "4f", "c", "A-3", "p\u00EAche"},
new System.String[]{ "B",
"y a", "5", "3.4028235E38", "i", "B-10", "HAT"},
new System.String[]{ "C",
"x a b c", "2147483647", "1.0", "j", "A-2",
"p\u00E9ch\u00E9"},
new System.String[]{ "D",
"y a b c", "-1", "0.0f", "a", "C-0", "HUT"},
new System.String[]{ "E",
"x a b c d", "5", "2f", "h", "B-8", "peach"},
new System.String[]{ "F",
"y a b c d", "2", "3.14159f", "g", "B-1", "H\u00C5T"},
new System.String[]{ "G",
"x a b c d", "3", "-1.0", "f", "C-100", "sin"},
new System.String[]{ "H",
"y a b c d", "0", "1.4E-45", "e", "C-88", "H\u00D8T"},
new System.String[]{ "I",
"x a b c d e f", "-2147483648", "1.0e+0", "d", "A-10", "s\u00EDn"},
new System.String[]{ "J",
"y a b c d e f", "4", ".5", "b", "C-7", "HOT"},
new System.String[]{ "W",
"g", "1", null, null, null, null},
new System.String[]{ "X",
"g", "1", "0.1", null, null, null},
new System.String[]{ "Y",
"g", "1", "0.2", null, null, null},
new System.String[]{ "Z",
"f g", null, null, null, null, null}};
// create an index of all the documents, or just the x, or just
the y documents
private Searcher GetIndex(bool even, bool odd)
{
RAMDirectory indexStore = new RAMDirectory();
IndexWriter writer = new IndexWriter(indexStore, new
SimpleAnalyzer(), true);
for (int i = 0; i < data.Length; ++i)
{
if (((i % 2) == 0 && even) || ((i % 2) == 1 &&
odd))
{
Lucene.Net.Documents.Document doc = new
Lucene.Net.Documents.Document();
doc.Add(new Field("tracer", data[i][0],
Field.Store.YES, Field.Index.NO));
doc.Add(new Field("contents",
data[i][1], Field.Store.NO, Field.Index.TOKENIZED));
if (data[i][2] != null)
doc.Add(new Field("int",
data[i][2], Field.Store.NO, Field.Index.UN_TOKENIZED));
if (data[i][3] != null)
doc.Add(new Field("float",
data[i][3], Field.Store.NO, Field.Index.UN_TOKENIZED));
if (data[i][4] != null)
doc.Add(new Field("string",
data[i][4], Field.Store.NO, Field.Index.UN_TOKENIZED));
if (data[i][5] != null)
doc.Add(new Field("custom",
data[i][5], Field.Store.NO, Field.Index.UN_TOKENIZED));
if (data[i][6] != null)
doc.Add(new Field("i18n",
data[i][6], Field.Store.NO, Field.Index.UN_TOKENIZED));
doc.SetBoost(2); // produce some scores
above 1.0
writer.AddDocument(doc);
}
}
writer.Optimize();
writer.Close();
IndexReader reader = IndexReader.Open(indexStore);
reader.DeleteDocument(reader.NumDocs()/2);
reader.Close();
return new IndexSearcher(indexStore);
}
#endregion Test Index Directory Generation
}
}
