I routinely have server applications consuming 3.5 - 5 GB of RAM on a
64bit machine primarily consumed by cached QueryFilters (16GB total
physical RAM).
You will not get a RAMDirectory over 2 GB in a 64 bit machine either,
but do to an internal limit.
Unhandled Exception: System.ArgumentOutOfRangeException: Index was out
of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.Collections.ArrayList.get_Item(Int32 index)
at System.Collections.ArrayList.SyncArrayList.get_Item(Int32 index)
at Lucene.Net.Store.RAMInputStream.ReadInternal(Byte[] dest, Int32
destOffset, Int32 len)
at Lucene.Net.Store.BufferedIndexInput.ReadBytes(Byte[] b, Int32
offset, Int32 len)
at
Lucene.Net.Index.CompoundFileReader.CSIndexInput.ReadInternal(Byte[] b,
Int32 offset, Int32 len)
at Lucene.Net.Store.BufferedIndexInput.Refill()
at Lucene.Net.Store.BufferedIndexInput.ReadByte()
at Lucene.Net.Store.IndexInput.ReadInt()
at Lucene.Net.Index.SegmentTermEnum..ctor(IndexInput i, FieldInfos
fis, Boolean isi)
at Lucene.Net.Index.TermInfosReader..ctor(Directory dir, String seg,
FieldInfos fis)
at Lucene.Net.Index.SegmentReader.Initialize(SegmentInfo si)
at Lucene.Net.Index.SegmentReader.Get(Directory dir, SegmentInfo si,
SegmentInfos sis, Boolean closeDir, Boolean ownDir)
at Lucene.Net.Index.IndexReader.AnonymousClassWith.DoBody()
at Lucene.Net.Store.Lock.With.Run()
at Lucene.Net.Index.IndexReader.Open(Directory directory, Boolean
closeDirectory)
at Lucene.Net.Search.IndexSearcher..ctor(Directory directory)
at Index.Performance.QueryExecution.InitializeRAM() in C:\Visual
Studio 2005\Projects\Sandbox\Index.Performance\QueryExecution.cs:line 74
at Index.Performance.Program.Main(String[] args) in C:\Visual Studio
2005\Projects\Sandbox\Index.Performance\Program.cs:line 43
This error is due to overflowing the index of the RAMFile.buffers member
(a synchronized ArrayList) after reading more than 2147483647 bytes.
The limit is the ordinal of the ArrayList collection being an integer.
Not much can be done about that - just a known limit for now. I've seen
mention of the same issue in the Java email lists
[http://www.gossamer-threads.com/lists/lucene/java-dev/24140?do=post_view_threaded].
If you really want to use a RAMDirectory for your index, you will need
to split it up into several shards less than 2 GB each. From the
testing I've done for my application, using a RAMDirectory only made a
significant difference during heavy I/O operations, such as when
retrieving data in stored fields in a HitCollector. For the types of
queries I am doing I did not see a major difference in the speed of the
search itself, but you may see different results.
Michael
Dean Harding wrote:
Thanks a lot for you reply. Currently the memory limit is not the problem
cause i enabled "LARGEADDRESSAWARE"-Option.
Have you got 4GB of PHYSICAL memory in your computer? If not, /3GB +
/LARGEADDRESSAWARE would be quite useless (in terms of a performance
increase anyway) and you'd be much better off just running straight off the
disk.
Besides, the way the garbage collector in .NET works, you actually need a
fair amount of extra virtual memory than what you are actually using. So
even with /largeaddressaware, you're probably not going to get very close to
the 3GB limit because of the overhead.
Of course, if you have a 64-bit machine, things are a little different...
Dean.
