Re: Solr memory usage off-heap

2014-03-20 Thread Avishai Ish-Shalom
thanks! On Tue, Mar 18, 2014 at 4:37 PM, Erick Erickson erickerick...@gmail.comwrote: Avishai: It sounds like you already understand mmap. Even so you might be interested in this excellent writeup of MMapDirectory and Lucene by Uwe:

RE: Solr memory usage off-heap

2014-03-18 Thread Doug Turnbull
How large is your index on disk? Solr memory maps the index into memory. Thus the virtual memory used will often be quite large. Your numbers don't sound inconceivable. A good reference point is Grant Ingersoll's blog post on searchhub:

Re: Solr memory usage off-heap

2014-03-18 Thread Shawn Heisey
On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: My solr instances are configured with 10GB heap (Xmx) but linux shows resident size of 16-20GB. even with thread stack and permgen taken into account i'm still far off from these numbers. Could it be that jvm IO buffers take so much space? does

Re: Solr memory usage off-heap

2014-03-18 Thread Erick Erickson
Avishai: It sounds like you already understand mmap. Even so you might be interested in this excellent writeup of MMapDirectory and Lucene by Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Best, Erick On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom

Re: Solr memory usage off-heap

2014-03-18 Thread Shawn Heisey
On 3/18/2014 8:37 AM, Erick Erickson wrote: It sounds like you already understand mmap. Even so you might be interested in this excellent writeup of MMapDirectory and Lucene by Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html There is some actual bad memory