Re: Solr RSIZE memory overusage

2018-09-11 Thread Erick Erickson
bq. We're using NRTCachingDirectoryFactory Which uses MMapDirectory under the covers. The file handle counts will vary. During merging, files are held open and while segments are merged so new and old segments are open. Once merged, the files in the old segment will be deleted so some variance

Re: Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
On Tue, 2018-09-11 at 12:43 -0600, Shawn Heisey wrote: > On 9/11/2018 12:14 PM, Boris Pasko wrote: > > > > > > > > Run top, press shift-M to sort by memory usage, then grab a > > atop: http://oi68.tinypic.com/10pokkk.jpg > > top: http://oi63.tinypic.com/msbpfp.jpg > Looking at the second one: > >

Re: Solr RSIZE memory overusage

2018-09-11 Thread Shawn Heisey
On 9/11/2018 12:14 PM, Boris Pasko wrote: Run top, press shift-M to sort by memory usage, then grab a atop: http://oi68.tinypic.com/10pokkk.jpg top: http://oi63.tinypic.com/msbpfp.jpg Looking at the second one: The SHR value is showing 90GB. Your Java process is in actuality only using in

Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
Hi. We're running Solr 6.6.1 (SolrCloud, 3 clusters). Recently I noticed it became significantly slower to respond and did some basic checks on servers. There is little IO, a bit of CPU usage (110% user, 3090% idle), but one thing is very strange - the resident memory usage of the Solr. Despite

Re: Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
>Run top, press shift-M to sort by memory usage, then grab a atop: http://oi68.tinypic.com/10pokkk.jpg top: http://oi63.tinypic.com/msbpfp.jpg – The information contained in this message and any attachments may be confidential and/or restricted and protected

Re: Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
On Tue, 2018-09-11 at 10:26 -0700, Erick Erickson wrote: > The memory usage is probably MMapDirectory, see: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.ht > ml, > that's not a problem I'd guess. We're using NRTCachingDirectoryFactory > The open file handles... and ones

Re: Solr RSIZE memory overusage

2018-09-11 Thread Shawn Heisey
On 9/11/2018 11:07 AM, Boris Pasko wrote: Hi. We're running Solr 6.6.1 (SolrCloud, 3 nodes). Recently I noticed it became significantly slower to respond and did some basic checks on servers. There is little IO, a bit of CPU usage (110% user, 3090% idle), but one thing is very strange - the

Re: Solr RSIZE memory overusage

2018-09-11 Thread Erick Erickson
The memory usage is probably MMapDirectory, see: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html, that's not a problem I'd guess. The open file handles... and ones being open many times. 1> how many _total_ segment files do you have in all the replicas in the JVM? A single

Re: Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
> Same picture on Solr 6.6.2, tested on various Oracle JVMs ranging > from 1.8.0_171 to 1.8.0_171 From 1.8.0_171 to 1.8.0_181 – The information contained in this message and any attachments may be confidential and/or restricted and protected from

Solr RSIZE memory overusage

2018-09-11 Thread Boris Pasko
Hi. We're running Solr 6.6.1 (SolrCloud, 3 nodes). Recently I noticed it became significantly slower to respond and did some basic checks on servers. There is little IO, a bit of CPU usage (110% user, 3090% idle), but one thing is very strange - the resident memory usage of the Solr. Despite the