Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Thank you for the detailed response Shawn! I've read it several times. Yes, that particular machine has 12 cores that are hyper-threaded. Does Solr do something special when not running in HDFS to allocate memory that would result in VIRT showing memory required for index data size? In my e

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Shawn Heisey
On 8/18/2017 1:05 PM, Joe Obernberger wrote: > Thank you Shawn.  Please see: > http://www.lovehorsepower.com/Vesta > for screen shots of top > (http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg) and > several screen shots over various times of jvisualvm. > > There is also the GC log and th

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Ah! Yes - that makes much more sense: CPU: http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_CPU.jpg Mem: http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_Mem.jpg -Joe On 8/18/2017 3:35 PM, Michael Braun wrote: When I recommended JVisualVM, specifically the "Sampling" portion of the app -

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Hi Walter - I see what you are saying, but the machine is not actively swapping (that would be the concern - right?) It's the CPU usage that I'm trying to figure out. Htop reports that there is about 20G of disk cache in use, and about 76G of RAM in use by programs. VIRT memory is what was re

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Walter Underwood
I see a server with 100Gb of memory and processes (java and jsvc) using 203Gb of virtual memory. Hmm. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 18, 2017, at 12:05 PM, Joe Obernberger > wrote: > > Thank you Shawn. Please see: > http://w

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Thank you Shawn. Please see: http://www.lovehorsepower.com/Vesta for screen shots of top (http://www.lovehorsepower.com/Vesta/VestaSolr6.6.0_top.jpg) and several screen shots over various times of jvisualvm. There is also the GC log and the regular solr.log for one server (named Vesta). Ple

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Shawn Heisey
On 8/18/2017 10:37 AM, Joe Obernberger wrote: > Indexing about 15 million documents per day across 100 shards on 45 > servers.  Up until about 350 million documents, each of the solr > instances was taking up about 1 core (100% CPU).  Recently, they all > jumped to 700%.  Is this normal?  Anything

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
I was able to attach to one server by changing the startup and adding: |-Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.local.only=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.port=18983 \

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Thank you Michael. Oddly when I start jstatd on one of the servers, I see all the JVM processes in jvisualvm except the solr one! Any idea why? On 8/18/2017 12:43 PM, Michael Braun wrote: Have you attached JVisualVM or a similar application to the process to sample where the time is being sp

Re: Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Michael Braun
Have you attached JVisualVM or a similar application to the process to sample where the time is being spent? It can be very helpful for debugging this sort of problem. On Fri, Aug 18, 2017 at 12:37 PM, Joe Obernberger < joseph.obernber...@gmail.com> wrote: > Indexing about 15 million documents pe

Solr 6.6.0 - High CPU during indexing

2017-08-18 Thread Joe Obernberger
Indexing about 15 million documents per day across 100 shards on 45 servers. Up until about 350 million documents, each of the solr instances was taking up about 1 core (100% CPU). Recently, they all jumped to 700%. Is this normal? Anything that I can check for? I don't see anything unusua