Re: OOM Error

2016-11-09 Thread Susheel Kumar
Thanks, Shawn for looking into. Your summption is right, the end of graph is the OOM. I am trying to collect all the queries & ingestion numbers around 9:12 but one more observation and a question from today Observed that on 2-3 VM's out of 12, shows high usage of heap even though heavy ingestion

Re: OOM Error

2016-11-09 Thread Shawn Heisey
On 11/8/2016 12:49 PM, Susheel Kumar wrote: > Ran into OOM Error again right after two weeks. Below is the GC log > viewer graph. The first time we run into this was after 3 months and > then second time in two weeks. After first incident reduced the cache > size and increase heap from 8 to 10G. In

Re: OOM Error

2016-11-08 Thread Susheel Kumar
Hello, Ran into OOM Error again right after two weeks. Below is the GC log viewer graph. The first time we run into this was after 3 months and then second time in two weeks. After first incident reduced the cache size and increase heap from 8 to 10G. Interestingly query and ingestion load is li

Re: OOM Error

2016-10-26 Thread Susheel Kumar
Hi Toke, I think your guess is right. We have ingestion running in batches. We have 6 shards & 6 replicas on 12 VM's each around 40+ million docs on each shard. Thanks everyone for the suggestions/pointers. Thanks, Susheel On Wed, Oct 26, 2016 at 1:52 AM, Toke Eskildsen wrote: > On Tue, 201

Re: OOM Error

2016-10-26 Thread Tom Evans
On Wed, Oct 26, 2016 at 4:53 AM, Shawn Heisey wrote: > On 10/25/2016 8:03 PM, Susheel Kumar wrote: >> Agree, Pushkar. I had docValues for sorting / faceting fields from >> begining (since I setup Solr 6.0). So good on that side. I am going to >> analyze the queries to find any potential issue. T

Re: OOM Error

2016-10-25 Thread Toke Eskildsen
On Tue, 2016-10-25 at 15:04 -0400, Susheel Kumar wrote: > Thanks, Toke.  Analyzing GC logs helped to determine that it was a > sudden > death.   > The peaks in last 20 mins... See   http://tinypic.com/r/n2zonb/9 Peaks yes, but there is a pattern of  1) Stable memory use 2) Temporary doubling of

Re: OOM Error

2016-10-25 Thread Shawn Heisey
On 10/25/2016 8:03 PM, Susheel Kumar wrote: > Agree, Pushkar. I had docValues for sorting / faceting fields from > begining (since I setup Solr 6.0). So good on that side. I am going to > analyze the queries to find any potential issue. Two questions which I am > puzzling with > > a) Should the b

Re: OOM Error

2016-10-25 Thread Erick Erickson
Off the top of my head: a) Should the below JVM parameter be included for Prod to get heap dump Makes sense. It may produce quite a large dump file, but then this is an extraordinary situation so that's probably OK. b) Currently OOM script just kills the Solr instance. Shouldn't it be enhanced t

Re: OOM Error

2016-10-25 Thread Susheel Kumar
Agree, Pushkar. I had docValues for sorting / faceting fields from begining (since I setup Solr 6.0). So good on that side. I am going to analyze the queries to find any potential issue. Two questions which I am puzzling with a) Should the below JVM parameter be included for Prod to get heap dum

Re: OOM Error

2016-10-25 Thread Pushkar Raste
You should look into using docValues. docValues are stored off heap and hence you would be better off than just bumping up the heap. Don't enable docValues on existing fields unless you plan to reindex data from scratch. On Oct 25, 2016 3:04 PM, "Susheel Kumar" wrote: > Thanks, Toke. Analyzin

Re: OOM Error

2016-10-25 Thread Susheel Kumar
Thanks, Toke. Analyzing GC logs helped to determine that it was a sudden death. The peaks in last 20 mins... See http://tinypic.com/r/n2zonb/9 Will look into the queries more closer and also adjusting the cache sizing. Thanks, Susheel On Tue, Oct 25, 2016 at 3:37 AM, Toke Eskildsen wrote:

Re: OOM Error

2016-10-25 Thread William Bell
I would also recommend that 8GB is cutting it close for Java 8 JVM with SOLR. We use 12GB and have had issues with 8GB. But your mileage may vary. On Tue, Oct 25, 2016 at 1:37 AM, Toke Eskildsen wrote: > On Mon, 2016-10-24 at 18:27 -0400, Susheel Kumar wrote: > > I am seeing OOM script killed so

Re: OOM Error

2016-10-25 Thread Toke Eskildsen
On Mon, 2016-10-24 at 18:27 -0400, Susheel Kumar wrote: > I am seeing OOM script killed solr (solr 6.0.0) on couple of our VM's > today. So far our solr cluster has been running fine but suddenly > today many of the VM's Solr instance got killed. As you have the GC-logs, you should be able to dete

Re: OOM Error

2016-10-24 Thread Susheel Kumar
Thanks, Pushkar. The Solr was already killed by OOM script so i believe we can't get heap dump. Hi Shawn, I used Solr service scripts to launch Solr and it looks like bin/solr doesn't include by default the below JVM parameter. "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/the/dump"

Re: OOM Error

2016-10-24 Thread Shawn Heisey
On 10/24/2016 4:27 PM, Susheel Kumar wrote: > I am seeing OOM script killed solr (solr 6.0.0) on couple of our VM's > today. So far our solr cluster has been running fine but suddenly today > many of the VM's Solr instance got killed. I had 8G of heap allocated on 64 > GB machines with 20+ GB of in

Re: OOM Error

2016-10-24 Thread Pushkar Raste
Did you look into the heap dump ? On Mon, Oct 24, 2016 at 6:27 PM, Susheel Kumar wrote: > Hello, > > I am seeing OOM script killed solr (solr 6.0.0) on couple of our VM's > today. So far our solr cluster has been running fine but suddenly today > many of the VM's Solr instance got killed. I had

Re: OOM error during merge - index still ok?

2009-09-25 Thread Yonik Seeley
On Fri, Sep 25, 2009 at 8:20 AM, Phillip Farber wrote: >  Can I expect the index to be left in a usable state ofter an out of memory > error during a merge or it it most likely to be corrupt? It should be in the state it was after the last successful commit. -Yonik http://www.lucidimagination.co