Re: solr blocking and client timeout issue

2015-07-23 Thread Jeremy Ashcraft
: Shawn Heisey apa...@elyograg.org Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded

Re: solr blocking and client timeout issue

2015-07-23 Thread Erick Erickson
Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting

Re: solr blocking and client timeout issue

2015-07-21 Thread Jeremy Ashcraft
Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting the big GC pauses this morning, even after applying the GC tuning options

Re: solr blocking and client timeout issue

2015-07-21 Thread Daniel Collins
Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting the big GC pauses this morning, even after applying the GC tuning options. Everything was fine throughout the weekend. My biggest concern is that this instance had been

Re: solr blocking and client timeout issue

2015-07-20 Thread Shawn Heisey
On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting the big GC pauses this morning, even after applying the GC tuning options. Everything was fine throughout the weekend. My

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
? From: Shawn Heisey apa...@elyograg.org Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get

Re: solr blocking and client timeout issue

2015-07-20 Thread Jeremy Ashcraft
that be causing some of the problems? From: Shawn Heisey apa...@elyograg.org Sent: Monday, July 20, 2015 11:44 AM To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading

Re: solr blocking and client timeout issue

2015-07-20 Thread Erick Erickson
blocking and client timeout issue On 7/20/2015 11:54 AM, Jeremy Ashcraft wrote: I'm ugrading to the 1.8 JDK on our dev VM now and testing. Hopefully i can get production upgraded tonight. still getting the big GC pauses this morning, even after applying the GC tuning options. Everything was fine

Re: solr blocking and client timeout issue

2015-07-19 Thread Shawn Heisey
On 7/19/2015 12:46 AM, Jeremy Ashcraft wrote: That did the trick. The GC tuning options also seems to be working, but I guess we'll see when traffic ramps back up on monday. Thanks for all your help! On 7/18/2015 8:16 AM, Shawn Heisey wrote: The first thing I'd try is removing the

Re: solr blocking and client timeout issue

2015-07-19 Thread Jeremy Ashcraft
That did the trick. The GC tuning options also seems to be working, but I guess we'll see when traffic ramps back up on monday. Thanks for all your help! On 7/18/2015 8:16 AM, Shawn Heisey wrote: The first thing I'd try is removing the UseLargePages option and see if it goes away.

Re: solr blocking and client timeout issue

2015-07-18 Thread Jeremy Ashcraft
I turned on GC logging and verified that its definitely being caused by a GC pause. I tried the tuning option from the article and get this warning: OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory (errno = 1). any recommendations on how to get rid of that warning, and

Re: solr blocking and client timeout issue

2015-07-18 Thread Shawn Heisey
On 7/18/2015 12:42 AM, Jeremy Ashcraft wrote: I turned on GC logging and verified that its definitely being caused by a GC pause. I tried the tuning option from the article and get this warning: OpenJDK 64-Bit Server VM warning: Failed to reserve shared memory (errno = 1). any

Re: solr blocking and client timeout issue

2015-07-18 Thread Jeremy Ashcraft
I will try that. traffic is pretty dead over the weekend, so I probably won't be able to tell if its effective or not until monday. thanks again! On 7/18/2015 8:16 AM, Shawn Heisey wrote: On 7/18/2015 12:42 AM, Jeremy Ashcraft wrote: I turned on GC logging and verified that its definitely

Re: solr blocking and client timeout issue

2015-07-17 Thread Shawn Heisey
On 7/17/2015 4:02 PM, Jeremy Ashcraft wrote: having an issue where solr becomes unresponsive for unknown reasons. Client requests timeout for minutes at a time (sometimes only some requests time out while others work fine). The logs don't reveal any clues, other than just a big gap snip

Re: solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
To: solr-user@lucene.apache.org Subject: Re: solr blocking and client timeout issue Since there's basically no information about your setup here, my best guess is that you're seeing huge garbage collection pauses. There are two problems that can lead to long GC pauses. One is a super large heap

solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
having an issue where solr becomes unresponsive for unknown reasons. Client requests timeout for minutes at a time (sometimes only some requests time out while others work fine). The logs don't reveal any clues, other than just a big gap example: INFO - 2015-07-17 14:39:57.195;

Re: solr blocking and client timeout issue

2015-07-17 Thread Jeremy Ashcraft
thanks! i will definitely try some of those and let you know how it turns out On 7/17/2015 4:14 PM, Shawn Heisey wrote: On 7/17/2015 4:39 PM, Jeremy Ashcraft wrote: Solr 4.4.0 /usr/bin/java -version java version 1.7.0_25 OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64) OpenJDK 64-Bit

Re: solr blocking and client timeout issue

2015-07-17 Thread Shawn Heisey
On 7/17/2015 4:39 PM, Jeremy Ashcraft wrote: Solr 4.4.0 /usr/bin/java -version java version 1.7.0_25 OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64) OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) CentOS6 /usr/bin/java -Xms1G -Xmx4G -Dsolr.solr.home=/opt/solr/solr