Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Bruce Ritchie]

You may want to try upgrading ES - release notes for 1.2.0 indicate a 
change wrt throttling indexing when merges fall behind and earlier release 
notes post 1.1.0 have notes about a potential memory leak fix among many 
other improvements and fixes.

Best I can think of :|


Bruce

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/58147218-c37f-467a-bdd6-3d7457b8dabd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Michael Hart]

Removing the "-XX+UseCMSInitiatingOccupancyOnly" flag extended the time it 
took before the JVM started full GC's from about 2 hours to 7 hours in my 
cluster, but now it's back to constant full GC's. I'm out of ideas. 
Suggestions?

mike


On Monday, June 23, 2014 10:25:20 AM UTC-4, Michael Hart wrote:
>
> My nodes are in Rackspace, so they are VM's, but they are configured 
> without swap.
>
> I'm not entirely sure what the searches are up to, I'm going to 
> investigate that further.
>
> I did correlate a rapid increase in Heap used, number of segments (up from 
> the norm of ~400 to 15,000) and consequently Old GC counts when the cluster 
> attempts to merge a 5GB segment. It seems that in spite of my really fast 
> disk the merge of a 5GB segment takes up to 15 minutes. I've made two 
> changes this morning, namely set these:
>
> index.merge.scheduler.max_thread_count: 3
> index.merge.policy.max_merged_segment: 1gb
>
> The first is in the hope that while a large segment merge is underway, the 
> two other threads can still keep the small segment merges going. The second 
> is to keep the larger segment merges under control. I was ending up with 
> two 5GB segments, and a long tail of smaller ones. A quick model shows that 
> by dropping this to 1GB I'll have 12 x 1GB segments and a similar long tail 
> of smaller segments (about 50?).
>
> I've also enabled GC logging on one node, I'll leave it running for the 
> day and tomorrow remove the "-XX+UseCMSInitiatingOccupancyOnly" flag (used 
> by default for elasticsearch) and see if there's any difference. I'll 
> report back here incase this is of any use for anyone.
>
> thanks
> mike
>
> On Friday, June 20, 2014 6:31:54 PM UTC-4, Clinton Gormley wrote:
>
> * Do you have swap disabled?  (any swap plays havoc with GC)
> * You said you're doing scan/scroll - how many documents are you 
> retrieving at a time? Consider reducing the number
> * Are you running on a VM - that can cause you to swap even though your VM 
> guest thinks that swap is disabled, or steal CPU (slowing down GC)
>
> Essentially, if all of the above are false, you shouldn't be getting slow 
> GCs unless you're under very heavy memory pressure (and I see that your old 
> gen is not too bad, so that doesn't look likely).
>
>
> On 20 June 2014 16:03, Michael Hart  wrote:
>
> Thanks I do see the GC warnings in the logs, such as
>
> [2014-06-19 20:17:06,603][WARN ][monitor.jvm  ] [redacted] [gc
> ][old][179386][22718] duration [11.4s], collections [1]/[12.2s], total [
> 11.4s]/[25.2m], memory [7.1gb]->[6.9gb]/[7.2gb], all_pools {[young] [
> 158.7mb]->[7.4mb]/[266.2mb]}{[survivor] [32.4mb]->[0b]/[33.2mb]}{[old] [
> 6.9gb]->[6.9gb]/[6.9gb]}
>
> CPU Idle is around 50% when the merge starts, and drops to zero by the 
> time that first GC old warning is logged. During recovery my SSD's sustain 
> 2400 IOPS and during yesterday's outage I only see about 800 IOPS before ES 
> died. While I can throw more hardware at it, I'd prefer to do some tuning 
> first if possible. 
>
> The reason I was thinking of adding more shards is that largest segment is 
> 4.9GB (just under the default maximum set 
> by index.merge.policy.max_merged_segment). I suppose the other option is to 
> reduce the index.merge.policy.max_merged_segment setting to something 
> smaller, but I have no idea what the implications are.
>
> thoughts?
> mike
>
> On Friday, June 20, 2014 9:47:22 AM UTC-4, Ankush Jhalani wrote:
>
> Mike - The above sounds like happened due to machines sending too many 
> indexing requests and merging unable to keep up pace. Usual suspects would 
> be not enough cpu/disk speed bandwidth. 
> This doesn't sound related to memory constraints posted in the original 
> issue of this thread. Do you see memory GC traces in logs? 
>
> On Friday, June 20, 2014 9:40:48 AM UTC-4, Michael Hart wrote:
>
> We're seeing the same thing. ES 1.1.0, JDK 7u55 on Ubuntu 12.04, 5 data 
> nodes, 3 separate masters, all are 15GB hosts with 7.5GB Heaps, storage is 
> SSD. Data set is ~1.6TB according to Marvel.
>
> Our daily indices are roughly 33GB in size, with 5 shards and 2 replicas. 
> I'm still investigating what happened yesterday, but I do see in Marvel a 
> large spike in the "Indices Current Merges" graph just before the node 
> dies, and a corresponding increase in JVM Heap. When Heap hits 99% 
> everything grinds to a halt. Restarting the node "fixes" the issue, but 
> this is third or fourth time it's happened.
>
> I'm still researching how to deal with this, but a couple of things I am 
> looking at are:
>
>- increase the number of shards so that the segment merges stay 
>smaller (is that even a legitimate sentence?) I'm still reading through 
>this page the Index Module Merge page 
>
> 
>  for 
>more details. 
>- look at store level throttling 
>
> 

Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Ankush Jhalani]

Mike - The above sounds like happened due to machines sending too many 
indexing requests and merging unable to keep up pace. Usual suspects would 
be not enough cpu/disk speed bandwidth. 
This doesn't sound related to memory constraints posted in the original 
issue of this thread. Do you see memory GC traces in logs? 

On Friday, June 20, 2014 9:40:48 AM UTC-4, Michael Hart wrote:
>
> We're seeing the same thing. ES 1.1.0, JDK 7u55 on Ubuntu 12.04, 5 data 
> nodes, 3 separate masters, all are 15GB hosts with 7.5GB Heaps, storage is 
> SSD. Data set is ~1.6TB according to Marvel.
>
> Our daily indices are roughly 33GB in size, with 5 shards and 2 replicas. 
> I'm still investigating what happened yesterday, but I do see in Marvel a 
> large spike in the "Indices Current Merges" graph just before the node 
> dies, and a corresponding increase in JVM Heap. When Heap hits 99% 
> everything grinds to a halt. Restarting the node "fixes" the issue, but 
> this is third or fourth time it's happened.
>
> I'm still researching how to deal with this, but a couple of things I am 
> looking at are:
>
>- increase the number of shards so that the segment merges stay 
>smaller (is that even a legitimate sentence?) I'm still reading through 
>this page the Index Module Merge page 
>
> 
>  for 
>more details.
>- look at store level throttling 
>
> 
>.
>
> I would love to get some feedback on my ramblings. If I find anything more 
> I'll update this thread.
>
> cheers
> mike
>
>
>
>
> On Thursday, June 19, 2014 4:05:54 PM UTC-4, Bruce Ritchie wrote:
>
> Java 8 with G1GC perhaps? It'll have more overhead but perhaps it'll be 
> more consistent wrt pauses.
>
>
>
> On Wednesday, June 18, 2014 2:02:24 PM UTC-4, Eric Brandes wrote:
>
> I'd just like to chime in with a "me too".  Is the answer just more 
> nodes?  In my case this is happening every week or so.
>
> On Monday, April 21, 2014 9:04:33 PM UTC-5, Brian Flad wrote:
>
> My dataset currently is 100GB across a few "daily" indices (~5-6GB and 15 
> shards each). Data nodes are 12 CPU, 12GB RAM (6GB heap).
>
>
> On Mon, Apr 21, 2014 at 6:33 PM, Mark Walkom  
> wrote:
>
> How big are your data sets? How big are your nodes?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 22 April 2014 00:32, Brian Flad  wrote:
>
> We're seeing the same behavior with 1.1.1, JDK 7u55, 3 master nodes (2 min 
> master), and 5 data nodes. Interestingly, we see the repeated young GCs 
> only on a node or two at a time. Cluster operations (such as recovering 
> unassigned shards) grinds to a halt. After restarting a GCing node, 
> everything returns to normal operation in the cluster.
>
> Brian F
>
>
> On Wed, Apr 16, 2014 at 8:00 PM, Mark Walkom  
> wrote:
>
> In both your instances, if you can, have 3 master eligible nodes as it 
> will reduce the likelihood of a split cluster as you will always have a 
> majority quorum. Also look at discovery.zen.minimum_master_nodes to go with 
> that.
> However you may just be reaching the limit of your nodes, which means the 
> best option is to add another node (which also neatly solves your split 
> brain!).
>
> Ankush it would help if you can update java, most people recommend u25 but 
> we run u51 with no problems.
>
>
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 17 April 2014 07:31, Dominiek ter Heide  wrote:
>
> We are seeing the same issue here. 
>
> Our environment:
>
> - 2 nodes
> - 30GB Heap allocated to ES
> - ~140GB of data
> - 639 indices, 10 shards per index
> - ~48M documents
>
> After starting ES everything is good, but after a couple of hours we see 
> the Heap build up towards 96% on one node and 80% on the other. We then see 
> the GC take very long on the 96% node:
>
>
>
>
>
>
>
>
>
> TOuKgmlzaVaFVA][elasticsearch1.trend1.bottlenose.com][inet[/192.99.45.125:
> 9300]]])
>
> [2014-04-16 12:04:27,845][INFO ][discovery] 
> [elasticsearch2.trend1] trend1/I3EHG_XjSayz2OsHyZpeZA
>
> [2014-04-16 12:04:27,850][INFO ][http ] [
> elasticsearch2.trend1] bound_address {inet[/0.0.0.0:9200]}, 
> publish_address {inet[/192.99.45.126:9200]}
>
> [2014-04-16 12:04:27,851][INFO ][node ] 
> [elasticsearch2.trend1] started
>
> [2014-04-16 12:04:32,669][INFO ][indices.store] 
> [elasticsearch2.trend1] updating indices.store.throttle.max_bytes_per_sec 
> from [20mb] to [1gb], note, type is [MERGE]
>
> [2014-04-16 12:04:32,669][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_r

Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Michael Hart]

We're seeing the same thing. ES 1.1.0, JDK 7u55 on Ubuntu 12.04, 5 data 
nodes, 3 separate masters, all are 15GB hosts with 7.5GB Heaps, storage is 
SSD. Data set is ~1.6TB according to Marvel.

Our daily indices are roughly 33GB in size, with 5 shards and 2 replicas. 
I'm still investigating what happened yesterday, but I do see in Marvel a 
large spike in the "Indices Current Merges" graph just before the node 
dies, and a corresponding increase in JVM Heap. When Heap hits 99% 
everything grinds to a halt. Restarting the node "fixes" the issue, but 
this is third or fourth time it's happened.

I'm still researching how to deal with this, but a couple of things I am 
looking at are:

   - increase the number of shards so that the segment merges stay smaller 
   (is that even a legitimate sentence?) I'm still reading through this page 
the 
   Index Module Merge page 
   

 for 
   more details.
   - look at store level throttling 
   

   .

I would love to get some feedback on my ramblings. If I find anything more 
I'll update this thread.

cheers
mike




On Thursday, June 19, 2014 4:05:54 PM UTC-4, Bruce Ritchie wrote:
>
> Java 8 with G1GC perhaps? It'll have more overhead but perhaps it'll be 
> more consistent wrt pauses.
>
>
>
> On Wednesday, June 18, 2014 2:02:24 PM UTC-4, Eric Brandes wrote:
>>
>> I'd just like to chime in with a "me too".  Is the answer just more 
>> nodes?  In my case this is happening every week or so.
>>
>> On Monday, April 21, 2014 9:04:33 PM UTC-5, Brian Flad wrote:
>>
>> My dataset currently is 100GB across a few "daily" indices (~5-6GB and 15 
>> shards each). Data nodes are 12 CPU, 12GB RAM (6GB heap).
>>
>>
>> On Mon, Apr 21, 2014 at 6:33 PM, Mark Walkom  
>> wrote:
>>
>> How big are your data sets? How big are your nodes?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 22 April 2014 00:32, Brian Flad  wrote:
>>
>> We're seeing the same behavior with 1.1.1, JDK 7u55, 3 master nodes (2 
>> min master), and 5 data nodes. Interestingly, we see the repeated young GCs 
>> only on a node or two at a time. Cluster operations (such as recovering 
>> unassigned shards) grinds to a halt. After restarting a GCing node, 
>> everything returns to normal operation in the cluster.
>>
>> Brian F
>>
>>
>> On Wed, Apr 16, 2014 at 8:00 PM, Mark Walkom  
>> wrote:
>>
>> In both your instances, if you can, have 3 master eligible nodes as it 
>> will reduce the likelihood of a split cluster as you will always have a 
>> majority quorum. Also look at discovery.zen.minimum_master_nodes to go with 
>> that.
>> However you may just be reaching the limit of your nodes, which means the 
>> best option is to add another node (which also neatly solves your split 
>> brain!).
>>
>> Ankush it would help if you can update java, most people recommend u25 
>> but we run u51 with no problems.
>>
>>
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 17 April 2014 07:31, Dominiek ter Heide  wrote:
>>
>> We are seeing the same issue here. 
>>
>> Our environment:
>>
>> - 2 nodes
>> - 30GB Heap allocated to ES
>> - ~140GB of data
>> - 639 indices, 10 shards per index
>> - ~48M documents
>>
>> After starting ES everything is good, but after a couple of hours we see 
>> the Heap build up towards 96% on one node and 80% on the other. We then see 
>> the GC take very long on the 96% node:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> TOuKgmlzaVaFVA][elasticsearch1.trend1.bottlenose.com][inet[/192.99.45.125
>> :9300]]])
>>
>> [2014-04-16 12:04:27,845][INFO ][discovery] 
>> [elasticsearch2.trend1] trend1/I3EHG_XjSayz2OsHyZpeZA
>>
>> [2014-04-16 12:04:27,850][INFO ][http ] [
>> elasticsearch2.trend1] bound_address {inet[/0.0.0.0:9200]}, 
>> publish_address {inet[/192.99.45.126:9200]}
>>
>> [2014-04-16 12:04:27,851][INFO ][node ] 
>> [elasticsearch2.trend1] started
>>
>> [2014-04-16 12:04:32,669][INFO ][indices.store] 
>> [elasticsearch2.trend1] updating indices.store.throttle.max_bytes_per_sec 
>> from [20mb] to [1gb], note, type is [MERGE]
>>
>> [2014-04-16 12:04:32,669][INFO ][cluster.routing.allocation.decider] 
>> [elasticsearch2.trend1] updating 
>> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
>> to [50]
>>
>> [2014-04-16 12:04:32,670][INFO ][indices.recovery ] 
>> [elasticsearch2.trend1] updating [indices.recovery.max_bytes_per_sec] 
>> from [200mb] to [2gb]
>>
>> [2014-04-16 12:04:32,670][INFO ][cluster.routing.allocation.decider] 
>> [elasticsearch2.trend1] updating 
>> [cluster.routing.allocation.node

Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Bruce Ritchie]

Java 8 with G1GC perhaps? It'll have more overhead but perhaps it'll be 
more consistent wrt pauses.



On Wednesday, June 18, 2014 2:02:24 PM UTC-4, Eric Brandes wrote:
>
> I'd just like to chime in with a "me too".  Is the answer just more 
> nodes?  In my case this is happening every week or so.
>
> On Monday, April 21, 2014 9:04:33 PM UTC-5, Brian Flad wrote:
>
> My dataset currently is 100GB across a few "daily" indices (~5-6GB and 15 
> shards each). Data nodes are 12 CPU, 12GB RAM (6GB heap).
>
>
> On Mon, Apr 21, 2014 at 6:33 PM, Mark Walkom  
> wrote:
>
> How big are your data sets? How big are your nodes?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 22 April 2014 00:32, Brian Flad  wrote:
>
> We're seeing the same behavior with 1.1.1, JDK 7u55, 3 master nodes (2 min 
> master), and 5 data nodes. Interestingly, we see the repeated young GCs 
> only on a node or two at a time. Cluster operations (such as recovering 
> unassigned shards) grinds to a halt. After restarting a GCing node, 
> everything returns to normal operation in the cluster.
>
> Brian F
>
>
> On Wed, Apr 16, 2014 at 8:00 PM, Mark Walkom  
> wrote:
>
> In both your instances, if you can, have 3 master eligible nodes as it 
> will reduce the likelihood of a split cluster as you will always have a 
> majority quorum. Also look at discovery.zen.minimum_master_nodes to go with 
> that.
> However you may just be reaching the limit of your nodes, which means the 
> best option is to add another node (which also neatly solves your split 
> brain!).
>
> Ankush it would help if you can update java, most people recommend u25 but 
> we run u51 with no problems.
>
>
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 17 April 2014 07:31, Dominiek ter Heide  wrote:
>
> We are seeing the same issue here. 
>
> Our environment:
>
> - 2 nodes
> - 30GB Heap allocated to ES
> - ~140GB of data
> - 639 indices, 10 shards per index
> - ~48M documents
>
> After starting ES everything is good, but after a couple of hours we see 
> the Heap build up towards 96% on one node and 80% on the other. We then see 
> the GC take very long on the 96% node:
>
>
>
>
>
>
>
>
>
> TOuKgmlzaVaFVA][elasticsearch1.trend1.bottlenose.com][inet[/192.99.45.125:
> 9300]]])
>
> [2014-04-16 12:04:27,845][INFO ][discovery] 
> [elasticsearch2.trend1] trend1/I3EHG_XjSayz2OsHyZpeZA
>
> [2014-04-16 12:04:27,850][INFO ][http ] [
> elasticsearch2.trend1] bound_address {inet[/0.0.0.0:9200]}, 
> publish_address {inet[/192.99.45.126:9200]}
>
> [2014-04-16 12:04:27,851][INFO ][node ] 
> [elasticsearch2.trend1] started
>
> [2014-04-16 12:04:32,669][INFO ][indices.store] 
> [elasticsearch2.trend1] updating indices.store.throttle.max_bytes_per_sec 
> from [20mb] to [1gb], note, type is [MERGE]
>
> [2014-04-16 12:04:32,669][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 12:04:32,670][INFO ][indices.recovery ] 
> [elasticsearch2.trend1] updating [indices.recovery.max_bytes_per_sec] from 
> [200mb] to [2gb]
>
> [2014-04-16 12:04:32,670][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 12:04:32,670][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 15:25:21,409][WARN ][monitor.jvm  ] 
> [elasticsearch2.trend1] [gc][old][11876][106] duration [1.1m], 
> collections [1]/[1.1m], total [1.1m]/[1.4m], memory [28.7gb]->[22gb]/[
> 29.9gb], all_pools {[young] [67.9mb]->[268.9mb]/[665.6mb]}{[survivor] [
> 60.5mb]->[0b]/[83.1mb]}{[old] [28.6gb]->[21.8gb]/[29.1gb]}
>
> [2014-04-16 16:02:32,523][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][13996][144] duration [1.4m], collections 
> [1]/[1.4m], total [1.4m]/[3m], memory [28.8gb]->[23.5gb]/[29.9gb], 
> all_pools {[young] [21.8mb]->[238.2mb]/[665.6mb]}{[survivor] [82.4mb]->[0b
> ]/[83.1mb]}{[old] [28.7gb]->[23.3gb]/[29.1gb]}
>
> [2014-04-16 16:14:12,386][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][14603][155] duration [1.3m], collections 
> [2]/[1.3m], total [1.3m]/[4.4m], memory [29.2gb]->[23.9gb]/[29.9gb], 
> all_pools {[young] [289mb]->[161.3mb]/[665.6mb]}{[survivor] [58.3mb]->[0b
> ]/[83.1mb]}{[old] [28.8gb]->[23.8gb]/[29.1gb]}
>
> [2014-04-16 16:17:55,480][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][14745][158] duration [1.3m], collections 
> [1]/[1.3m], total [1.3m]/[5.7m], memory [29.7gb]->[24.1gb]/[29.9g

Re: ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Eric Brandes]

I'd just like to chime in with a "me too".  Is the answer just more nodes?  
In my case this is happening every week or so.

On Monday, April 21, 2014 9:04:33 PM UTC-5, Brian Flad wrote:
>
> My dataset currently is 100GB across a few "daily" indices (~5-6GB and 15 
> shards each). Data nodes are 12 CPU, 12GB RAM (6GB heap).
>
>
> On Mon, Apr 21, 2014 at 6:33 PM, Mark Walkom  > wrote:
>
> How big are your data sets? How big are your nodes?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 22 April 2014 00:32, Brian Flad > 
> wrote:
>
> We're seeing the same behavior with 1.1.1, JDK 7u55, 3 master nodes (2 min 
> master), and 5 data nodes. Interestingly, we see the repeated young GCs 
> only on a node or two at a time. Cluster operations (such as recovering 
> unassigned shards) grinds to a halt. After restarting a GCing node, 
> everything returns to normal operation in the cluster.
>
> Brian F
>
>
> On Wed, Apr 16, 2014 at 8:00 PM, Mark Walkom  > wrote:
>
> In both your instances, if you can, have 3 master eligible nodes as it 
> will reduce the likelihood of a split cluster as you will always have a 
> majority quorum. Also look at discovery.zen.minimum_master_nodes to go with 
> that.
> However you may just be reaching the limit of your nodes, which means the 
> best option is to add another node (which also neatly solves your split 
> brain!).
>
> Ankush it would help if you can update java, most people recommend u25 but 
> we run u51 with no problems.
>
>
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 17 April 2014 07:31, Dominiek ter Heide  > wrote:
>
> We are seeing the same issue here. 
>
> Our environment:
>
> - 2 nodes
> - 30GB Heap allocated to ES
> - ~140GB of data
> - 639 indices, 10 shards per index
> - ~48M documents
>
> After starting ES everything is good, but after a couple of hours we see 
> the Heap build up towards 96% on one node and 80% on the other. We then see 
> the GC take very long on the 96% node:
>
>
>
>
>
>
>
>
>
> TOuKgmlzaVaFVA][elasticsearch1.trend1.bottlenose.com][inet[/192.99.45.125:
> 9300]]])
>
> [2014-04-16 12:04:27,845][INFO ][discovery] 
> [elasticsearch2.trend1] trend1/I3EHG_XjSayz2OsHyZpeZA
>
> [2014-04-16 12:04:27,850][INFO ][http ] [
> elasticsearch2.trend1] bound_address {inet[/0.0.0.0:9200]}, 
> publish_address {inet[/192.99.45.126:9200]}
>
> [2014-04-16 12:04:27,851][INFO ][node ] 
> [elasticsearch2.trend1] started
>
> [2014-04-16 12:04:32,669][INFO ][indices.store] 
> [elasticsearch2.trend1] updating indices.store.throttle.max_bytes_per_sec 
> from [20mb] to [1gb], note, type is [MERGE]
>
> [2014-04-16 12:04:32,669][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 12:04:32,670][INFO ][indices.recovery ] 
> [elasticsearch2.trend1] updating [indices.recovery.max_bytes_per_sec] from 
> [200mb] to [2gb]
>
> [2014-04-16 12:04:32,670][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 12:04:32,670][INFO ][cluster.routing.allocation.decider] 
> [elasticsearch2.trend1] updating 
> [cluster.routing.allocation.node_initial_primaries_recoveries] from [4] 
> to [50]
>
> [2014-04-16 15:25:21,409][WARN ][monitor.jvm  ] 
> [elasticsearch2.trend1] [gc][old][11876][106] duration [1.1m], 
> collections [1]/[1.1m], total [1.1m]/[1.4m], memory [28.7gb]->[22gb]/[
> 29.9gb], all_pools {[young] [67.9mb]->[268.9mb]/[665.6mb]}{[survivor] [
> 60.5mb]->[0b]/[83.1mb]}{[old] [28.6gb]->[21.8gb]/[29.1gb]}
>
> [2014-04-16 16:02:32,523][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][13996][144] duration [1.4m], collections 
> [1]/[1.4m], total [1.4m]/[3m], memory [28.8gb]->[23.5gb]/[29.9gb], 
> all_pools {[young] [21.8mb]->[238.2mb]/[665.6mb]}{[survivor] [82.4mb]->[0b
> ]/[83.1mb]}{[old] [28.7gb]->[23.3gb]/[29.1gb]}
>
> [2014-04-16 16:14:12,386][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][14603][155] duration [1.3m], collections 
> [2]/[1.3m], total [1.3m]/[4.4m], memory [29.2gb]->[23.9gb]/[29.9gb], 
> all_pools {[young] [289mb]->[161.3mb]/[665.6mb]}{[survivor] [58.3mb]->[0b
> ]/[83.1mb]}{[old] [28.8gb]->[23.8gb]/[29.1gb]}
>
> [2014-04-16 16:17:55,480][WARN ][monitor.jvm  ] [
> elasticsearch2.trend1] [gc][old][14745][158] duration [1.3m], collections 
> [1]/[1.3m], total [1.3m]/[5.7m], memory [29.7gb]->[24.1gb]/[29.9gb], 
> all_pools {[young] [633.8mb]->[149.7mb]/[665.6mb]}{[survivor] [68.6mb]->[
> 0b]/[83.1mb]}{[old] [29gb]->[24gb]/[29.1gb]}
>
> [2014-04-16 16:21:17,950][WARN ][monitor.

ES v1.1 continuous young gc pauses old gc, stops the world when old gc happens and splits cluster

[Ankush Jhalani]

We have a 4 node (2 client only, 2 data/master nodes with 25G memory 
allocated to ES and 12 cores each) ES cluster, storing an index with 16 
shards, ~200GB and 1 replica. 

Recently running scan/scroll requests to dump data and other faceting 
requests, the nodes disconnected from each and we had a split condition. 
All requests were being run sequentially, not in parellel. From the logs we 
notice that for 5-10 minutes, just the young gc was running even though old 
memory and total memory used was over 75% (which I think is the default 
CMSInitiatingOccupancyFraction=75). When finally old gc ran, it took 1.7 
minutes which caused nodes to disconnect (3 pings of 30 seconds failed). 
 Below are some of the last traces from log - 

[2014-04-15 22:07:33,526][INFO ][monitor.jvm  ] [ny1.node2] 
[gc][young][149283][9270] duration [782ms], collections [1]/[1s], total 
[782ms]/[8.2m], memory [21.5gb]->[20.8gb]/[24.8gb], all_pools {[young] 
[1gb]->[147mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old] [20
.3gb]->[20.5gb]/[23.5gb]}
[2014-04-15 22:07:35,264][INFO ][monitor.jvm  ] [ny1.node2] 
[gc][young][149284][9271] duration [743ms], collections [1]/[1.7s], total 
[743ms]/[8.2m], memory [20.8gb]->[20.8gb]/[24.8gb], all_pools {[young] 
[147mb]->[10.1mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old
] [20.5gb]->[20.6gb]/[23.5gb]}
[2014-04-15 22:07:36,814][INFO ][monitor.jvm  ] [ny1.node2] 
[gc][young][149285][9272] duration [786ms], collections [1]/[1.5s], total 
[786ms]/[8.2m], memory [20.8gb]->[20.9gb]/[24.8gb], all_pools {[young] 
[10.1mb]->[2.8mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old
] [20.6gb]->[20.8gb]/[23.5gb]}
[2014-04-15 22:07:38,880][INFO ][monitor.jvm  ] [ny1.node2] 
[gc][young][149287][9273] duration [835ms], collections [1]/[1s], total 
[835ms]/[8.2m], memory [21.5gb]->[21.1gb]/[24.8gb], all_pools {[young] 
[655.9mb]->[1.2mb]/[1.1gb]}{[survivor] [149.7mb]->[149.7mb]/[149.7mb]}{[old]
 [20.7gb]->[20.9gb]/[23.5gb]}
[2014-04-15 22:09:24,215][INFO ][monitor.jvm  ] [ny1.node2] 
[gc][young][149290][9274] duration [786ms], collections [1]/[1.7m], total 
[786ms]/[8.3m], memory [21.7gb]->[2.4gb]/[24.8gb], all_pools {[young] 
[727.2mb]->[13.8mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [2
0.8gb]->[2.4gb]/[23.5gb]}
[2014-04-15 22:09:24,215][WARN ][monitor.jvm  ] [ny1.node2] 
[gc][old][149290][25] duration [1.7m], collections [2]/[1.7m], total 
[1.7m]/[1.7m], memory [21.7gb]->[2.4gb]/[24.8gb], all_pools {[young] 
[727.2mb]->[13.8mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
[20.8gb]
->[2.4gb]/[23.5gb]}

CPU usage was pretty low on the machine, but it's confusing why the old gc 
was put on pause for so long and why did it take so much when it finally 
ran? We currently use Java 7 update 10, would the latest 51 help?  Would 
switching to GC1 help ? 

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2bb88e62-4240-442e-b3aa-16cfd7817550%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.