[jira] [Updated] (CASSANDRA-10449) OOM on bootstrap due to long GC pause

2015-10-07 Thread Robbie Strickland (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbie Strickland updated CASSANDRA-10449:
--
Attachment: thread_dump.log

I've attached a thread dump taken after the streaming hangs.

> OOM on bootstrap due to long GC pause
> -
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
> Attachments: system.log.10-05, thread_dump.log
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10449) OOM on bootstrap due to long GC pause

2015-10-06 Thread Robbie Strickland (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbie Strickland updated CASSANDRA-10449:
--
Attachment: system.log.10-05

> OOM on bootstrap due to long GC pause
> -
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
> Attachments: system.log.10-05
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10449) OOM on bootstrap due to long GC pause

2015-10-05 Thread Robbie Strickland (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbie Strickland updated CASSANDRA-10449:
--
Environment: Ubuntu 14.04, AWS  (was: Ubuntu 14.04)
Description: 
I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
provision additional nodes, but bootstrapping OOMs every time after about 10 
hours with a sudden long GC pause:

{noformat}
INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
...
ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
CassandraDaemon.java:223 - Exception in thread 
Thread[MemtableFlushWriter:454,5,main]
java.lang.OutOfMemoryError: Java heap space
{noformat}

I have tried increasing max heap to 48G just to get through the bootstrap, to 
no avail.

  was:
I have a 20-node cluster with vnodes (default of 256) and 500-700GB per node.  
SSTable counts are <10 per table.  I am attempting to provision additional 
nodes, but bootstrapping OOMs every time after about 10 hours with a sudden 
long GC pause:

{noformat}
INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
...
ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
CassandraDaemon.java:223 - Exception in thread 
Thread[MemtableFlushWriter:454,5,main]
java.lang.OutOfMemoryError: Java heap space
{noformat}

I have tried increasing max heap to 48G just to get through the bootstrap, to 
no avail.


> OOM on bootstrap due to long GC pause
> -
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10449) OOM on bootstrap due to long GC pause

2015-10-05 Thread Robbie Strickland (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbie Strickland updated CASSANDRA-10449:
--
Description: 
I have a 20-node cluster with vnodes (default of 256) and 500-700GB per node.  
SSTable counts are <10 per table.  I am attempting to provision additional 
nodes, but bootstrapping OOMs every time after about 10 hours with a sudden 
long GC pause:

{noformat}
INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
...
ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
CassandraDaemon.java:223 - Exception in thread 
Thread[MemtableFlushWriter:454,5,main]
java.lang.OutOfMemoryError: Java heap space
{noformat}

I have tried increasing max heap to 48G just to get through the bootstrap, to 
no avail.

  was:
I have a 20-node cluster with vnodes (default of 256) and 500-700GB per node.  
SSTable counts are <10 per node.  I am attempting to provision additional 
nodes, but bootstrapping OOMs every time after about 10 hours with a sudden 
long GC pause:

{noformat}
INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
...
ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
CassandraDaemon.java:223 - Exception in thread 
Thread[MemtableFlushWriter:454,5,main]
java.lang.OutOfMemoryError: Java heap space
{noformat}

I have tried increasing max heap to 48G just to get through the bootstrap, to 
no avail.


> OOM on bootstrap due to long GC pause
> -
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04
>Reporter: Robbie Strickland
>  Labels: gc
>
> I have a 20-node cluster with vnodes (default of 256) and 500-700GB per node. 
>  SSTable counts are <10 per table.  I am attempting to provision additional 
> nodes, but bootstrapping OOMs every time after about 10 hours with a sudden 
> long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10449) OOM on bootstrap due to long GC pause

2015-10-05 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-10449:

Fix Version/s: 2.1.x

> OOM on bootstrap due to long GC pause
> -
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Ubuntu 14.04, AWS
>Reporter: Robbie Strickland
>  Labels: gc
> Fix For: 2.1.x
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 
> 500-700GB per node.  SSTable counts are <10 per table.  I am attempting to 
> provision additional nodes, but bootstrapping OOMs every time after about 10 
> hours with a sudden long GC pause:
> {noformat}
> INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old 
> Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 
> CassandraDaemon.java:223 - Exception in thread 
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to 
> no avail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)