[
https://issues.apache.org/jira/browse/CASSANDRA-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969215#comment-14969215
]
Robbie Strickland commented on CASSANDRA-10449:
-----------------------------------------------
Also, for reference, tpstats shows nothing in the queues:
{noformat}
ubuntu@eventcass4x087:~$ nodetool tpstats
Pool Name Active Pending Completed Blocked All
time blocked
MutationStage 0 0 85431226 0
0
ReadStage 0 0 0 0
0
RequestResponseStage 0 0 48 0
0
ReadRepairStage 0 0 0 0
0
CounterMutationStage 0 0 0 0
0
MiscStage 0 0 0 0
0
HintedHandoff 0 0 29 0
0
GossipStage 0 0 565556 0
0
CacheCleanupExecutor 0 0 0 0
0
InternalResponseStage 0 0 0 0
0
CommitLogArchiver 0 0 0 0
0
CompactionExecutor 0 0 12774 0
0
ValidationExecutor 0 0 0 0
0
MigrationStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
PendingRangeCalculator 0 0 3 0
0
Sampler 0 0 0 0
0
MemtableFlushWriter 0 0 7157 0
0
MemtablePostFlush 0 0 10083 0
0
MemtableReclaimMemory 0 0 9340 0
0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
MUTATION 0
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
{noformat}
> OOM on bootstrap after long GC pause
> ------------------------------------
>
> Key: CASSANDRA-10449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10449
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, AWS
> Reporter: Robbie Strickland
> Labels: gc
> Fix For: 2.1.x
>
> Attachments: GCpath.txt, heap_dump.png, system.log.10-05,
> thread_dump.log, threads.txt
>
>
> I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and
> 500-700GB per node. SSTable counts are <10 per table. I am attempting to
> provision additional nodes, but bootstrapping OOMs every time after about 10
> hours with a sudden long GC pause:
> {noformat}
> INFO [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old
> Generation GC in 1586126ms. G1 Old Gen: 49213756976 -> 49072277176;
> ...
> ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380
> CassandraDaemon.java:223 - Exception in thread
> Thread[MemtableFlushWriter:454,5,main]
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> I have tried increasing max heap to 48G just to get through the bootstrap, to
> no avail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
