[ https://issues.apache.org/jira/browse/CASSANDRA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864029#comment-16864029 ]
Chakravarthi Manepalli commented on CASSANDRA-12005: ---------------------------------------------------- I have faced a similar issue, heap overflows after 15-30 min time from the node restart. Setup environment - single node running in localhost with 64GB RAM and 8GB HEAP memory nodetool info: ID : 35c11cc6-f881-4685-b211-654c4a21bf59 Gossip active : true Thrift active : false Native Transport active: true Load : 4.57 MiB Generation No : 1560507197 Uptime (seconds) : 250 Heap Memory (MB) : 301.67 / 8112.00 Off Heap Memory (MB) : 0.01 Data Center : datacenter1 Rack : rack1 Exceptions : 0 Key Cache : entries 128, size 16.5 KiB, capacity 100 MiB, 375 hits, 510 requests, 0.735 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache : entries 19, size 1.19 MiB, capacity 480 MiB, 495 misses, 1126 requests, 0.560 recent hit rate, 48.843 microseconds miss latency Percent Repaired : 100.0% Token : (invoke with -T/--tokens to see all 256 tokens) The node shows running in service status but unable to take cqlsh or access nodetool. ERROR [Native-Transport-Requests-1] 2019-06-14 14:41:35,747 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error: java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.db.Clustering.make(Clustering.java:81) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.3.jar:3.11.3] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.3.jar:3.11.3] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] WARN [epollEventLoopGroup-2-5] 2019-06-14 14:41:35,748 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.db.Clustering.make(Clustering.java:81) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) ~[apache-cassandra-3.11.3.jar:3.11.3] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.3.jar:3.11.3] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) ~[apache-cassandra-3.11.3.jar:3.11.3] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] > Out of memory error in MessagingService > --------------------------------------- > > Key: CASSANDRA-12005 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12005 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Streaming and Messaging > Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP > x86_64 > Cassandra ReleaseVersion: 2.2.5 > java version "1.8.0_31" > Java(TM) SE Runtime Environment (build 1.8.0_31-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode) > Reporter: Chris Powell > Priority: Normal > > I am periodically loosing nodes due to the below OOM error. The nodes restart > perfectly fine. It appears intermittent and randomly affects nodes. There are > no other warnings or errors in the log files. > I am using the {{GCG1}} with the following options: > {quote} > JVM_OPTS="$JVM_OPTS -XX:+UseG1GC" > JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" > JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" > JVM_OPTS="$JVM_OPTS -XX:+UseTLAB" > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch" > JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking" > JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB" > JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500" > JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10" > JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25" > {quote} > ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237 > CassandraDaemon.java:185 - Exception in thread > Thread[MessagingService-Incoming-/10.184.11.109,5,main] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) > ~[apache-cassandra-2.2.5.jar:2.2.5] > ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239 > JVMStabilityInspector.java:117 - JVM state determined to be unstable. > Exiting forcefully due to: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91) > ~[apache-cassandra-2.2.5.jar:2.2.5] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org