[ https://issues.apache.org/jira/browse/CASSANDRA-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184081#comment-17184081 ]
Brandon Williams commented on CASSANDRA-16060: ---------------------------------------------- That sounds like a plausible cause, though 150k isn't that bad. Can you query less rows at a time or something else to mitigate this? > Cassandra crashes with OutOfMemory Exception > -------------------------------------------- > > Key: CASSANDRA-16060 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16060 > Project: Cassandra > Issue Type: Bug > Reporter: Thomas De Keulenaer > Priority: Normal > Attachments: cassandra.yaml > > > Out Cassandra instance has run perfectly for almost 5 years, but aout a month > ago, the performance has dropped significantly. Reads are incredibly slow or > time out, making the cluster almost useless. The load on the nodes has > skyrocketed to almost 100% CPU usage. On top of that, the nodes crash with > OutOfMemoryError. > I also have [heap > dumps|https://arcelormittal-my.sharepoint.com/:f:/g/personal/sidtdke_sidmar_be/Es7PaQq15jdKgIvDfMFfBRoBR6sr60t156AxbA0dFZvzJg?e=rohgHn]. > > Cassandra version: 3.11.1 > Java version: OpenJDK 1.8.0_201 > We have upgraded our nodes to version 3.11.7 and applied the recommended > OS/system setting, but this has not improved performance. > {code:java} > ERROR [ReadStage-9] 2020-08-10 14:56:01,399 JVMStabilityInspector.java:142 - > JVM state determined to be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) > ~[na:1.8.0_201] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_201] > at > org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:159) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:413) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:210) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$serializeRowBody$0(UnfilteredSerializer.java:248) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer$$Lambda$120/205506350.accept(Unknown > Source) ~[na:na] > at > org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at org.apache.cassandra.db.rows.BTreeRow.apply(BTreeRow.java:172) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:236) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:205) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:137) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:125) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:137) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:92) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:308) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:167) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:160) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:156) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:346) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_201] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) > ~[apache-cassandra-3.11.1.jar:3.11.1] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org