Brad Schoening created CASSANDRA-18762: ------------------------------------------
Summary: Repair triggers OOM with direct buffer memory Key: CASSANDRA-18762 URL: https://issues.apache.org/jira/browse/CASSANDRA-18762 Project: Cassandra Issue Type: Bug Reporter: Brad Schoening Attachments: Cluster-dm-metrics-1.PNG We are seeing repeated failures of nodes with 16GB of heap and the same size (16GB) for direct memory derived from -Xms. This seems to be related to [CASSANDRA-15202| https://issues.apache.org/jira/browse/CASSANDRA-15202] which moved merkel trees off-heap in 4.0. {noformat} 2023-08-09 04:30:57,470 [INFO ] [AntiEntropyStage:1] cluster_id=101 ip_address=169.0.0.1 RepairSession.java:202 - [repair #5e55a3b0-366d-11ee-a644-d91df26add5e] Received merkle tree for table_a from /169.102.200.241:7000 2023-08-09 04:30:57,567 [INFO ] [AntiEntropyStage:1] cluster_id=101 ip_address=169.0.0.1 RepairSession.java:202 - [repair #5e0d2900-366d-11ee-a644-d91df26add5e] Received merkle tree for table_b from /169.93.192.29:7000 2023-08-09 04:30:57,568 [INFO ] [AntiEntropyStage:1] cluster_id=101 ip_address=169.0.0.1 RepairSession.java:202 - [repair #5e1dcad0-366d-11ee-a644-d91df26add5e] Received merkle tree for table_c from /169.104.171.134:7000 2023-08-09 04:30:57,591 [INFO ] [AntiEntropyStage:1] cluster_id=101 ip_address=169.0.0.1 RepairSession.java:202 - [repair #5e69a0e0-366d-11ee-a644-d91df26add5e] Received merkle tree for table_b from /169.79.232.67:7000 2023-08-09 04:30:57,876 [INFO ] [Service Thread] cluster_id=101 ip_address=169.0.0.1 GCInspector.java:294 - G1 Old Generation GC in 282ms. Compressed Class Space: 8444560 -> 8372152; G1 Eden Space: 7809794048 -> 0; G1 Old Gen: 1453478400 -> 820942800; G1 Survivor Space: 419430400 -> 0; Metaspace: 80411136 -> 80176528 2023-08-09 04:30:58,387 [ERROR] [AntiEntropyStage:1] cluster_id=101 ip_address=169.0.0.1 JVMStabilityInspector.java:102 - OutOfMemory error letting the JVM handle the error: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742) at org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780) at org.apache.cassandra.utils.MerkleTree.deserializeTree(MerkleTree.java:751) at org.apache.cassandra.utils.MerkleTree.deserialize(MerkleTree.java:720) at org.apache.cassandra.utils.MerkleTree.deserialize(MerkleTree.java:698) at org.apache.cassandra.utils.MerkleTrees$MerkleTreesSerializer.deserialize(MerkleTrees.java:416) at org.apache.cassandra.repair.messages.ValidationResponse$1.deserialize(ValidationResponse.java:100) at org.apache.cassandra.repair.messages.ValidationResponse$1.deserialize(ValidationResponse.java:84) at org.apache.cassandra.net.Message$Serializer.deserializePost40(Message.java:782) at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:642) at org.apache.cassandra.net.InboundMessageHandler$LargeMessage.deserialize(InboundMessageHandler.java:364) at org.apache.cassandra.net.InboundMessageHandler$LargeMessage.access$1100(InboundMessageHandler.java:317) at org.apache.cassandra.net.InboundMessageHandler$ProcessLargeMessage.provideMessage(InboundMessageHandler.java:504) at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:429) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834)no* further _formatting_ is done here{noformat} -XX:+AlwaysPreTouch -XX:+CrashOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -XX:+ResizeTLAB -XX:+UseG1GC -XX:+UseNUMA -XX:+UseTLAB -XX:+UseThreadPriorities -XX:-UseBiasedLocking -XX:CompileCommandFile=/opt/nosql/clusters/cassandra-101/conf/hotspot_compiler -XX:G1RSetUpdatingPauseTimePercent=5 -XX:G1ReservePercent=20 -XX:HeapDumpPath=/opt/nosql/data/cluster_101/cassandra-1691623098-pid2804737.hprof -XX:InitiatingHeapOccupancyPercent=70 -XX:MaxGCPauseMillis=200 -XX:StringTableSize=60013 -Xlog:gc*:file=/opt/nosql/clusters/cassandra-101/logs/gc.log:time,uptime:filecount=10,filesize=10485760 -Xms16G -Xmx16G -Xss256k >From our Prometheus metrics, the behavior shows the direct buffer memory >ramping up until it reaches the max and then causes an OOM. It would appear >that direct memory is never being released by the JVM until its exhausted. !Cluster-dm-metrics.PNG! -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org