[
https://issues.apache.org/jira/browse/CASSANDRA-19831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brad Schoening updated CASSANDRA-19831:
---------------------------------------
Description:
When we see Direct buffer memory errors, the JVMStabilityInspector appears to
suggest it's a heap OOM to casual users. This can confuse users who think that
increasing the heap may help solve the problem. A vendors support staff
suggested increasing the heap in this scenario.
A more clear error message could be something like:
Off-heap OOM forcing heap space OutOfMemoryError after OutOfMemoryError: Direct
buffer memory
{quote}2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6] cluster_id=001
ip_address=172.0.0.1 JVMStabilityInspector.java:151 - Force heap space
OutOfMemoryError in the presence of
java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
at
org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)
{quote}
The comment in the code is clear, but
{quote} /**
* Intentionally produce a heap space OOM upon seeing a non heap memory OOM.
* Direct buffer OOM cannot trigger JVM OOM error related options,
* e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
* See CASSANDRA-15214 and CASSANDRA-17128 for more details
*/
{quote}
but the log message users see begins with 'Force heap space OutOfMemoryError',
which suggests a heap memory issue, not off-heap memory. Very confusing for
general users and junior support staff.
was:
When we see Direct buffer memory errors, the JVMStabilityInspector appears to
suggest it's a heap OOM to casual users. This can confuse users who think that
increasing the heap may help solve the problem. A vendors support staff
suggested increasing the heap in this scenario.
A more clear error message could be something like:
Off-heap OOM will forcing heap space OutOfMemoryError after OutOfMemoryError:
Direct buffer memory
{quote}2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6] cluster_id=001
ip_address=172.0.0.1 JVMStabilityInspector.java:151 - Force heap space
OutOfMemoryError in the presence of
java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
at
org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)
{quote}
The comment in the code is clear, but
{quote} /**
* Intentionally produce a heap space OOM upon seeing a non heap memory OOM.
* Direct buffer OOM cannot trigger JVM OOM error related options,
* e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
* See CASSANDRA-15214 and CASSANDRA-17128 for more details
*/
{quote}
> Improve JVMStabilityInspector error message to clarify off-heap vs heap OOM
> errors
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-19831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19831
> Project: Cassandra
> Issue Type: Task
> Components: Local/Other
> Reporter: Brad Schoening
> Priority: Low
>
> When we see Direct buffer memory errors, the JVMStabilityInspector appears to
> suggest it's a heap OOM to casual users. This can confuse users who think
> that increasing the heap may help solve the problem. A vendors support staff
> suggested increasing the heap in this scenario.
> A more clear error message could be something like:
> Off-heap OOM forcing heap space OutOfMemoryError after OutOfMemoryError:
> Direct buffer memory
> {quote}2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6]
> cluster_id=001 ip_address=172.0.0.1 JVMStabilityInspector.java:151 - Force
> heap space OutOfMemoryError in the presence of
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
> at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
> at
> org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)
> {quote}
> The comment in the code is clear, but
> {quote} /**
> * Intentionally produce a heap space OOM upon seeing a non heap memory
> OOM.
> * Direct buffer OOM cannot trigger JVM OOM error related options,
> * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
> * See CASSANDRA-15214 and CASSANDRA-17128 for more details
> */
> {quote}
> but the log message users see begins with 'Force heap space
> OutOfMemoryError', which suggests a heap memory issue, not off-heap memory.
> Very confusing for general users and junior support staff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
