Purushottam baghel created CASSANDRA-18556:
----------------------------------------------
Summary: Leak detected issue in cassandra 3.11.1
Key: CASSANDRA-18556
URL: https://issues.apache.org/jira/browse/CASSANDRA-18556
Project: Cassandra
Issue Type: Bug
Reporter: Purushottam baghel
Attachments: Screenshot 2023-05-30 at 10.50.32 AM.png
We are running 36 nodes Cassandra cluster. All nodes have the some config,
Cassandra version is 3.11.1. Suddenly, we are seeing fluctuation in storage
available graphs for the 2 nodes. Attaching the last 7 days used storage graph
for the Cassandra nodes.
Further investigation, we saw errors in the logs of these 2 Cassandra node
logs.
{code:java}
ERROR [CompactionExecutor:164360] 2023-05-04 00:30:58,094
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:164360,1,main]
java.lang.IllegalArgumentException: null
at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_131]
at
org.apache.cassandra.io.util.SafeMemoryWriter.reallocate(SafeMemoryWriter.java:59)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.util.SafeMemoryWriter.setCapacity(SafeMemoryWriter.java:68)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.prepareToCommit(IndexSummaryBuilder.java:250)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:524)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:379)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
WARN [CompactionExecutor:164909] 2023-05-04 00:31:54,573
IndexSummaryBuilder.java:115 - min_index_interval of 128 is too low for
4298508892 expected keys of avg size 64; using interval of 145 instead
ERROR [Reference-Reaper:1] 2023-05-04 00:32:02,326 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@4f374614) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1106289026:Memory@[7fa6ab9ec010..7fa72ad798d0)
was not released before the reference was garbage collected {code}
Compactions also running continually run on these nodes.
I am unable to understand the root cause of it. Any help is appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
