Tomer Yuval created CASSANDRA-15084:
---------------------------------------
Summary: failed compactions in specific cluster version 3.11.2
Key: CASSANDRA-15084
URL: https://issues.apache.org/jira/browse/CASSANDRA-15084
Project: Cassandra
Issue Type: Bug
Reporter: Tomer Yuval
Fix For: 3.11.x
We are using Cassandra version 3.11.2
In our biggest cluster, in the biggest table compactions keep failing.
The amount of data in every node is around 900GB.
After we run the compaction, we monitor the process using nodetool
compactionstats and can see the progress continue until it reached 99.99% of
the progress.
In this point the compaction fail with the error below
In addition, we increased index_summary_capacity_in_mb few times until the size
of 4GB, but with no help.
ERROR [CompactionExecutor:26976] 2019-04-11 13:55:24,917
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:26976,1,main]
java.lang.IllegalArgumentException: null
at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_144]
at
org.apache.cassandra.io.util.SafeMemoryWriter.reallocate(SafeMemoryWriter.java:59)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.io.util.SafeMemoryWriter.setCapacity(SafeMemoryWriter.java:68)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.prepareToCommit(IndexSummaryBuilder.java:250)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:524)
~[apache-cassandra-3.11.2.jar:3.11.
2]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.2.
jar:3.11.2]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364)
~[apache-cassandra-3.11.2.ja
r:3.11.2]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.2.
jar:3.11.2]
at
org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:379)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.2.
jar:3.11.2]
at
[org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:112|http://org.apache.cassandra.db.compaction.writers.compactionawarewriter.doprepare%28compactionawarewriter.java:112/])
~[apache-cassandra-3.11.2.jar:3.1
1.2]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
~[apache-cassandra-3.11.2.
jar:3.11.2]
at
org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
~[apache-cassandra-3.11.2.jar:3.11.
2]
at
[org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:122|http://org.apache.cassandra.db.compaction.writers.compactionawarewriter.finish%28compactionawarewriter.java:122/])
~[apache-cassandra-3.11.2.jar:3.11.2
]
at
[org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220|http://org.apache.cassandra.db.compaction.compactiontask.runmaythrow%28compactiontask.java:220/])
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
[org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85|http://org.apache.cassandra.db.compaction.compactiontask.executeinternal%28compactiontask.java:85/])
~[apache-cassandra-3.11.2.jar:3.11.2]
at
[org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61|http://org.apache.cassandra.db.compaction.abstractcompactiontask.execute%28abstractcompactiontask.java:61/])
~[apache-cassandra-3.11.2.jar:3.11.2]
at
[org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:746|http://org.apache.cassandra.db.compaction.compactionmanager%2410.runmaythrow%28compactionmanager.java:746/])
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[na:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_144]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
