Hello,
I have Cassandra 4.0.1 on a server failing to start. The server was
power cycled after it experienced an unrecoverable memory error detected
by EDAC. The memory error was transitory, and AFAIK it has disappeared.
But Cassandra is not starting. The logs are:
INFO [main] 2022-08-06 18:11:53,494 ColumnFamilyStore.java:2242 -
Truncating system.size_estimates
DEBUG [MemtablePostFlush:1] 2022-08-06 18:11:53,495
ColumnFamilyStore.java:933 - forceFlush requested but everything is
clean in size_estimates
INFO [main] 2022-08-06 18:11:53,496 ColumnFamilyStore.java:2279 -
Truncating system.size_estimates with truncatedAt=1659805913495
DEBUG [main] 2022-08-06 18:11:53,496
CompactionStrategyManager.java:519 - Recreating compaction strategy
- disk boundaries are out of date for system.size_estimates.
DEBUG [main] 2022-08-06 18:11:53,497 DiskBoundaryManager.java:55 -
Refreshing disk boundary cache for system.size_estimates
DEBUG [main] 2022-08-06 18:11:53,497 DiskBoundaryManager.java:94 -
Got local ranges
[Full(/**.**.**.**:7000,(-9223372036854775808,-9223372036854775808])]
(ringVersion = 428)
DEBUG [main] 2022-08-06 18:11:53,498 DiskBoundaryManager.java:58 -
Updating boundaries from
DiskBoundaries{directories=[DataDirectory{location=/var/lib/cassandra/data}],
positions=null, ringVersion=1, directoriesVersion=0} to DiskBoundarie
s{directories=[DataDirectory{location=/var/lib/cassandra/data}],
positions=[max(9223372036854775807)], ringVersion=428,
directoriesVersion=0} for system.size_estimates
DEBUG [MemtablePostFlush:1] 2022-08-06 18:11:53,498
ColumnFamilyStore.java:933 - forceFlush requested but everything is
clean in size_estimates
ERROR [main] 2022-08-06 18:11:53,502 CassandraDaemon.java:909 -
Exception encountered during startup
org.apache.cassandra.exceptions.TruncateException: Error during
truncate: java.lang.IllegalArgumentException: Requested permits (0)
must be positive
at
org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:96)
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:323)
at
org.apache.cassandra.db.SystemKeyspace.clearAllEstimates(SystemKeyspace.java:1360)
at
org.apache.cassandra.service.StorageService.cleanupSizeEstimates(StorageService.java:4002)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:373)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
Caused by: java.lang.RuntimeException:
java.lang.IllegalArgumentException: Requested permits (0) must be
positive
at
org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2378)
at
org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2325)
at
org.apache.cassandra.db.ColumnFamilyStore.truncateBlocking(ColumnFamilyStore.java:2302)
at
org.apache.cassandra.cql3.statements.TruncateStatement.executeLocally(TruncateStatement.java:92)
... 6 common frames omitted
Caused by: java.lang.IllegalArgumentException: Requested permits (0)
must be positive
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:189)
at
com.google.common.util.concurrent.RateLimiter.checkPermits(RateLimiter.java:430)
at
com.google.common.util.concurrent.RateLimiter.reserve(RateLimiter.java:285)
at
com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:273)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1849)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2029)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2005)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1993)
at
org.apache.cassandra.db.ColumnFamilyStore$2.run(ColumnFamilyStore.java:2288)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2374)
... 9 common frames omitted
I've tried to restart Cassandra service multiple times on the server
without any success.
I searched the Internet and found this on StackOverflow:
https://stackoverflow.com/questions/66795461/cassandra-4-0-beta4-exception-encountered-during-startup-requested-permits-0-m
* The 1st answer, by Erick Ramirez, suggested removing the
"snapshot" directories from size_estimates and table_estimates
tables in the system keyspace. I have tried this and it did not
fix my issue.
* The 2nd answer, by 贾博超, suggested removing saved_caches AND the
entire system keyspace's data directory, which sounds wrong and
I did not try.
* The 3rd answer, by Aaron, suggested removing the system_schema
keyspace's data directory, which also sounds wrong and I did not
try.
* The last answer, Shane Gannon, is docker specific, which doesn't
apply in my case.
Does anyone know how do I fix this and start this node? Is this a known
bug for Cassandra 4.0?
Best regards,
Bowen
