[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245584#comment-15245584
]
n0rad commented on CASSANDRA-11595:
-----------------------------------
it's mixed up with gc but here is the final logs of the process. After that
cassandra restart
{code}
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:163)
~[apache-cassandra-3.0.5.jar:3.0.5]
Metaspace used 40691K, capacity 42565K, committed 43164K, reserved
1087488K
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.<init>(MemoryMappedSegment.java:47)
~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-3.0.5.jar:3.0.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77-Debian]
class space used 4325K, capacity 4685K, committed 4856K, reserved 1048576K
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:163)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.<init>(MemoryMappedSegment.java:47)
ERROR 07:06:06 JVM state determined to be unstable. Exiting forcefully due to:
Heap
at
org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log
(Too many open files)
par new generation total 1887488K, used 711037K [0x00000005c0000000,
0x0000000640000000, 0x0000000640000000)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_77-Debian]
eden space 1677824K, 40% used [0x00000005c0000000, 0x00000005e9cd77d0,
0x0000000626680000)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
~[na:1.8.0_77-Debian]
from space 209664K, 12% used [0x0000000633340000, 0x0000000634cc7ff8,
0x0000000640000000)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log
(Too many open files)
at java.lang.Thread.run(Thread.java:745)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
~[na:1.8.0_77-Debian]
to space 209664K, 0% used [0x0000000626680000, 0x0000000626680000,
0x0000000633340000)
at java.io.RandomAccessFile.open0(Native Method)
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
~[apache-cassandra-3.0.5.jar:3.0.5]
concurrent mark-sweep generation total 6291456K, used 4072089K
[0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000)
{code}
> Cassandra cannot start because of empty commitlog
> -------------------------------------------------
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
> Reporter: n0rad
> Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
