[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245803#comment-15245803
]
Benjamin Lerer commented on CASSANDRA-11595:
--------------------------------------------
Let me try to re-explain the problem.
When C* is creating a new commit log segment file, it first create a file then
resize it before memory mapping it. The resize of the file is done with an
extra 'connection' to the file.
What happen in your case is that your system hit a 'too many open files' when
C* requested a connection to the commit log segment file to resize it. At that
point the file was created but empty.
C* caught the exception and decided to shut itself down.
When C* was restarted it replayed the commit log files until it reached that
empty file and crashed again.
> Cassandra cannot start because of empty commitlog
> -------------------------------------------------
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
> Reporter: n0rad
> Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
