[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416539#comment-15416539 ] Andy Tolbert commented on CASSANDRA-11595: -- Was talking with a user who also encountered this same problem (also on windows), however in their case their Commitlog was 32MB, but was completely zero'd out. Their system had been power cycled and upon restart they hit this error. I was looking at the code and saw that when reading commitlogs prior to 2.1 a [different code path|https://github.com/apache/cassandra/blob/cassandra-3.7/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L358] would be taken for reading commitlogs which seemed to [gracefully proceed|https://github.com/apache/cassandra/blob/cassandra-3.7/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L464-L468] when reading 0. Although in that case no header would be read from like it is for [newer commitlogs|https://github.com/apache/cassandra/blob/cassandra-3.7/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L370]. Should it be considered mandatory that the header content is present in the commitlog as it currently is? In this case it seems like there is no risk in the user simply deleting the file and moving on since it doesn't contain anything anyways. Although maybe there is still a problem somewhere since the file has no header in the first place. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416712#comment-15416712 ] n0rad commented on CASSANDRA-11595: --- I forgot to tell that to be able to restart, I moved commitlogs away. After an upgrade to 3.0.6, I stopped failed node and add back thos commitlog to replay them. It worked. For me the problem was solved in 3.0.6. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420902#comment-15420902 ] Joshua McKenzie commented on CASSANDRA-11595: - bq. Should it be considered mandatory that the header content is present in the commitlog as it currently is? In this case it seems like there is no risk in the user simply deleting the file and moving on since it doesn't contain anything anyways. In general, our philosophy is that if there's a problem with your commitlog we die and notify you unless you specifically configure otherwise. Also, the possibility of introducing a bug that leads to data loss with auto-deletion is sufficient for me to advocate against it. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421432#comment-15421432 ] n0rad commented on CASSANDRA-11595: --- I also forgot to tell that the following CASSANDRA-11594, I had multiple crashes. Empty files, but also on 2 servers all files with the same size, but the server was unable to replay them, and not telling which file was wrong. This is why I moved away all commit logs files waiting for a new version of C*. Like I sayed, I was able to merge old commitlogs with live one after bumping to 3.0.6. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421501#comment-15421501
]
Andy Tolbert commented on CASSANDRA-11595:
--
{quote}
In general, our philosophy is that if there's a problem with your commitlog we
die and notify you unless you specifically configure otherwise. Also, the
possibility of introducing a bug that leads to data loss with auto-deletion is
sufficient for me to advocate against it.
{quote}
That seems fair to me. Something went wrong, the user should take a step back
and think about it.
Although in this case it's not particularly user friendly, there's nothing in
the file, so there really is no recovery path for that particular file. The
user may not understand that the file is empty - since it is 32MB after all,
albeit zero'd out - and may not understand the only corrective course is to
remove/move away the commitlog.
Overall, I agree the fact that the file is empty in the first place is the main
problem (although caused by some external factor like powercycling the system)
- not that C* refuses to start because of it.
> Cassandra cannot start because of empty commitlog
> -
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
>Reporter: n0rad
>Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421547#comment-15421547
]
Benjamin Lerer commented on CASSANDRA-11595:
Let's not mix up everything. This ticket has been opened on the 3.0.x release
and the problem was caused by a 'too many open files' problem when C* was
trying to resize the file. The file ended up with a size of zero.
In that case there is unfortunatly nothing that can be done to correct the
problem before shutdown.
I think the safest choice here is to prevent C* from starting and to let the
administrator deal with the problem. As in this case the file is obviously
empty the solution to the problem is pretty clear.
[~andrew.tolbert] you are talking in my opinion of a different problem in the
same area that affect 2.1 where C* has probably crashed after the resize.
I will clause the ticket as {{Not A Problem}} as C* behaved as expected.
> Cassandra cannot start because of empty commitlog
> -
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
>Reporter: n0rad
>Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421556#comment-15421556
]
Andy Tolbert commented on CASSANDRA-11595:
--
{quote}
I think the safest choice here is to prevent C* from starting and to let the
administrator deal with the problem. As in this case the file is obviously
empty the solution to the problem is pretty clear.
Andy Tolbert you are talking in my opinion of a different problem in the same
area that affect 2.1 where C* has probably crashed after the resize.
I will clause the ticket as Not A Problem as C* behaved as expected.
{quote}
(y), agreed.
> Cassandra cannot start because of empty commitlog
> -
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
>Reporter: n0rad
>Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421832#comment-15421832 ] n0rad commented on CASSANDRA-11595: --- I disagree, It's up to you, but the thing that an empty commitlog file prevent cassandra to start is a bug to me. I don't see the point of asking an admin to remove a empty file, while there is nothing else to do. A warn at start looks better to me. A little bit a context, CASSANDRA-11594 makes my cluster nodes crash one after the other during a wend. This bug prevent my nodes to restart on his own and makes my cluster down with a production issue after all replicate were not available. There may be bugs in Cassandra that cause crashes, the thing that it's coded in let it crash is ok and here to handle those situations. The fact that this issue prevent the node to restart and ask an admin to connect to the server and remove an empty file to be able to restart, and there is nothing else he can do, looks not ok. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422350#comment-15422350 ] Benjamin Lerer commented on CASSANDRA-11595: [~n0rad] there might be other scenario that result in an empty CommitLog files and not all of them need to be handle in the same way. The problem is that C* does not know what has happend before or what the administrator is trying to do. In this case it always safer to let the administrator, who know the full context to handle the problem. Your real problem is CASSANDRA-11594. Without it you will rarely bump into an empty CommitLog segment. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245520#comment-15245520 ] n0rad commented on CASSANDRA-11595: --- I think the 2 problems are linked. I have found : `java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log (Too many open files)` during the too many open files issue, before the restart and `org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file /data/commitlog/CommitLog-6-1460633590316.log` after the restart > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245538#comment-15245538 ] Benjamin Lerer commented on CASSANDRA-11595: Could you provide me the full stacktrace for the `java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log (Too many open files)` error? > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245584#comment-15245584
]
n0rad commented on CASSANDRA-11595:
---
it's mixed up with gc but here is the final logs of the process. After that
cassandra restart
{code}
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:163)
~[apache-cassandra-3.0.5.jar:3.0.5]
Metaspace used 40691K, capacity 42565K, committed 43164K, reserved
1087488K
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.(MemoryMappedSegment.java:47)
~[apache-cassandra-3.0.5.jar:3.0.5]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[apache-cassandra-3.0.5.jar:3.0.5]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77-Debian]
class spaceused 4325K, capacity 4685K, committed 4856K, reserved 1048576K
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:163)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
~[apache-cassandra-3.0.5.jar:3.0.5]
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.(MemoryMappedSegment.java:47)
ERROR 07:06:06 JVM state determined to be unstable. Exiting forcefully due to:
Heap
at
org.apache.cassandra.db.commitlog.CommitLogSegment.createSegment(CommitLogSegment.java:122)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log
(Too many open files)
par new generation total 1887488K, used 711037K [0x0005c000,
0x00064000, 0x00064000)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:122)
at java.io.RandomAccessFile.open0(Native Method) ~[na:1.8.0_77-Debian]
eden space 1677824K, 40% used [0x0005c000, 0x0005e9cd77d0,
0x00062668)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
~[na:1.8.0_77-Debian]
from space 209664K, 12% used [0x00063334, 0x000634cc7ff8,
0x00064000)
java.io.FileNotFoundException: /data/commitlog/CommitLog-6-1460633590316.log
(Too many open files)
at java.lang.Thread.run(Thread.java:745)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
~[na:1.8.0_77-Debian]
to space 209664K, 0% used [0x00062668, 0x00062668,
0x00063334)
at java.io.RandomAccessFile.open0(Native Method)
at
org.apache.cassandra.db.commitlog.MemoryMappedSegment.createBuffer(MemoryMappedSegment.java:62)
~[apache-cassandra-3.0.5.jar:3.0.5]
concurrent mark-sweep generation total 6291456K, used 4072089K
[0x00064000, 0x0007c000, 0x0007c000)
{code}
> Cassandra cannot start because of empty commitlog
> -
>
> Key: CASSANDRA-11595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11595
> Project: Cassandra
> Issue Type: Bug
>Reporter: n0rad
>Assignee: Benjamin Lerer
>
> After the crash of CASSANDRA-11594.
> Cassandra try to restart and fail because of commit log replay.
> Same on 4 of the crashed nodes out of 6.
> ```
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Could not read commit log descriptor in file
> /data/commitlog/CommitLog-6-1460632496764.log
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644)
> [apache-cassandra-3.0.5.jar:3.0.5]
> ```
> This file is empty and is not the commitlog with the latest date.
> ```
> ...
> -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log
> -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log
> -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log
> -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log
> -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log
> -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log
> -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log
> -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log
> -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log
> -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log
> -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log
> ...
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245600#comment-15245600 ] Benjamin Lerer commented on CASSANDRA-11595: Thanks > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245711#comment-15245711 ] Benjamin Lerer commented on CASSANDRA-11595: When a new segment is allocated the file is first created and open. The file is then open a second time, due to a bug on Windows (CASSANDRA-8308), to be resize. It seems that the limit of open files was reached when the file was open that second time. Due to that nothing was written to the file. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245751#comment-15245751 ] n0rad commented on CASSANDRA-11595: --- It happens on 4 out of 6 nodes of the cluster. the 2 others are : - 1 ok node was restarted during the wend so never reach the too many open files - 1 was still on the too many open files and did not rebooted. This one do not have empty files Is there any commitlog process while jvm is detected as in bad state that could cause the creation of the empty file while already on too many open files ? > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245803#comment-15245803 ] Benjamin Lerer commented on CASSANDRA-11595: Let me try to re-explain the problem. When C* is creating a new commit log segment file, it first create a file then resize it before memory mapping it. The resize of the file is done with an extra 'connection' to the file. What happen in your case is that your system hit a 'too many open files' when C* requested a connection to the commit log segment file to resize it. At that point the file was created but empty. C* caught the exception and decided to shut itself down. When C* was restarted it replayed the commit log files until it reached that empty file and crashed again. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11595) Cassandra cannot start because of empty commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248361#comment-15248361 ] DOAN DuyHai commented on CASSANDRA-11595: - Ok so it's like a vicious circle. But then all those things are just *consequences* or another problem, which is *too many opened files*. Sure we can fix this side effect but we need to find the root cause of why there are too many opened files. > Cassandra cannot start because of empty commitlog > - > > Key: CASSANDRA-11595 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11595 > Project: Cassandra > Issue Type: Bug >Reporter: n0rad >Assignee: Benjamin Lerer > > After the crash of CASSANDRA-11594. > Cassandra try to restart and fail because of commit log replay. > Same on 4 of the crashed nodes out of 6. > ``` > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file > /data/commitlog/CommitLog-6-1460632496764.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:644) > [apache-cassandra-3.0.5.jar:3.0.5] > ``` > This file is empty and is not the commitlog with the latest date. > ``` > ... > -rw-r--r-- 1 root root 32M Apr 16 21:46 CommitLog-6-1460632496761.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496762.log > -rw-r--r-- 1 root root 32M Apr 16 21:47 CommitLog-6-1460632496763.log > -rw-r--r-- 1 root root 0 Apr 16 21:47 CommitLog-6-1460632496764.log > -rw-r--r-- 1 root root 32M Apr 16 21:50 CommitLog-6-1460843401097.log > -rw-r--r-- 1 root root 32M Apr 16 21:51 CommitLog-6-1460843513346.log > -rw-r--r-- 1 root root 32M Apr 16 21:53 CommitLog-6-1460843619271.log > -rw-r--r-- 1 root root 32M Apr 16 21:55 CommitLog-6-1460843730533.log > -rw-r--r-- 1 root root 32M Apr 16 21:57 CommitLog-6-1460843834129.log > -rw-r--r-- 1 root root 32M Apr 16 21:58 CommitLog-6-1460843935094.log > -rw-r--r-- 1 root root 32M Apr 16 22:00 CommitLog-6-1460844038543.log > -rw-r--r-- 1 root root 32M Apr 16 22:02 CommitLog-6-1460844141003.log > ... > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
