[jira] [Comment Edited] (CASSANDRA-10593) Unintended interactions between commitlog archiving and commitlog recycling
[
https://issues.apache.org/jira/browse/CASSANDRA-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064976#comment-15064976
]
Aleksey Yeschenko edited comment on CASSANDRA-10593 at 12/18/15 11:41 PM:
--
Committed as
[3740f815c21254bd625ad1cbe8d47aa657727a83|https://github.com/apache/cassandra/commit/3740f815c21254bd625ad1cbe8d47aa657727a83]
to 2.1 and merged with 2.2, 3.0, and trunk. Thanks.
was (Author: iamaleksey):
Committed to
[3740f815c21254bd625ad1cbe8d47aa657727a83|https://github.com/apache/cassandra/commit/3740f815c21254bd625ad1cbe8d47aa657727a83]
and merged with 2.2, 3.0, and trunk. Thanks.
> Unintended interactions between commitlog archiving and commitlog recycling
> ---
>
> Key: CASSANDRA-10593
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10593
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: J.B. Langston
>Assignee: Ariel Weisberg
> Fix For: 2.1.13, 2.2.5, 3.2, 3.0.3
>
> Attachments: cassandra.yaml, commitlog_archiving.properties,
> system.log
>
>
> Currently the comments in commitlog_archiving.properties suggest using either
> cp or ln for the archive_command.
> Using ln is problematic because commitlog recycling marks segments as
> recycled once the corresponding memtables are flushed and Cassandra will no
> longer replay them. This means it's only possible to do PITR on any records
> that were written since the last flush.
> Using cp works, and this is currently how OpsCenter does for PITR, however
> [~brandon.williams] has pointed out this could have some performance impact
> because of the additional I/O overhead of copying the commitlog segments.
> Starting in 2.1, we can disable commit log recycling in cassandra.yaml so I
> thought this would allow me to do PITR without the extra overhead of using
> cp. However, when I disable commitlog recycling and try to do a PITR,
> Cassandra blows up when trying to replay the restored commit logs:
> {code}
> ERROR 16:56:42 Exception encountered during startup
> java.lang.IllegalStateException: Cannot safely construct descriptor for
> segment, as name and header descriptors do not match ((4,1445878452545) vs
> (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335)
> ~[dse-core-4.8.0.jar:4.8.0]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at com.datastax.bdp.DseModule.main(DseModule.java:75)
> [dse-core-4.8.0.jar:4.8.0]
> java.lang.IllegalStateException: Cannot safely construct descriptor for
> segment, as name and header descriptors do not match ((4,1445878452545) vs
> (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352)
> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537)
> at com.datastax.bdp.DseModule.main(DseModule.java:75)
> Exception encountered during startup: Cannot safely construct descriptor for
> segment, as name and header descriptors do not match ((4,1445878452545) vs
> (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log
> INFO 16:56:42 DSE shutting down...
> INFO 16:56:42 All plugins are stopped.
> ERROR 16:56:42 Exception in thread Thread[Thread-2,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1403)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:196)
> ~[dse-core-4.8.0.jar:4.8.0]
> at com.datastax.bdp.server.DseDaemon.preStop(DseDaemon.java:426)
> ~[dse-core-4.8.0.jar:4.8.0]
> at com.datastax.bdp.server.DseDaemon.safeStop(DseDaemon.java:436)
> ~[dse-core-4.8.0.jar:4.8.0]
> at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:676)
> ~[dse-core-4.8.0.jar:4.8.0]
> at
[jira] [Comment Edited] (CASSANDRA-10593) Unintended interactions between commitlog archiving and commitlog recycling
[
https://issues.apache.org/jira/browse/CASSANDRA-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038770#comment-15038770
]
Ariel Weisberg edited comment on CASSANDRA-10593 at 12/7/15 8:55 PM:
-
Proposed fix including a new dtest. I still need to evaluate what happens in
2.2+ to see how much of this if any is going to be necessary in later versions.
|[2.1
code|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-10593-2.1?expand=1]|[dtest
code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-dtest/]|
|[2.2
code|https://github.com/apache/cassandra/compare/cassandra-2.2...aweisberg:CASSANDRA-10593-2.2?expand=1]|[dtest
code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.2-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.2-dtest/]|
was (Author: aweisberg):
Proposed fix including a new dtest. I still need to evaluate what happens in
2.2+ to see how much of this if any is going to be necessary in later versions.
|[2.1
code|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-10593-2.1?expand=1]|[dtest
code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-dtest/]|
> Unintended interactions between commitlog archiving and commitlog recycling
> ---
>
> Key: CASSANDRA-10593
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10593
> Project: Cassandra
> Issue Type: Bug
>Reporter: J.B. Langston
>Assignee: Ariel Weisberg
> Attachments: cassandra.yaml, commitlog_archiving.properties,
> system.log
>
>
> Currently the comments in commitlog_archiving.properties suggest using either
> cp or ln for the archive_command.
> Using ln is problematic because commitlog recycling marks segments as
> recycled once the corresponding memtables are flushed and Cassandra will no
> longer replay them. This means it's only possible to do PITR on any records
> that were written since the last flush.
> Using cp works, and this is currently how OpsCenter does for PITR, however
> [~brandon.williams] has pointed out this could have some performance impact
> because of the additional I/O overhead of copying the commitlog segments.
> Starting in 2.1, we can disable commit log recycling in cassandra.yaml so I
> thought this would allow me to do PITR without the extra overhead of using
> cp. However, when I disable commitlog recycling and try to do a PITR,
> Cassandra blows up when trying to replay the restored commit logs:
> {code}
> ERROR 16:56:42 Exception encountered during startup
> java.lang.IllegalStateException: Cannot safely construct descriptor for
> segment, as name and header descriptors do not match ((4,1445878452545) vs
> (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335)
> ~[dse-core-4.8.0.jar:4.8.0]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537)
> ~[cassandra-all-2.1.9.791.jar:2.1.9.791]
> at com.datastax.bdp.DseModule.main(DseModule.java:75)
> [dse-core-4.8.0.jar:4.8.0]
> java.lang.IllegalStateException: Cannot safely construct descriptor for
> segment, as name and header descriptors do not match ((4,1445878452545) vs
> (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log
> at
> org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352)
> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537)
> at
