[jira] [Comment Edited] (CASSANDRA-10593) Unintended interactions between commitlog archiving and commitlog recycling
[ https://issues.apache.org/jira/browse/CASSANDRA-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15064976#comment-15064976 ] Aleksey Yeschenko edited comment on CASSANDRA-10593 at 12/18/15 11:41 PM: -- Committed as [3740f815c21254bd625ad1cbe8d47aa657727a83|https://github.com/apache/cassandra/commit/3740f815c21254bd625ad1cbe8d47aa657727a83] to 2.1 and merged with 2.2, 3.0, and trunk. Thanks. was (Author: iamaleksey): Committed to [3740f815c21254bd625ad1cbe8d47aa657727a83|https://github.com/apache/cassandra/commit/3740f815c21254bd625ad1cbe8d47aa657727a83] and merged with 2.2, 3.0, and trunk. Thanks. > Unintended interactions between commitlog archiving and commitlog recycling > --- > > Key: CASSANDRA-10593 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10593 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: J.B. Langston >Assignee: Ariel Weisberg > Fix For: 2.1.13, 2.2.5, 3.2, 3.0.3 > > Attachments: cassandra.yaml, commitlog_archiving.properties, > system.log > > > Currently the comments in commitlog_archiving.properties suggest using either > cp or ln for the archive_command. > Using ln is problematic because commitlog recycling marks segments as > recycled once the corresponding memtables are flushed and Cassandra will no > longer replay them. This means it's only possible to do PITR on any records > that were written since the last flush. > Using cp works, and this is currently how OpsCenter does for PITR, however > [~brandon.williams] has pointed out this could have some performance impact > because of the additional I/O overhead of copying the commitlog segments. > Starting in 2.1, we can disable commit log recycling in cassandra.yaml so I > thought this would allow me to do PITR without the extra overhead of using > cp. However, when I disable commitlog recycling and try to do a PITR, > Cassandra blows up when trying to replay the restored commit logs: > {code} > ERROR 16:56:42 Exception encountered during startup > java.lang.IllegalStateException: Cannot safely construct descriptor for > segment, as name and header descriptors do not match ((4,1445878452545) vs > (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log > at > org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335) > ~[dse-core-4.8.0.jar:4.8.0] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at com.datastax.bdp.DseModule.main(DseModule.java:75) > [dse-core-4.8.0.jar:4.8.0] > java.lang.IllegalStateException: Cannot safely construct descriptor for > segment, as name and header descriptors do not match ((4,1445878452545) vs > (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log > at > org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352) > at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) > at com.datastax.bdp.DseModule.main(DseModule.java:75) > Exception encountered during startup: Cannot safely construct descriptor for > segment, as name and header descriptors do not match ((4,1445878452545) vs > (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log > INFO 16:56:42 DSE shutting down... > INFO 16:56:42 All plugins are stopped. > ERROR 16:56:42 Exception in thread Thread[Thread-2,5,main] > java.lang.AssertionError: null > at > org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1403) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at com.datastax.bdp.gms.DseState.setActiveStatus(DseState.java:196) > ~[dse-core-4.8.0.jar:4.8.0] > at com.datastax.bdp.server.DseDaemon.preStop(DseDaemon.java:426) > ~[dse-core-4.8.0.jar:4.8.0] > at com.datastax.bdp.server.DseDaemon.safeStop(DseDaemon.java:436) > ~[dse-core-4.8.0.jar:4.8.0] > at com.datastax.bdp.server.DseDaemon$1.run(DseDaemon.java:676) > ~[dse-core-4.8.0.jar:4.8.0] > at
[jira] [Comment Edited] (CASSANDRA-10593) Unintended interactions between commitlog archiving and commitlog recycling
[ https://issues.apache.org/jira/browse/CASSANDRA-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038770#comment-15038770 ] Ariel Weisberg edited comment on CASSANDRA-10593 at 12/7/15 8:55 PM: - Proposed fix including a new dtest. I still need to evaluate what happens in 2.2+ to see how much of this if any is going to be necessary in later versions. |[2.1 code|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-10593-2.1?expand=1]|[dtest code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-dtest/]| |[2.2 code|https://github.com/apache/cassandra/compare/cassandra-2.2...aweisberg:CASSANDRA-10593-2.2?expand=1]|[dtest code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.2-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.2-dtest/]| was (Author: aweisberg): Proposed fix including a new dtest. I still need to evaluate what happens in 2.2+ to see how much of this if any is going to be necessary in later versions. |[2.1 code|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-10593-2.1?expand=1]|[dtest code|https://github.com/riptano/cassandra-dtest/pull/697]|[utests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10593-2.1-dtest/]| > Unintended interactions between commitlog archiving and commitlog recycling > --- > > Key: CASSANDRA-10593 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10593 > Project: Cassandra > Issue Type: Bug >Reporter: J.B. Langston >Assignee: Ariel Weisberg > Attachments: cassandra.yaml, commitlog_archiving.properties, > system.log > > > Currently the comments in commitlog_archiving.properties suggest using either > cp or ln for the archive_command. > Using ln is problematic because commitlog recycling marks segments as > recycled once the corresponding memtables are flushed and Cassandra will no > longer replay them. This means it's only possible to do PITR on any records > that were written since the last flush. > Using cp works, and this is currently how OpsCenter does for PITR, however > [~brandon.williams] has pointed out this could have some performance impact > because of the additional I/O overhead of copying the commitlog segments. > Starting in 2.1, we can disable commit log recycling in cassandra.yaml so I > thought this would allow me to do PITR without the extra overhead of using > cp. However, when I disable commitlog recycling and try to do a PITR, > Cassandra blows up when trying to replay the restored commit logs: > {code} > ERROR 16:56:42 Exception encountered during startup > java.lang.IllegalStateException: Cannot safely construct descriptor for > segment, as name and header descriptors do not match ((4,1445878452545) vs > (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log > at > org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335) > ~[dse-core-4.8.0.jar:4.8.0] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) > ~[cassandra-all-2.1.9.791.jar:2.1.9.791] > at com.datastax.bdp.DseModule.main(DseModule.java:75) > [dse-core-4.8.0.jar:4.8.0] > java.lang.IllegalStateException: Cannot safely construct descriptor for > segment, as name and header descriptors do not match ((4,1445878452545) vs > (4,1445876822565)): /opt/dse/backup/CommitLog-4-1445876822565.log > at > org.apache.cassandra.db.commitlog.CommitLogArchiver.maybeRestoreArchive(CommitLogArchiver.java:207) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:116) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:352) > at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:335) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) > at