[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans
[ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4021: -- Attachment: patch_4021.txt Committed Boris's patch from the dev list (attached). > CFS.scrubDataDirectories tries to delete nonexistent orphans > > > Key: CASSANDRA-4021 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4021 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.7 beta 2 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Minor > Labels: datastax_qa > Attachments: 4021.txt, node1.log, patch_4021.txt > > > The check only looks for a missing data file, then deletes all other > components, however it's possible for the data file and another component to > be missing, causing an error: > {noformat} > WARN 17:19:28,765 Removing orphans for > /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: > [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db] > ERROR 17:19:28,766 Exception encountered during startup > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > Exception encountered during startup: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans
[ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4021: Attachment: node1.log dtestbot managed to randomly reproduce this morning. It looks like a race between compaction cleanup and forcible shutdown, then startup. Log attached. > CFS.scrubDataDirectories tries to delete nonexistent orphans > > > Key: CASSANDRA-4021 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4021 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.7 beta 2 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Minor > Labels: datastax_qa > Attachments: 4021.txt, node1.log > > > The check only looks for a missing data file, then deletes all other > components, however it's possible for the data file and another component to > be missing, causing an error: > {noformat} > WARN 17:19:28,765 Removing orphans for > /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: > [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db] > ERROR 17:19:28,766 Exception encountered during startup > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > Exception encountered during startup: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans
[ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4021: -- Fix Version/s: (was: 1.0.11) > CFS.scrubDataDirectories tries to delete nonexistent orphans > > > Key: CASSANDRA-4021 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4021 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.7 beta 2 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Minor > Labels: datastax_qa > Attachments: 4021.txt > > > The check only looks for a missing data file, then deletes all other > components, however it's possible for the data file and another component to > be missing, causing an error: > {noformat} > WARN 17:19:28,765 Removing orphans for > /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: > [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db] > ERROR 17:19:28,766 Exception encountered during startup > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > Exception encountered during startup: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans
[ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4021: Attachment: 4021.txt Apparently this goes all the way back to CASSANDRA-1471 but occurs very rarely. Patch to attempt deletion without caring about success. > CFS.scrubDataDirectories tries to delete nonexistent orphans > > > Key: CASSANDRA-4021 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4021 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.7 beta 2 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Minor > Fix For: 0.8.11, 1.0.9 > > Attachments: 4021.txt > > > The check only looks for a missing data file, then deletes all other > components, however it's possible for the data file and another component to > be missing, causing an error: > {noformat} > WARN 17:19:28,765 Removing orphans for > /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: > [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db] > ERROR 17:19:28,766 Exception encountered during startup > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > Exception encountered during startup: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4021) CFS.scrubDataDirectories tries to delete nonexistent orphans
[ https://issues.apache.org/jira/browse/CASSANDRA-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-4021: Component/s: Core Affects Version/s: 0.7 beta 2 Fix Version/s: 1.0.9 0.8.11 Assignee: Brandon Williams > CFS.scrubDataDirectories tries to delete nonexistent orphans > > > Key: CASSANDRA-4021 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4021 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.7 beta 2 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Minor > Fix For: 0.8.11, 1.0.9 > > > The check only looks for a missing data file, then deletes all other > components, however it's possible for the data file and another component to > be missing, causing an error: > {noformat} > WARN 17:19:28,765 Removing orphans for > /var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-24492: > [Index.db, Filter.db, Digest.sha1, Statistics.db, Data.db] > ERROR 17:19:28,766 Exception encountered during startup > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > java.lang.AssertionError: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:49) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:357) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:352) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:105) > Exception encountered during startup: attempted to delete non-existing file > system-HintsColumnFamily-hd-24492-Index.db > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira