Replying on the user group. ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand
@aaronmorton http://www.thelastpickle.com On 14/04/2013, at 3:50 PM, Boris Yen <yulin...@gmail.com> wrote: > Hi All, > > Recently, we encountered an error on 1.0.12 that prevented cassandra from > starting up. From the log messages, it looked like the table/keyspace was > opened before the scrubDataDirectories was executed. This created a race > condition between two threads. One was trying to rename files while the > other was trying to remove tmp files. I was wondering if anyone could > provide us some information or workaround for this. > > INFO [MemoryMeter:1] 2013-04-09 02:49:39,868 Memtable.java (line 186) > CFS(Keyspace='fmzd', ColumnFamily='alarm.fmzd_alarm_category') liveRatio is > 3.7553409423470883 (just-counted was 3.1413828689370487). calculation took > 2ms for 265 columns > INFO [SSTableBatchOpen:1] 2013-04-09 02:49:39,868 SSTableReader.java (line > 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshRole-hd-2 (83 bytes) > INFO [SSTableBatchOpen:2] 2013-04-09 02:49:39,868 SSTableReader.java (line > 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshRole-hd-1 (123 bytes) > INFO [Creating index: alarm.fmzd_alarm_category] 2013-04-09 02:49:39,874 > ColumnFamilyStore.java (line 705) Enqueuing flush of > Memtable-alarm.fmzd_alarm_category@413535513(14025/65835 serialized/live > bytes, 275 ops) > INFO [OptionalTasks:1] 2013-04-09 02:49:39,877 SecondaryIndexManager.java > (line 184) Creating new index : ColumnDefinition{name=6d65736853534944, > validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, > index_name='fmzd_ap_meshSSID'} > INFO [SSTableBatchOpen:1] 2013-04-09 02:49:39,895 SSTableReader.java (line > 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshSSID-hd-1 (122 bytes) > INFO [SSTableBatchOpen:2] 2013-04-09 02:49:39,896 SSTableReader.java (line > 153) Opening /test/db/data/fmzd/ap.fmzd_ap_meshSSID-hd-2 (82 bytes) > INFO [OptionalTasks:1] 2013-04-09 02:49:39,900 SecondaryIndexManager.java > (line 184) Creating new index : > ColumnDefinition{name=6d6f62696c6974795a6f6e6555554944, > validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, > index_name='fmzd_ap_mobilityZoneUUID'} > ERROR [FlushWriter:1] 2013-04-09 02:49:39,916 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[FlushWriter:1,5,main] > java.io.IOError: java.io.IOException: rename failed of > /test/db/data/fmzd/alarm.fmzd_alarm_alarmCode-hd-21-Data.db > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:375) > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:319) > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:302) > at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:276) > at org.apache.cassandra.db.Memtable.access$400(Memtable.java:49) > at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:299) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: rename failed of > /test/db/data/fmzd/alarm.fmzd_alarm_alarmCode-hd-21-Data.db > at > org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:355) > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:371) > ... 9 more > INFO [SSTableBatchOpen:1] 2013-04-09 02:49:39,917 SSTableReader.java (line > 153) Opening /test/db/data/fmzd/ap.fmzd_ap_mobilityZoneUUID-hd-1 (312 bytes) > INFO [FlushWriter:2] 2013-04-09 02:49:39,916 Memtable.java (line 246) > Writing Memtable-alarm.fmzd_alarm_alarmCode@402202831(2958/22542 > serialized/live bytes, 58 ops) > ERROR [main] 2013-04-09 02:49:39,916 AbstractCassandraDaemon.java (line > 373) Exception encountered during startup > java.io.IOError: java.io.IOException: Failed to delete > /test/db/data/fmzd/alarm.fmzd_alarm_alarmCode-tmp-hd-21-Statistics.db > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:372) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:415) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:193) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:356) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107) > Caused by: java.io.IOException: Failed to delete > /test/db/data/fmzd/alarm.fmzd_alarm_alarmCode-tmp-hd-21-Statistics.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) > at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141) > at > org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:368) > ... 4 more > INFO [OptionalTasks:1] 2013-04-09 02:49:39,923 SecondaryIndexManager.java > (line 184) Creating new index : ColumnDefinition{name=6d6f64656c, > validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, > index_name='fmzd_ap_model'} > > Thanks and Regards, > Boris
