We have similar issue. I'll try to reproduce it and upload logs soon. On Wed, May 9, 2012 at 3:30 AM, André Cruz <andre.c...@co.sapo.pt> wrote: > Here it is: https://issues.apache.org/jira/browse/CASSANDRA-4230 > > Please let me know if you need further info. > > Best regards, > André > > On May 8, 2012, at 23:55 , aaron morton wrote: > > Could you please create a ticket for this > on https://issues.apache.org/jira/browse/CASSANDRA > > Please include: > * operating system > * keyspace / column family definition > * output of of "ls -lah" for the "/var/lib/cassandra/data/Disco/Client/" > directory after the error occurs. > > Thanks > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 8/05/2012, at 9:00 AM, André Cruz wrote: > > Hello. > > Since I upgraded to Cassandra 1.1, I get the following error when trying to > delete a CF. After this happens the CF is not accessible anymore, but I > cannot create another one with the same name until I restart the server. > > INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java (line > 634) Enqueuing flush of Memtable-schema_columnfamilies@1128094887(978/1222 > serialized/live bytes, 21 ops) > INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266) > Writing Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live > bytes, 21 ops) > INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307) > Completed flushing > /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db > (1041 bytes) > INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java (line > 634) Enqueuing flush of Memtable-schema_columns@1599271050(392/490 > serialized/live bytes, 8 ops) > INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266) > Writing Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 > ops) > INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722 CompactionTask.java > (line 114) Compacting > [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam > ilies-hc-27-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')] > INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307) > Completed flushing > /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db > (447 bytes) > INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811 CompactionTask.java > (line 225) Compacted to > [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,]. > 24,797 to 21,431 > (~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms. > ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line 158) > Unable to create hard link > com.sun.jna.LastErrorException: errno was 17 > at org.apache.cassandra.utils.CLibrary.link(Native Method) > at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150) > at > org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483) > at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512) > at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403) > at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270) > at > org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java (line > 204) Error occurred during processing of message. > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.io.IOError: java.io.IOException: Unable to create hard link from > /var/lib/cassandra/data/Disco/Client/Client.json to /var/lib/cassandra/data/ > Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17) > at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372) > at > org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191) > at > org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182) > at > org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948) > at > org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348) > at > org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.util.concurrent.ExecutionException: java.io.IOError: > java.io.IOException: Unable to create hard link from > /var/lib/cassandra/data/Disco/Client/Client.json to > /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json > (errno 17) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368) > ... 11 more > Caused by: java.io.IOError: java.io.IOException: Unable to create hard link > from /var/lib/cassandra/data/Disco/Client/Client.json to > /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json > (errno 17) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483) > at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512) > at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403) > at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270) > at > org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.io.IOException: Unable to create hard link from > /var/lib/cassandra/data/Disco/Client/Client.json to > /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json > (errno 17) > at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163) > at > org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450) > ... 10 more > ERROR [MigrationStage:1] 2012-05-07 18:10:12,899 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[MigrationStage:1,5,main] > java.io.IOError: java.io.IOException: Unable to create hard link from > /var/lib/cassandra/data/Disco/Client/Client.json to > /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json > (errno 17) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483) > at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512) > at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403) > at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270) > at > org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Unable to create hard link from > /var/lib/cassandra/data/Disco/Client/Client.json to > /var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json > (errno 17) > at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163) > at > org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343) > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450) > ... 10 more > > > > I've tried recreating the data dirs, to see if this was some kind of > permissions problem, but the error happens every time and with any CF. Has > anyone seen this error? > > I'm using the Cassandra Debian package on Debian squeeze and the Sun JVM > (build 1.6.0_26-b03). > > Best regards, > André > > >
-- -- Thanks, Charlie (@mujiang) 一个 木匠 ======= Data Architect Developer http://mujiang.blogspot.com
