Hello.
Since I upgraded to Cassandra 1.1, I get the following error when
trying to delete a CF. After this happens the CF is not accessible
anymore, but I cannot create another one with the same name until I
restart the server.
INFO [MigrationStage:1] 2012-05-07 18:10:12,682 ColumnFamilyStore.java
(line 634) Enqueuing flush of
Memtable-schema_columnfamilies@1128094887(978/1222 serialized/live
bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,682 Memtable.java (line 266)
Writing Memtable-schema_columnfamilies@1128094887(978/1222
serialized/live bytes, 21 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,720 Memtable.java (line 307)
Completed flushing
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db (1041
bytes)
INFO [MigrationStage:1] 2012-05-07 18:10:12,721 ColumnFamilyStore.java
(line 634) Enqueuing flush of
Memtable-schema_columns@1599271050(392/490 serialized/live bytes, 8 ops)
INFO [FlushWriter:2] 2012-05-07 18:10:12,722 Memtable.java (line 266)
Writing Memtable-schema_columns@1599271050(392/490 serialized/live
bytes, 8 ops)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,722
CompactionTask.java (line 114) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-26-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-28-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfam
ilies-hc-27-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-25-Data.db')]
INFO [FlushWriter:2] 2012-05-07 18:10:12,806 Memtable.java (line 307)
Completed flushing
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hc-23-Data.db (447
bytes)
INFO [CompactionExecutor:8] 2012-05-07 18:10:12,811
CompactionTask.java (line 225) Compacted to
[/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-29-Data.db,]. 24,797 to
21,431
(~86% of original) bytes for 2 keys at 0.232252MB/s. Time: 88ms.
ERROR [MigrationStage:1] 2012-05-07 18:10:12,895 CLibrary.java (line
158) Unable to create hard link
com.sun.jna.LastErrorException: errno was 17
at org.apache.cassandra.utils.CLibrary.link(Native Method)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:150)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thrift:17] 2012-05-07 18:10:12,898 CustomTThreadPoolServer.java
(line 204) Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/
Disco/Client/snapshots/1336410612893-Client/Client.json (errno 17)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:372)
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:191)
at
org.apache.cassandra.service.MigrationManager.announceColumnFamilyDrop(MigrationManager.java:182)
at
org.apache.cassandra.thrift.CassandraServer.system_drop_column_family(CassandraServer.java:948)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3348)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_drop_column_family.getResult(Cassandra.java:3336)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
... 11 more
Caused by: java.io.IOError: java.io.IOException: Unable to create hard
link from /var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
ERROR [MigrationStage:1] 2012-05-07 18:10:12,899
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[MigrationStage:1,5,main]
java.io.IOError: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.DefsTable.dropColumnFamily(DefsTable.java:512)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:403)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at
org.apache.cassandra.service.MigrationManager$1.call(MigrationManager.java:214)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/Disco/Client/Client.json to
/var/lib/cassandra/data/Disco/Client/snapshots/1336410612893-Client/Client.json (errno
17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 10 more
I've tried recreating the data dirs, to see if this was some kind of
permissions problem, but the error happens every time and with any CF.
Has anyone seen this error?
I'm using the Cassandra Debian package on Debian squeeze and the Sun
JVM (build 1.6.0_26-b03).
Best regards,
André
