Running a 3 node cluster with cassandra-0.8.0-beta1
I'm seeing the first node logging many (thousands) times lines like
Caused by: java.io.IOException: Unable to create hard link
from
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-5504-Data.db
to
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/snapshots/compact-thrift_no_finntech_countstats_count_Count_1299479381593068337/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-5504-Data.db
(errno 17)
This seems to happen for all column families (including system).
It happens a lot during startup.
The hardlinks do exist. Stopping, deleting the hardlinks, and starting
again does not help.
But i haven't seen it once on the other nodes...
~mck
ps the stacktrace
java.io.IOError: java.io.IOException: Unable to create hard link from
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-3875-Data.db
to
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/snapshots/compact-thrift_no_finntech_countstats_count_Count_1299479381593068337/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-3875-Data.db
(errno 17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1629)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1654)
at org.apache.cassandra.db.Table.snapshot(Table.java:198)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:504)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:146)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:112)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-3875-Data.db
to
/iad/finn/countstatistics/cassandra-data/countstatisticsCount/snapshots/compact-thrift_no_finntech_countstats_count_Count_1299479381593068337/thrift_no_finntech_countstats_count_Count_1299479381593068337-f-3875-Data.db
(errno 17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:155)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:713)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1622)
... 10 more
