I got cassandra-reaper up and running the other day on our 5 node 2.2.8 cluster. Now every once in awhile a node will get the below exception and the node shuts down its cql port. Is this a bug? something I am doing wrong? Are there any workarounds?
-Jacob Willoughby ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,609 CassandraDaemon.java:195 - Exception in thread Thread[AntiEntropyStage:1,5,main] org.apache.cassandra.io.FSWriteError: java.nio.file.NoSuchFileException: /data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db -> /data/cassandra/data/O bjects/Objects-11baea008ccf11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:92) ~[apache-cassandra-2.2.8.jar:2.2.8] at org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1777) ~[apache-cassandra-2.2.8.jar:2.2.8] at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2432) ~[apache-cassandra-2.2.8.jar:2.2.8] at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2557) ~[apache-cassandra-2.2.8.jar:2.2.8] at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:110) ~[apache-cassandra-2.2.8.jar:2.2.8] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.8.jar:2.2.8] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.nio.file.NoSuchFileException: /data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db -> /data/cassandra/data/Objects/Objects-11baea008ccf 11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[na:1.8.0_121] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_121] at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476) ~[na:1.8.0_121] at java.nio.file.Files.createLink(Files.java:1086) ~[na:1.8.0_121] at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:88) ~[apache-cassandra-2.2.8.jar:2.2.8] ... 10 common frames omitted ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:388 - Stopping gossiper WARN [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:284 - Stopping gossip by operator request INFO [AntiEntropyStage:1] 2017-01-25 12:55:25,611 Gossiper.java:1451 - Announcing shutdown INFO [AntiEntropyStage:1] 2017-01-25 12:55:25,612 StorageService.java:1918 - Node /10.0.3.229 state jump to shutdown