[
https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453978#comment-16453978
]
Thomas Steinmaurer edited comment on CASSANDRA-8743 at 4/26/18 1:05 PM:
------------------------------------------------------------------------
As we are facing the same issue with Cassandra 2.1.18 and NFS (although on
Linux and not Windows) I wonder if there was any movement in the last
years/months which has not been commented on this ticket? Thanks.
What might happen in context of a sequential repair is (although no evidence,
due to lack of test environment):
* Cassandra deletes the snapshotted SSTable (symlink) without closing the file
handle
* NFS created a hidden file ".nfsSOMETHING" for a deleted file where a process
still have a file handle open
* Cassandra then will fail upon deleting the snapshot directory
Just a very wild, speculative guess.
was (Author: tsteinmaurer):
As we are facing the same issue with Cassandra 2.1.18 and NFS (although on
Linux and not Windows) I wonder if there was any movement in the last
years/months which has not been commented on this ticket? Thanks.
What might happen is (although no evidence, due to lack of test environment):
* Cassandra deletes the snapshotted SSTable (symlink) without closing the file
handle
* NFS created a hidden file ".nfsSOMETHING" for a deleted file where a process
still have a file handle open
* Cassandra then will fail upon deleting the snapshot directory
Just a very wild, speculative guess.
> NFS doesn't behave on Windows
> -----------------------------
>
> Key: CASSANDRA-8743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8743
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Streaming and Messaging
> Reporter: Tamar Nirenberg
> Priority: Minor
> Fix For: 2.2.x
>
> Attachments: docker-system.log
>
>
> Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes
> the ring:
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 -
> Failed creating a merkle tree for [repair
> #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS,
> (11621838520493020277529637175352775759,11853478749048239324667887059881170862]],
> /10.1.234.63 (see log for details)
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153
> - Exception in thread Thread[ValidationExecutor:2,1,main]
> org.apache.cassandra.io.FSWriteError:
> java.nio.file.DirectoryNotEmptyException:
> /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2223)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:939)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.nio.file.DirectoryNotEmptyException:
> /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> ~[na:1.7.0_71]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.7.0_71]
> at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ... 10 common frames omitted
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:383
> - Stopping gossiper
> WARN [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:291
> - Stopping gossip by operator request
> INFO [ValidationExecutor:2] 2015-01-22 11:48:14,829 Gossiper.java:1318 -
> Announcing shutdown
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
