[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456498#comment-17456498
]
Brandon Williams edited comment on CASSANDRA-17136 at 12/9/21, 3:21 PM:
[circle for
trunk|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136-trunk].
I'll check back later to commit if that's good.
was (Author: brandon.williams):
[circle for
trunk|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136-trunk].
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConne
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454201#comment-17454201
]
Brandon Williams edited comment on CASSANDRA-17136 at 12/7/21, 5:43 PM:
We can't reproduce exactly the same error without being able to become another
unix user, but I added a dtest
[here|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-17136] that
reproduces the NPE by reducing permissions.
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1309/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1309/pipeline]
was (Author: brandon.williams):
We can't reproduce exactly the same error without being able to become another
unix user, but I added a dtest
[here:https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-17136] that
reproduces the NPE by reducing permissions.
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1309/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1309/pipeline]
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Defau
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454201#comment-17454201
]
Brandon Williams edited comment on CASSANDRA-17136 at 12/6/21, 7:24 PM:
We can't reproduce exactly the same error without being able to become another
unix user, but I added a dtest
[here:https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-17136] that
reproduces the NPE by reducing permissions.
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1309/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1309/pipeline]
was (Author: brandon.williams):
We can't reproduce exactly the same error without being able to become another
unix user, but I added a dtest
[here:https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-17136] that
reproduces the NPE by reducing permissions.
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1308/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1308/pipeline]
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Defau
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446053#comment-17446053
]
Brandon Williams edited comment on CASSANDRA-17136 at 11/18/21, 5:14 PM:
-
I created an enableWithoutClean method for JMX to use.
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-17136]|[circle|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136],
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1287/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1287/pipeline]|
|[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-17136]|[circle|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136-trunk],
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1288/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1288/pipeline]|
was (Author: brandon.williams):
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-17136]|[circle|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136],
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1287/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1287/pipeline]|
|[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-17136]|[circle|https://app.circleci.com/pipelines/github/driftx/cassandra?branch=CASSANDRA-17136-trunk],
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1288/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1288/pipeline]|
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442767#comment-17442767
]
Brendan Cicchi edited comment on CASSANDRA-17136 at 11/12/21, 2:25 PM:
---
Okay here are the exact commands that repro the issue for me on 4.0.1 (checked
centos and ubuntu just in case). All commands run as root:
{code:java}
mkdir -p /tmp/dir/
chown cassandra:cassandra /tmp/dir
mkdir /tmp/dir/sub
touch /tmp/dir/sub/file
nodetool enablefullquerylog --path /tmp/dir {code}
Let me know if that does not work. Here are the permissions:
{code:java}
/tmp/dir:
total 12
drwxr-xr-x 3 cassandra cassandra 4096 Nov 12 14:18 .
drwxrwxrwt 16 root root 4096 Nov 12 14:18 ..
drwxr-xr-x 2 root root 4096 Nov 12 14:18 sub
/tmp/dir/sub:
total 8
drwxr-xr-x 2 root root 4096 Nov 12 14:18 .
drwxr-xr-x 3 cassandra cassandra 4096 Nov 12 14:18 ..
-rw-r--r-- 1 root root 0 Nov 12 14:18 file {code}
was (Author: bcicchi):
Okay here are the exact commands that repro the issue for me on 4.0.1 (checked
centos and ubuntu just in case). All commands run as root:
{code:java}
mkdir -p /tmp/dir/
chown cassandra:cassandra /tmp/dir
mkdir /tmp/dir/sub
touch /tmp/dir/sub/file
nodetool enablefullquerylog --path /tmp/dir {code}
Let me know if that does not work.
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.m
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442767#comment-17442767
]
Brendan Cicchi edited comment on CASSANDRA-17136 at 11/12/21, 2:24 PM:
---
Okay here are the exact commands that repro the issue for me on 4.0.1 (checked
centos and ubuntu just in case). All commands run as root:
{code:java}
mkdir -p /tmp/dir/
chown cassandra:cassandra /tmp/dir
mkdir /tmp/dir/sub
touch /tmp/dir/sub/file
nodetool enablefullquerylog --path /tmp/dir {code}
Let me know if that does not work.
was (Author: bcicchi):
Okay here are the exact commands that repro the issue for me on 4.0.1 (checked
centos and ubuntu just in case):
{code:java}
mkdir -p /tmp/dir/
chown cassandra:cassandra /tmp/dir
mkdir /tmp/dir/sub
touch /tmp/dir/sub/file
nodetool enablefullquerylog --path /tmp/dir {code}
Let me know if that does not work.
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
>
[jira] [Comment Edited] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[
https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442053#comment-17442053
]
Brandon Williams edited comment on CASSANDRA-17136 at 11/11/21, 2:55 AM:
-
This sounds fairly serious from the description, setting the severity
accordingly.
was (Author: brandon.williams):
This sounds fairly serious from the description, setting the priority
accordingly.
> FQL: Enabling via nodetool can trigger disk_failure_mode
>
>
> Key: CASSANDRA-17136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17136
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/fql
>Reporter: Brendan Cicchi
>Priority: Normal
> Fix For: 4.0.x
>
>
> When enabling fullquerylog via nodetool, if there is a non empty directory
> present under the location specified via --path which would trigger an
> java.nio.file.AccessDeniedException during cleaning, the node will trigger
> the disk_failure_policy which by default is stop. This is a fairly easy way
> to offline a cluster if someone executes this in parallel. I don't that think
> the behavior is desirable for enabling via nodetool.
>
> Repro (1 node cluster already up):
> {code:bash}
> mkdir /some/path/dir
> touch /some/path/dir/file
> chown -R user: /some/path/dir # Non Cassandra process user
> chmod 700 /some/path/dir
> nodetool enablefullquerylog --path /some/path
> {code}
> Nodetool will give back this error:
> {code:java}
> error: /some/path/dir/file
> -- StackTrace --
> java.nio.file.AccessDeniedException: /some/path/dir/file
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250)
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237)
> at
> org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492)
> at
> org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436)
> at
> org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106)
> at
> org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.ref
