[jira] [Updated] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[ https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-17136: - Fix Version/s: 4.0.2 (was: 4.0.x) Since Version: 4.0.0 Source Control Link: https://github.com/apache/cassandra/commit/aa82e3e4277397dbf630c7b805c5aaca22d660ad Resolution: Fixed Status: Resolved (was: Ready to Commit) Test run looks good now, committed. Thanks! > FQL: Enabling via nodetool can trigger disk_failure_mode > > > Key: CASSANDRA-17136 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17136 > Project: Cassandra > Issue Type: Bug > Components: Tool/fql >Reporter: Brendan Cicchi >Assignee: Brandon Williams >Priority: Normal > Fix For: 4.0.2 > > > When enabling fullquerylog via nodetool, if there is a non empty directory > present under the location specified via --path which would trigger an > java.nio.file.AccessDeniedException during cleaning, the node will trigger > the disk_failure_policy which by default is stop. This is a fairly easy way > to offline a cluster if someone executes this in parallel. I don't that think > the behavior is desirable for enabling via nodetool. > > Repro (1 node cluster already up): > {code:bash} > mkdir /some/path/dir > touch /some/path/dir/file > chown -R user: /some/path/dir # Non Cassandra process user > chmod 700 /some/path/dir > nodetool enablefullquerylog --path /some/path > {code} > Nodetool will give back this error: > {code:java} > error: /some/path/dir/file > -- StackTrace -- > java.nio.file.AccessDeniedException: /some/path/dir/file > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237) > at > org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492) > at > org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477) > at > org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436) > at > org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106) > at > org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) > at sun.r
[jira] [Updated] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[ https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-17136: - Status: Ready to Commit (was: Review In Progress) > FQL: Enabling via nodetool can trigger disk_failure_mode > > > Key: CASSANDRA-17136 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17136 > Project: Cassandra > Issue Type: Bug > Components: Tool/fql >Reporter: Brendan Cicchi >Assignee: Brandon Williams >Priority: Normal > Fix For: 4.0.x > > > When enabling fullquerylog via nodetool, if there is a non empty directory > present under the location specified via --path which would trigger an > java.nio.file.AccessDeniedException during cleaning, the node will trigger > the disk_failure_policy which by default is stop. This is a fairly easy way > to offline a cluster if someone executes this in parallel. I don't that think > the behavior is desirable for enabling via nodetool. > > Repro (1 node cluster already up): > {code:bash} > mkdir /some/path/dir > touch /some/path/dir/file > chown -R user: /some/path/dir # Non Cassandra process user > chmod 700 /some/path/dir > nodetool enablefullquerylog --path /some/path > {code} > Nodetool will give back this error: > {code:java} > error: /some/path/dir/file > -- StackTrace -- > java.nio.file.AccessDeniedException: /some/path/dir/file > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237) > at > org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492) > at > org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477) > at > org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436) > at > org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106) > at > org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Meth
[jira] [Updated] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[ https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Berenguer Blasi updated CASSANDRA-17136: Reviewers: Berenguer Blasi, Berenguer Blasi Berenguer Blasi, Berenguer Blasi (was: Berenguer Blasi) Status: Review In Progress (was: Patch Available) > FQL: Enabling via nodetool can trigger disk_failure_mode > > > Key: CASSANDRA-17136 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17136 > Project: Cassandra > Issue Type: Bug > Components: Tool/fql >Reporter: Brendan Cicchi >Assignee: Brandon Williams >Priority: Normal > Fix For: 4.0.x > > > When enabling fullquerylog via nodetool, if there is a non empty directory > present under the location specified via --path which would trigger an > java.nio.file.AccessDeniedException during cleaning, the node will trigger > the disk_failure_policy which by default is stop. This is a fairly easy way > to offline a cluster if someone executes this in parallel. I don't that think > the behavior is desirable for enabling via nodetool. > > Repro (1 node cluster already up): > {code:bash} > mkdir /some/path/dir > touch /some/path/dir/file > chown -R user: /some/path/dir # Non Cassandra process user > chmod 700 /some/path/dir > nodetool enablefullquerylog --path /some/path > {code} > Nodetool will give back this error: > {code:java} > error: /some/path/dir/file > -- StackTrace -- > java.nio.file.AccessDeniedException: /some/path/dir/file > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237) > at > org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492) > at > org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477) > at > org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436) > at > org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106) > at > org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.
[jira] [Updated] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[ https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-17136: - Test and Documentation Plan: Run CI Status: Patch Available (was: In Progress) > FQL: Enabling via nodetool can trigger disk_failure_mode > > > Key: CASSANDRA-17136 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17136 > Project: Cassandra > Issue Type: Bug > Components: Tool/fql >Reporter: Brendan Cicchi >Assignee: Brandon Williams >Priority: Normal > Fix For: 4.0.x > > > When enabling fullquerylog via nodetool, if there is a non empty directory > present under the location specified via --path which would trigger an > java.nio.file.AccessDeniedException during cleaning, the node will trigger > the disk_failure_policy which by default is stop. This is a fairly easy way > to offline a cluster if someone executes this in parallel. I don't that think > the behavior is desirable for enabling via nodetool. > > Repro (1 node cluster already up): > {code:bash} > mkdir /some/path/dir > touch /some/path/dir/file > chown -R user: /some/path/dir # Non Cassandra process user > chmod 700 /some/path/dir > nodetool enablefullquerylog --path /some/path > {code} > Nodetool will give back this error: > {code:java} > error: /some/path/dir/file > -- StackTrace -- > java.nio.file.AccessDeniedException: /some/path/dir/file > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237) > at > org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492) > at > org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477) > at > org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436) > at > org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106) > at > org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
[jira] [Updated] (CASSANDRA-17136) FQL: Enabling via nodetool can trigger disk_failure_mode
[ https://issues.apache.org/jira/browse/CASSANDRA-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-17136: - Bug Category: Parent values: Availability(12983)Level 1 values: Response Crash(12991) Complexity: Normal Discovered By: User Report Fix Version/s: 4.0.x Severity: Critical Status: Open (was: Triage Needed) This sounds fairly serious from the description, setting the priority accordingly. > FQL: Enabling via nodetool can trigger disk_failure_mode > > > Key: CASSANDRA-17136 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17136 > Project: Cassandra > Issue Type: Bug > Components: Tool/fql >Reporter: Brendan Cicchi >Priority: Normal > Fix For: 4.0.x > > > When enabling fullquerylog via nodetool, if there is a non empty directory > present under the location specified via --path which would trigger an > java.nio.file.AccessDeniedException during cleaning, the node will trigger > the disk_failure_policy which by default is stop. This is a fairly easy way > to offline a cluster if someone executes this in parallel. I don't that think > the behavior is desirable for enabling via nodetool. > > Repro (1 node cluster already up): > {code:bash} > mkdir /some/path/dir > touch /some/path/dir/file > chown -R user: /some/path/dir # Non Cassandra process user > chmod 700 /some/path/dir > nodetool enablefullquerylog --path /some/path > {code} > Nodetool will give back this error: > {code:java} > error: /some/path/dir/file > -- StackTrace -- > java.nio.file.AccessDeniedException: /some/path/dir/file > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:250) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:237) > at > org.apache.cassandra.utils.binlog.BinLog.deleteRecursively(BinLog.java:492) > at > org.apache.cassandra.utils.binlog.BinLog.cleanDirectory(BinLog.java:477) > at > org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:436) > at > org.apache.cassandra.fql.FullQueryLogger.enable(FullQueryLogger.java:106) > at > org.apache.cassandra.service.StorageService.enableFullQueryLogger(StorageService.java:5915) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) > at > javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >