[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679332#comment-17679332 ] Brandon Williams commented on CASSANDRA-11537: -- LGTM, the cqlshlib failures can be ignored and are fallout from CASSANDRA-18088 being committed. +1, back to you, [~paulo]. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679287#comment-17679287 ] Brandon Williams commented on CASSANDRA-11537: -- Circle is running: [j8|https://app.circleci.com/pipelines/github/driftx/cassandra/798/workflows/99acc46f-1ece-4798-b17d-ac98d203b208], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/798/workflows/ff5ac8b8-fc26-4891-9265-bc77009b49f4]. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679262#comment-17679262 ] William Nguyen commented on CASSANDRA-11537: [https://github.com/apache/cassandra/pull/2060|https://github.com/apache/cassandra/pull/2060.] Changed the approach because we didn't have permission forĀ {{StorageService.isInitialized,}} which is why the jmx auth test failed. The new approach finishes registering the mbeans in {{StorageService}} at the end of its initialization. This will cause an {{InstanceNotFoundException}} when the nodetool command is issued and the server isn't ready. I catch and convert this exception to an {{{}IllegalArgumentException{}}}. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655543#comment-17655543 ] Brandon Williams commented on CASSANDRA-11537: -- bq. that will probably time out The multiplexed tests finished and passed, so it's just the jmx auth dtests that need to be addressed. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655531#comment-17655531 ] Brandon Williams commented on CASSANDRA-11537: -- Unfortunately, the jmx auth dtest failures are legit. I didn't notice that every nodetool test was being multiplexed, so that will probably time out, but if it doesn't fail on anything first I think that's sufficient. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655479#comment-17655479 ] Brandon Williams commented on CASSANDRA-11537: -- The failures look unrelated, but I think we should also get the benefit of multiplexing these tests, so we can confirm while doing that. I've started circle runs here: [j8|https://app.circleci.com/pipelines/github/driftx/cassandra/742/workflows/384f3b13-7cae-47a7-8397-35ee2a8d7855], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/742/workflows/2893bb8b-0589-4ef5-8cbe-decfb7a02c93] to that end. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655160#comment-17655160 ] Paulo Motta commented on CASSANDRA-11537: - the patch is ready for a second round of review. Would you have cycles to take a look [~brandon.williams] ? > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655159#comment-17655159 ] Paulo Motta commented on CASSANDRA-11537: - I tested the patch locally and it makes nodetool commands fail gracefully when the node is not initialized: BEFORE: {noformat} $ nodetool ring error: null -- StackTrace -- java.util.NoSuchElementException at com.google.common.collect.LinkedHashMultimap$1.next(LinkedHashMultimap.java:532) at com.google.common.collect.LinkedHashMultimap$1.next(LinkedHashMultimap.java:520) at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47) at java.base/java.util.Collections.max(Unknown Source) at org.apache.cassandra.tools.nodetool.Ring.execute(Ring.java:78) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:372) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:357) at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:260) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:83) {noformat} AFTER {noformat} nodetool: Node is not initialized yet. See 'nodetool help' or 'nodetool help '. {noformat} I'd just slightly update the message to something like: {{Server is not initialized yet, cannot run nodetool.}} I submitted a CI run but failures look unrelated: * https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/2153/ One concern mentioned earlier was if commands like {{nodetool join}} where the server does not join the ring will be affected by this, but it doesn't seem to be the case as we have a test for that that passed: * [test_bootstrap_on_write_survey|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/2153/testReport/dtest-novnode.bootstrap_test/TestBootstrap/test_bootstrap_on_write_survey/] > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 20m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655020#comment-17655020 ] William Nguyen commented on CASSANDRA-11537: I created a PR here: [https://github.com/apache/cassandra/pull/2060.] I took Paulo's suggestion to throw an error at the end of{{{} NodeProbe.connect(){}}}, but doing so I needed to initially not start the test with {{requireNetwork()}} because it sets the initialized flag for {{{}StorageServiceMBean{}}}. Though for this to work, I needed to call {{Class.forName("org.apache.cassandra.service.StorageProxy")}} in the setup to the test. The test itself runs nodetool ring without the initialization set. I also needed to update a series of nodetool tests to include {{requireNetwork(),}} now thatĀ {{NodeProbe}} checks if it is initialized. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Assignee: William Nguyen >Priority: Low > Labels: lhf > Time Spent: 10m > Remaining Estimate: 0h > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644889#comment-17644889 ] Paulo Motta commented on CASSANDRA-11537: - Thanks for the feedback. I agree this should be a rare case, and you can always use other JMX tools to inspect the server. If this turns out to be a problem, we could introduce a {{--skip-init-check}} flag to skip this safer behavior. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Priority: Low > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644872#comment-17644872 ] Brandon Williams commented on CASSANDRA-11537: -- I think the only problem might be if the server wouldn't start due to some kind of problem and you couldn't introspect anything to find out what's going on, but that should be a rare case and generally I don't think you'd find the solution over JMX anyway. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Priority: Low > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17644827#comment-17644827 ] Paulo Motta commented on CASSANDRA-11537: - I might take a stab at this prehistoric ticket. I think the cleanest way to do that would be to check if {{StorageServiceMBean.isInitialized()}} at the end of [NodeProbe.connect()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/NodeProbe.java#L292] and throw a friendly error if not. However, I'm not sure this is the right thing to do, as other Mbeans (ie. GCInspectorMXBean) would be inacessible before {{StorageServiceMBean}} is ready. Do you know if this would be a problem [~brandon.williams]? Otherwise we need to check if {{StorageServiceMBean.isInitialized()}} per {{NodeProbe}} method, and exclude any method which should be accessible before storage service is ready, which would make this a bit uglier and more fragile. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Observability, Local/Startup and Shutdown >Reporter: Edward Capriolo >Priority: Low > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401149#comment-15401149 ] Robert Stupp commented on CASSANDRA-11537: -- [~appodictic], are you still working on this one? > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338599#comment-15338599 ] Robert Stupp commented on CASSANDRA-11537: -- Hm - now other tests fail: http://cassci.datastax.com/job/snazy-CASSANDRA-11537-2-testall/2/#showFailuresLink > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335269#comment-15335269 ] Edward Capriolo commented on CASSANDRA-11537: - Fixed java test issues https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:CASSANDRA-11537-2?expand=1 > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325218#comment-15325218 ] Robert Stupp commented on CASSANDRA-11537: -- The patch breaks a couple of unit tests since these use some methods that are now "guarded" by the check-expression that throws {{IllegalStateException: Can not execute command because startup is not complete.}}. Can you take a look at these? ||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:CASSANDRA-11537-2]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-CASSANDRA-11537-2-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-CASSANDRA-11537-2-dtest/lastSuccessfulBuild/] > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316159#comment-15316159 ] Edward Capriolo commented on CASSANDRA-11537: - Re based to trunk: https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:CASSANDRA-11537-2?expand=1 > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266778#comment-15266778 ] Robert Stupp commented on CASSANDRA-11537: -- Hm. Not really. The process in general is fine. [~iamaleksey] wrote another, nicer [contributor's doc|http://www.mail-archive.com/dev@cassandra.apache.org/msg08672.html]. The [Tick Tock doc|http://www.planetcassandra.org/blog/cassandra-2-2-3-0-and-beyond/] is probably the best source for the general release scheduleI. It actually says, that 2.1 reaches EOL when 3.0 is released, 2.2 is in maintenance mode (critical fixes only) and 3.0 only gets bug fixes. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266761#comment-15266761 ] Edward Capriolo commented on CASSANDRA-11537: - [~snazy] The document: http://wiki.apache.org/cassandra/HowToContribute suggest: You'll want to checkout out the branch corresponding to the lowest version in which you want to introduce the change. Does this documentation require update? > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266674#comment-15266674 ] Joel Knighton commented on CASSANDRA-11537: --- If rebasing on trunk, it's worth noting that the semantics of {{StorageService.isInitialized}} were cleaned up in [CASSANDRA-10134|https://issues.apache.org/jira/browse/CASSANDRA-10134?focusedCommentId=15253719&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15253719] such that it actually matches the description of the {{startupDone}} flag Sylvain suggested above. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1525#comment-1525 ] Robert Stupp commented on CASSANDRA-11537: -- Hmm. The branch you provided is against 2.1, which is EOL. 2.2 and 3.0 are in maintenance mode. I'd categorize this patch as a new feature (not a bug fix) and new features need to go into trunk. Can you rebase against trunk? > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266651#comment-15266651 ] Edward Capriolo commented on CASSANDRA-11537: - This should be good. https://github.com/apache/cassandra/compare/cassandra-2.1...edwardcapriolo:exception-on-startup?expand=1 > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266196#comment-15266196 ] Robert Stupp commented on CASSANDRA-11537: -- Yup, please rebase and squash. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265864#comment-15265864 ] Edward Capriolo commented on CASSANDRA-11537: - It turns out CassandraDaemon was a touch grizzly. I followed you advice with StorageService. Please check the branch. If you like I will squash it. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265814#comment-15265814 ] Edward Capriolo commented on CASSANDRA-11537: - Agreed. That IllegalStateException is the way to go. I will make that change now. Last concept why don't we put isSetupComplete() on the entire server startup sequence? The way I look at this is technically the components that rely on each other could change at any time. Each nodetool command could need N things properly initialized. Rather than go through and figure out which pieces need to be initialized for which command lets just assume there are two cases: * commands like nodetool join that need to be run before full startup (which we wont protect) * commands like compact that we will want to protect with IllegalsStateOperation > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253854#comment-15253854 ] Robert Stupp commented on CASSANDRA-11537: -- I think the better indicator if C* startup has completed, is {{o.a.c.service.CassandraDaemon#setupCompleted}}. I'd rather not like to see a new, custom exception being thrown and serialized to nodetool or another management tool. Just stick with the Java ones, probably IllegalStateException. Java serialization issues could otherwise hide the real cause (so, instead of NotInitializedException you'd maybe get a strange ClassNotFoundExcepion). You could also expose {{o.a.c.service.StorageService#isSetupCompleted}} via JMX, which would be a nice LHF IMO. This would also allow tools to check this status _before_ these actually try to execute a command. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253848#comment-15253848 ] Sylvain Lebresne commented on CASSANDRA-11537: -- bq. I think the user wants the stack trace. IE if my job is to script a cluster wide compaction using nodetool, the 'compact' command should let me know that this failed. The best is likely the non 0 result and message bubble all the way out to the user. I completely agree that we want a non 0 return status for the command on such error, but we can totally have that without a trace. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245724#comment-15245724 ] Edward Capriolo commented on CASSANDRA-11537: - All points well taken with the startupDone flag. I *think* the user wants the stack trace. IE if my job is to script a cluster wide compaction using nodetool, the 'compact' command should let me know that this failed. The best is likely the non 0 result and message bubble all the way out to the user. For example in this case. Imagine if you are going to perform an upgrade/scrub on all nodes after upgrading your cassandra version. If you trigger the command early and there is no error return. This could lead you to wrongfully assume the data files are upgraded. You want to know if the command failed or not. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245412#comment-15245412 ] Sylvain Lebresne commented on CASSANDRA-11537: -- I'm not sure using {{Keyspace.initialized}} for this is the best option: {{Keyspace.setInitialized}} is called pretty early during startup and arguably doesn't guarantee everything is ready. For instance, it's set before the commit log is replayed. What I would suggest here is to set some {{startupDone}} flag in {{StorageService}}, which could be set at the end of {{StorageService.initServer()}} and that would be exposed by {{StorageServiceMBean}}. {{nodetool}} could then just check this as part of its startup and exit if the server is not ready. We'd also get the "no stack" for free which I agree would be neat since it doesn't add anything imo. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > Labels: lhf > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235059#comment-15235059 ] Edward Capriolo commented on CASSANDRA-11537: - We could. Let me explain how this works (as I understand it). Nodetool is making JMX calls and printing the results. The old code path was asserting, this code path throws an exception. We could have handled this by returning a status code and mapping the result on the nodetool side to a message. My opinion your approach of trapping the exception and controlling the output is valid. My opinion is that this code is more-or-less a fail safe and under normal circumstances the average user never sees this. Because of my opinion I think we should keep it lean. If however we were going to do a more in-depth patch I would focus on making sure we know which nodetool commands could be implemented at which state, and we build that more into the protocol/storage proxy. > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234926#comment-15234926 ] Christopher Batey commented on CASSANDRA-11537: --- Given we know exactly what this exception could we avoid showing the stack trace? The fewer stack traces users see the better > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for it so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11537) Give clear error when certain nodetool commands are issued before server is ready
[ https://issues.apache.org/jira/browse/CASSANDRA-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232413#comment-15232413 ] Edward Capriolo commented on CASSANDRA-11537: - WIth the patch above the user now sees this: {noformat} [edward@jackintosh apache-cassandra-2.1.13-SNAPSHOT]$ bin/nodetool compact error: Can not execute command because startup is not complete. -- StackTrace -- org.apache.cassandra.service.NotReadyException: Can not execute command because startup is not complete. at org.apache.cassandra.service.StorageService.throwIfNotInitialized(StorageService.java:4447) at org.apache.cassandra.service.StorageService.forceKeyspaceCompaction(StorageService.java:2449) {noformat} > Give clear error when certain nodetool commands are issued before server is > ready > - > > Key: CASSANDRA-11537 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11537 > Project: Cassandra > Issue Type: Improvement >Reporter: Edward Capriolo >Assignee: Edward Capriolo >Priority: Minor > > As an ops person upgrading and servicing Cassandra servers, I require a more > clear message when I issue a nodetool command that the server is not ready > for so that I am not confused. > Technical description: > If you deploy a new binary, restart, and issue nodetool > scrub/compact/updatess etc you get unfriendly assertion. An exception would > be easier to understand. Also if a user has turned assertions off it is > unclear what might happen. > {noformat} > EC1: Throw exception to make it clear server is still in start up process. > :~# nodetool upgradesstables > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:97) > at > org.apache.cassandra.service.StorageService.getValidKeyspace(StorageService.java:2573) > at > org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2661) > at > org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2421) > {noformat} > EC1: > Patch against 2.1 (branch) > https://github.com/apache/cassandra/compare/trunk...edwardcapriolo:exception-on-startup?expand=1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)