Re: cleaning up storm generated files
Storm is supposed to be cleaning them up itself. The supervisor is supposed to know when a topology is no longer needed and remove them after a specific timeout. If this is not happening for you there may be a bug in the code. Could you file a JIRA and post the JIRA number here. If you could include the version number of storm and any logs you have from the supervisor that would be helpful. I cannot guarantee how quickly the community can get to this though. - Bobby On Tuesday, March 3, 2015 4:39 PM, clay teahouse clayteaho...@gmail.com wrote: Hi all, How do I remove storm generated system files programmatically without stepping over the files that I shouldn't be deleting? I need this to cleanup the files left behind from the aborted topologies. I tried fuser and lsof, but with no luck. For example, fuser shows stormconf.ser and stormcode.ser not in use even in cases where these files are being used. thanksClay
Re: cleaning up storm generated files
Even if the supervisor is killed, when it comes back up it should do the cleanup. The way it knows is that it will download from zookeeper the list of workers that it is supposed to be running, and then compare that with what it knows about. If a worker is detected that it does not know about it will kill the worker, and clean up the files. This happens when you see a log message like Shutting down and clearing state for id... in the supervisor. It is possible that we got an error and the local state of the supervisor got off. The only way to know what is not needed is to look at zookeeper and do what the supervisor does to determine what should be running. - Bobby On Wednesday, March 4, 2015 11:46 AM, clay teahouse clayteaho...@gmail.com wrote: Is this also the case if you kill a topology or the machine goes down? Or do I need to write a jvm hook to handle this? But in any case, is there a way to know which files are in use by storm (as I said fuser and lsof are not of no help here). thanks. On Wed, Mar 4, 2015 at 11:34 AM, Bobby Evans ev...@yahoo-inc.com.invalid wrote: Storm is supposed to be cleaning them up itself. The supervisor is supposed to know when a topology is no longer needed and remove them after a specific timeout. If this is not happening for you there may be a bug in the code. Could you file a JIRA and post the JIRA number here. If you could include the version number of storm and any logs you have from the supervisor that would be helpful. I cannot guarantee how quickly the community can get to this though. - Bobby On Tuesday, March 3, 2015 4:39 PM, clay teahouse clayteaho...@gmail.com wrote: Hi all, How do I remove storm generated system files programmatically without stepping over the files that I shouldn't be deleting? I need this to cleanup the files left behind from the aborted topologies. I tried fuser and lsof, but with no luck. For example, fuser shows stormconf.ser and stormcode.ser not in use even in cases where these files are being used. thanksClay
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25820615 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server, jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) --- End diff -- Because `check_output` absolutes all outout from command. It is necessary to `print` alternatively to see stdout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77264323 @miguno Thank you for comment. I tested the error handling and exit code of current storm command. And I found when any exception has occurred `check_output` returns non-zero code(just only 1). So with this patch, storm command can detect the occurrence of exception. But error exit code always be 1, so in order to specify what exception is occurred, we must parse stdout and stderr. This is what I was trying to do with this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347674#comment-14347674
]
ASF GitHub Bot commented on STORM-695:
--
Github user Lewuathe commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25820615
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server,
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
--- End diff --
Because `check_output` absolutes all outout from command. It is necessary
to `print` alternatively to see stdout.
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
[jira] [Assigned] (STORM-477) Incorrectly set JAVA_HOME is not detected
[ https://issues.apache.org/jira/browse/STORM-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Ostashchuk reassigned STORM-477: - Assignee: Oleg Ostashchuk Incorrectly set JAVA_HOME is not detected - Key: STORM-477 URL: https://issues.apache.org/jira/browse/STORM-477 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.2-incubating Environment: Rhel6 Reporter: Paul Poulosky Assignee: Oleg Ostashchuk Priority: Minor Labels: newbie If JAVA_HOME is incorrectly set in a user's environment when launching storm, it fails with an error message that is confusing to end users. Traceback (most recent call last): File /home/y/bin/storm, line 485, in module main() File /home/y/bin/storm, line 482, in main (COMMANDS.get(COMMAND, unknown_command))(*ARGS) File /home/y/bin/storm, line 225, in listtopos extrajars=[USER_CONF_DIR, STORM_DIR + /bin]) File /home/y/bin/storm, line 153, in exec_storm_class ] + jvmopts + [klass] + list(args) File /home/y/bin/storm, line 97, in confvalue p = sub.Popen(command, stdout=sub.PIPE) File /usr/lib64/python2.6/subprocess.py, line 642, in __init__ errread, errwrite) File /usr/lib64/python2.6/subprocess.py, line 1234, in _execute_child raise child_exception It would be nice if this were either detected and a proper error message printed, or if it warned and fell back to the java found in PATH. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-698) Jackson incompatibility ring-json
thinker0 created STORM-698:
--
Summary: Jackson incompatibility ring-json
Key: STORM-698
URL: https://issues.apache.org/jira/browse/STORM-698
Project: Apache Storm
Issue Type: Bug
Reporter: thinker0
Priority: Minor
ring-json - com.fasterxml.jackson.core:jackson-core:2.3.1 incompatibility
{code}
2015-03-04 12:34:35 b.s.d.worker [ERROR] Error on initialization of server
mk-worker
java.lang.RuntimeException: java.io.InvalidClassException:
com.fasterxml.jackson.core.JsonFactory; local class incompatible: stream
classdesc serialVersionUID = 3306684576057132431, local class serialVersionUID
= 3194418244231611666
at
backtype.storm.serialization.DefaultSerializationDelegate.deserialize(DefaultSerializationDelegate.java:56)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.Utils.deserialize(Utils.java:95)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.Utils.getSetComponentObject(Utils.java:234)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$get_task_object.invoke(task.clj:79)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task_data$fn__4304.invoke(task.clj:181)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.util$assoc_apply_self.invoke(util.clj:894)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task_data.invoke(task.clj:174)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task.invoke(task.clj:185)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
backtype.storm.daemon.executor$mk_executor$fn__4499.invoke(executor.clj:337)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.core$map$fn__4245.invoke(core.clj:2557) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:na]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.6.0.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.6.0.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
~[clojure-1.6.0.jar:na]
at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
~[clojure-1.6.0.jar:na]
at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)
~[clojure-1.6.0.jar:na]
at clojure.core$reduce.invoke(core.clj:6289) ~[clojure-1.6.0.jar:na]
at clojure.core$into.invoke(core.clj:6341) ~[clojure-1.6.0.jar:na]
at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:337)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto__$reify__5033$iter__5038__5042$fn__5043.invoke(worker.clj:459)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.6.0.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.6.0.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.6.0.jar:na]
at clojure.core$dorun.invoke(core.clj:2856) ~[clojure-1.6.0.jar:na]
at clojure.core$doall.invoke(core.clj:2871) ~[clojure-1.6.0.jar:na]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto__$reify__5033.run(worker.clj:459)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_31]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_31]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto5032.invoke(worker.clj:433)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:178) [clojure-1.6.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:na]
at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:na]
at
backtype.storm.daemon.worker$fn__5031$mk_worker__5108.doInvoke(worker.clj:416)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:548)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.6.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
Caused by: java.io.InvalidClassException:
com.fasterxml.jackson.core.JsonFactory; local class incompatible: stream
classdesc serialVersionUID = 3306684576057132431, local class serialVersionUID
= 3194418244231611666
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
~[na:1.8.0_31]
at
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346591#comment-14346591
]
Michael Noll commented on STORM-695:
But I have a question. Is there anyway to handle exceptions and errors
through thrift interface? Though I am not familiar with thrift so much, I
want to know how to do it. Thank you.
Kai, unfortunately I don't know either (Thrift). :-/
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
))
{code}
As you can see the current implementation does not pass success/failure
information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346770#comment-14346770
]
Kai Sasaki commented on STORM-695:
--
[~miguno]Thanks. I'll find out how to do it.
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
))
{code}
As you can see the current implementation does not pass success/failure
information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346863#comment-14346863
]
ASF GitHub Bot commented on STORM-695:
--
GitHub user Lewuathe opened a pull request:
https://github.com/apache/storm/pull/456
[STORM-695] storm CLI tool reports zero exit code on error scenario
Handling `java.lang.RuntimeException` and `NotAliveException` in storm
command.
This feature can be useful for handling a exit code returned from shell
command.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Lewuathe/storm STORM-695
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #456
commit 7e36ce579f0dcbcf4dfe5942b906a6e94a1815e8
Author: lewuathe lewua...@me.com
Date: 2015-03-04T13:14:51Z
[STORM-695] storm CLI tool reports zero exit code on error scenario
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346870#comment-14346870
]
ASF GitHub Bot commented on STORM-695:
--
Github user Lewuathe commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-77156141
However the topology is submitted with `storm shell`, this cannot track
exception because `spawnvp` seems to return no stdout and stderr. I don't know
the reason why in this case `spawnvp` is called. Can I migrate these two cases
into `subprocess.check_output`?
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
))
{code}
As you can see the current implementation does not pass success/failure
information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77156141 However the topology is submitted with `storm shell`, this cannot track exception because `spawnvp` seems to return no stdout and stderr. I don't know the reason why in this case `spawnvp` is called. Can I migrate these two cases into `subprocess.check_output`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346877#comment-14346877
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25774104
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server,
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
+except sub.CalledProcessError as e:
+# Handling exception with stdout and stderr
+if e.returncode != 0:
--- End diff --
I think this check is redundant, given what the Python docs say:
subprocess.check_output(): [...] If the return code was non-zero it
raises a CalledProcessError.
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25774104 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server, jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) +except sub.CalledProcessError as e: +# Handling exception with stdout and stderr +if e.returncode != 0: --- End diff -- I think this check is redundant, given what the Python docs say: subprocess.check_output(): [...] If the return code was non-zero it raises a CalledProcessError. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346883#comment-14346883
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-77159580
Many thanks for the PR, Kai!
However if the topology is submitted with `storm shell`, this cannot
track exception because spawnvp seems to return no stdout and stderr.
See [my
comment](https://issues.apache.org/jira/browse/STORM-695?focusedCommentId=14340333page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14340333)
in the STORM-695 JIRA ticket. `os.spawnvp` returns the exit code of the
process:
```python
# return code can be 0 or non-zero, i.e. there will be no exception thrown
like for sub.check_output
return_code = os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
```
However the problem is that Storm is currently not returning any non-zero
return codes, even in the face of errors. I haven't had the chance to
test-drive your code yet, but the code in the PR might not work for the same
reason:
```python
try:
ret = sub.check_output(all_args, stderr=sub.STDOUT)
print(ret)
except sub.CalledProcessError as e:
```
If I understand the Python docs correctly, the `CalledProcessError` will
only be thrown if the process returns with a non-zero exit code, and with
current Storm this won't happen -- the return code will always be zero I think.
One workaround could be to slightly adjust the code in the PR to simply
grep for error messages in `STDOUT` / `STDERR` -- regardless of what the
return code of the process is -- and, if there is a match, consider the
process/command failed.
(And please correct me if I'm wrong.)
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77159580 Many thanks for the PR, Kai! However if the topology is submitted with `storm shell`, this cannot track exception because spawnvp seems to return no stdout and stderr. See [my comment](https://issues.apache.org/jira/browse/STORM-695?focusedCommentId=14340333page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14340333) in the STORM-695 JIRA ticket. `os.spawnvp` returns the exit code of the process: ```python # return code can be 0 or non-zero, i.e. there will be no exception thrown like for sub.check_output return_code = os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) ``` However the problem is that Storm is currently not returning any non-zero return codes, even in the face of errors. I haven't had the chance to test-drive your code yet, but the code in the PR might not work for the same reason: ```python try: ret = sub.check_output(all_args, stderr=sub.STDOUT) print(ret) except sub.CalledProcessError as e: ``` If I understand the Python docs correctly, the `CalledProcessError` will only be thrown if the process returns with a non-zero exit code, and with current Storm this won't happen -- the return code will always be zero I think. One workaround could be to slightly adjust the code in the PR to simply grep for error messages in `STDOUT` / `STDERR` -- regardless of what the return code of the process is -- and, if there is a match, consider the process/command failed. (And please correct me if I'm wrong.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
GitHub user Lewuathe opened a pull request: https://github.com/apache/storm/pull/456 [STORM-695] storm CLI tool reports zero exit code on error scenario Handling `java.lang.RuntimeException` and `NotAliveException` in storm command. This feature can be useful for handling a exit code returned from shell command. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Lewuathe/storm STORM-695 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/456.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #456 commit 7e36ce579f0dcbcf4dfe5942b906a6e94a1815e8 Author: lewuathe lewua...@me.com Date: 2015-03-04T13:14:51Z [STORM-695] storm CLI tool reports zero exit code on error scenario --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25774112 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server, jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) --- End diff -- Why the print? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346878#comment-14346878
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25774112
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype=-server,
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
--- End diff --
Why the print?
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Assignee: Kai Sasaki
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
))
{code}
As you can see the current implementation does not pass success/failure
[jira] [Commented] (STORM-675) Allow users to have storm-env.sh under config dir to set custom JAVA_HOME and other env variables
[ https://issues.apache.org/jira/browse/STORM-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346957#comment-14346957 ] ASF GitHub Bot commented on STORM-675: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/434#issuecomment-77167715 @ptgoetz can you please take a look at this. Allow users to have storm-env.sh under config dir to set custom JAVA_HOME and other env variables - Key: STORM-675 URL: https://issues.apache.org/jira/browse/STORM-675 Project: Apache Storm Issue Type: Bug Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-496) task.clj missing debug for logging spout and bolt emit values
[ https://issues.apache.org/jira/browse/STORM-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346954#comment-14346954 ] ASF GitHub Bot commented on STORM-496: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/446#issuecomment-77167619 @revans2 @ptgoetz can you please take a look at this. task.clj missing debug for logging spout and bolt emit values - Key: STORM-496 URL: https://issues.apache.org/jira/browse/STORM-496 Project: Apache Storm Issue Type: Bug Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani Labels: newbie, security task.clj in security branch missing debug config for logging spout , bolt emit values. https://github.com/apache/incubator-storm/blob/security/storm-core/src/clj/backtype/storm/daemon/task.clj https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L130 https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L133 https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L151 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-496. task.clj missing debug for logging ...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/446#issuecomment-77167619 @revans2 @ptgoetz can you please take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-675. Allow users to have storm-env.sh un...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/434#issuecomment-77167715 @ptgoetz can you please take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
