[jira] [Created] (STORM-698) Jackson incompatibility ring-json
thinker0 created STORM-698:
--
Summary: Jackson incompatibility ring-json
Key: STORM-698
URL: https://issues.apache.org/jira/browse/STORM-698
Project: Apache Storm
Issue Type: Bug
Reporter: thinker0
Priority: Minor
ring-json -> com.fasterxml.jackson.core:jackson-core:2.3.1 incompatibility
{code}
2015-03-04 12:34:35 b.s.d.worker [ERROR] Error on initialization of server
mk-worker
java.lang.RuntimeException: java.io.InvalidClassException:
com.fasterxml.jackson.core.JsonFactory; local class incompatible: stream
classdesc serialVersionUID = 3306684576057132431, local class serialVersionUID
= 3194418244231611666
at
backtype.storm.serialization.DefaultSerializationDelegate.deserialize(DefaultSerializationDelegate.java:56)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.Utils.deserialize(Utils.java:95)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.utils.Utils.getSetComponentObject(Utils.java:234)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$get_task_object.invoke(task.clj:79)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task_data$fn__4304.invoke(task.clj:181)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.util$assoc_apply_self.invoke(util.clj:894)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task_data.invoke(task.clj:174)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at backtype.storm.daemon.task$mk_task.invoke(task.clj:185)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
backtype.storm.daemon.executor$mk_executor$fn__4499.invoke(executor.clj:337)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.core$map$fn__4245.invoke(core.clj:2557) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:na]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.6.0.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.6.0.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
~[clojure-1.6.0.jar:na]
at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
~[clojure-1.6.0.jar:na]
at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)
~[clojure-1.6.0.jar:na]
at clojure.core$reduce.invoke(core.clj:6289) ~[clojure-1.6.0.jar:na]
at clojure.core$into.invoke(core.clj:6341) ~[clojure-1.6.0.jar:na]
at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:337)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto__$reify__5033$iter__5038__5042$fn__5043.invoke(worker.clj:459)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.6.0.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.6.0.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.6.0.jar:na]
at clojure.core$dorun.invoke(core.clj:2856) ~[clojure-1.6.0.jar:na]
at clojure.core$doall.invoke(core.clj:2871) ~[clojure-1.6.0.jar:na]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto__$reify__5033.run(worker.clj:459)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_31]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_31]
at
backtype.storm.daemon.worker$fn__5031$exec_fn__1694__auto5032.invoke(worker.clj:433)
~[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:178) [clojure-1.6.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:na]
at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:na]
at
backtype.storm.daemon.worker$fn__5031$mk_worker__5108.doInvoke(worker.clj:416)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:548)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.6.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source)
[storm-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
Caused by: java.io.InvalidClassException:
com.fasterxml.jackson.core.JsonFactory; local class incompatible: stream
classdesc serialVersionUID = 3306684576057132431, local class serialVersionUID
= 3194418244231611666
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
~[na:1.8.0_31]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStre
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346591#comment-14346591
]
Michael Noll commented on STORM-695:
> But I have a question. Is there anyway to handle exceptions and errors
> through thrift interface? Though I am not familiar with thrift so much, I
> want to know how to do it. Thank you.
Kai, unfortunately I don't know either (Thrift). :-/
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:kill wait-amt] true)
> ))
> {code}
> As you can see the current implementation does not pass success/failure
> information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346770#comment-14346770
]
Kai Sasaki commented on STORM-695:
--
[~miguno]Thanks. I'll find out how to do it.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:kill wait-amt] true)
> ))
> {code}
> As you can see the current implementation does not pass success/failure
> information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
GitHub user Lewuathe opened a pull request: https://github.com/apache/storm/pull/456 [STORM-695] storm CLI tool reports zero exit code on error scenario Handling `java.lang.RuntimeException` and `NotAliveException` in storm command. This feature can be useful for handling a exit code returned from shell command. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Lewuathe/storm STORM-695 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/456.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #456 commit 7e36ce579f0dcbcf4dfe5942b906a6e94a1815e8 Author: lewuathe Date: 2015-03-04T13:14:51Z [STORM-695] storm CLI tool reports zero exit code on error scenario --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346863#comment-14346863
]
ASF GitHub Bot commented on STORM-695:
--
GitHub user Lewuathe opened a pull request:
https://github.com/apache/storm/pull/456
[STORM-695] storm CLI tool reports zero exit code on error scenario
Handling `java.lang.RuntimeException` and `NotAliveException` in storm
command.
This feature can be useful for handling a exit code returned from shell
command.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Lewuathe/storm STORM-695
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #456
commit 7e36ce579f0dcbcf4dfe5942b906a6e94a1815e8
Author: lewuathe
Date: 2015-03-04T13:14:51Z
[STORM-695] storm CLI tool reports zero exit code on error scenario
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77156141 However the topology is submitted with `storm shell`, this cannot track exception because `spawnvp` seems to return no stdout and stderr. I don't know the reason why in this case `spawnvp` is called. Can I migrate these two cases into `subprocess.check_output`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346870#comment-14346870
]
ASF GitHub Bot commented on STORM-695:
--
Github user Lewuathe commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-77156141
However the topology is submitted with `storm shell`, this cannot track
exception because `spawnvp` seems to return no stdout and stderr. I don't know
the reason why in this case `spawnvp` is called. Can I migrate these two cases
into `subprocess.check_output`?
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:kill wait-amt] true)
> ))
> {code}
> As you can see the current implementation does not pass success/failure
> information back to the caller.
--
This message was sent by
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25774112 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) --- End diff -- Why the print? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346877#comment-14346877
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25774104
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server",
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
+except sub.CalledProcessError as e:
+# Handling exception with stdout and stderr
+if e.returncode != 0:
--- End diff --
I think this check is redundant, given what the Python docs say:
> subprocess.check_output(): [...] If the return code was non-zero it
raises a CalledProcessError.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nim
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25774104 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) +except sub.CalledProcessError as e: +# Handling exception with stdout and stderr +if e.returncode != 0: --- End diff -- I think this check is redundant, given what the Python docs say: > subprocess.check_output(): [...] If the return code was non-zero it raises a CalledProcessError. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346878#comment-14346878
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25774112
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server",
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
--- End diff --
Why the print?
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:ki
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user miguno commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77159580 Many thanks for the PR, Kai! > However if the topology is submitted with `storm shell`, this cannot track exception because spawnvp seems to return no stdout and stderr. See [my comment](https://issues.apache.org/jira/browse/STORM-695?focusedCommentId=14340333&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14340333) in the STORM-695 JIRA ticket. `os.spawnvp` returns the exit code of the process: ```python # return code can be 0 or non-zero, i.e. there will be no exception thrown like for sub.check_output return_code = os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) ``` However the problem is that Storm is currently not returning any non-zero return codes, even in the face of errors. I haven't had the chance to test-drive your code yet, but the code in the PR might not work for the same reason: ```python try: ret = sub.check_output(all_args, stderr=sub.STDOUT) print(ret) except sub.CalledProcessError as e: ``` If I understand the Python docs correctly, the `CalledProcessError` will only be thrown if the process returns with a non-zero exit code, and with current Storm this won't happen -- the return code will always be zero I think. One workaround could be to slightly adjust the code in the PR to simply grep for error messages in `STDOUT` / `STDERR` -- regardless of what the return code of the process is -- and, if there is a match, consider the process/command failed. (And please correct me if I'm wrong.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346883#comment-14346883
]
ASF GitHub Bot commented on STORM-695:
--
Github user miguno commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-77159580
Many thanks for the PR, Kai!
> However if the topology is submitted with `storm shell`, this cannot
track exception because spawnvp seems to return no stdout and stderr.
See [my
comment](https://issues.apache.org/jira/browse/STORM-695?focusedCommentId=14340333&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14340333)
in the STORM-695 JIRA ticket. `os.spawnvp` returns the exit code of the
process:
```python
# return code can be 0 or non-zero, i.e. there will be no exception thrown
like for sub.check_output
return_code = os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
```
However the problem is that Storm is currently not returning any non-zero
return codes, even in the face of errors. I haven't had the chance to
test-drive your code yet, but the code in the PR might not work for the same
reason:
```python
try:
ret = sub.check_output(all_args, stderr=sub.STDOUT)
print(ret)
except sub.CalledProcessError as e:
```
If I understand the Python docs correctly, the `CalledProcessError` will
only be thrown if the process returns with a non-zero exit code, and with
current Storm this won't happen -- the return code will always be zero I think.
One workaround could be to slightly adjust the code in the PR to simply
grep for error messages in `STDOUT` / `STDERR` -- regardless of what the
return code of the process is -- and, if there is a match, consider the
process/command failed.
(And please correct me if I'm wrong.)
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only
[GitHub] storm pull request: STORM-496. task.clj missing debug for logging ...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/446#issuecomment-77167619 @revans2 @ptgoetz can you please take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-675) Allow users to have storm-env.sh under config dir to set custom JAVA_HOME and other env variables
[ https://issues.apache.org/jira/browse/STORM-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346957#comment-14346957 ] ASF GitHub Bot commented on STORM-675: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/434#issuecomment-77167715 @ptgoetz can you please take a look at this. > Allow users to have storm-env.sh under config dir to set custom JAVA_HOME and > other env variables > - > > Key: STORM-675 > URL: https://issues.apache.org/jira/browse/STORM-675 > Project: Apache Storm > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-496) task.clj missing debug for logging spout and bolt emit values
[ https://issues.apache.org/jira/browse/STORM-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346954#comment-14346954 ] ASF GitHub Bot commented on STORM-496: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/446#issuecomment-77167619 @revans2 @ptgoetz can you please take a look at this. > task.clj missing debug for logging spout and bolt emit values > - > > Key: STORM-496 > URL: https://issues.apache.org/jira/browse/STORM-496 > Project: Apache Storm > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Labels: newbie, security > > task.clj in security branch missing debug config for logging spout , bolt > emit values. > https://github.com/apache/incubator-storm/blob/security/storm-core/src/clj/backtype/storm/daemon/task.clj > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L130 > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L133 > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L151 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-689) SimpleACLAuthorizer should provide a way to restrict who can submit topologies
[ https://issues.apache.org/jira/browse/STORM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346956#comment-14346956 ] ASF GitHub Bot commented on STORM-689: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/445#issuecomment-77167658 @revans2 can you please take a look at this. > SimpleACLAuthorizer should provide a way to restrict who can submit topologies > -- > > Key: STORM-689 > URL: https://issues.apache.org/jira/browse/STORM-689 > Project: Apache Storm > Issue Type: Improvement >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani >Priority: Trivial > > SimpleACLAuthorizer currently allows anyone with a valid kerberos ticket to > submit topologies. There are cases where storm admins want to allow only > selected users to submit topologies. I am proposing nimbus.users config > option if its added to storm.yaml only the listed users can deploy the storm > topologies. > cc [~revans2] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/445#issuecomment-77167658 @revans2 can you please take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-675. Allow users to have storm-env.sh un...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/434#issuecomment-77167715 @ptgoetz can you please take a look at this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (STORM-625) Netty context never forgets about a Client.
[ https://issues.apache.org/jira/browse/STORM-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-625: - Assignee: Robert Joseph Evans > Netty context never forgets about a Client. > --- > > Key: STORM-625 > URL: https://issues.apache.org/jira/browse/STORM-625 > Project: Apache Storm > Issue Type: Bug >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: newbie > > The Netty Context > (https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/messaging/netty/Context.java) > keeps track of all connections so that it can close them when it is closed > in term. But it never removes any of them from its list until it is > terminated. If workers are rescheduled to new locations this can cause the > old connections to be leaked. > We should have the Connections remove themselves from the list when they are > closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: cleaning up storm generated files
Storm is supposed to be cleaning them up itself. The supervisor is supposed to know when a topology is no longer needed and remove them after a specific timeout. If this is not happening for you there may be a bug in the code. Could you file a JIRA and post the JIRA number here. If you could include the version number of storm and any logs you have from the supervisor that would be helpful. I cannot guarantee how quickly the community can get to this though. - Bobby On Tuesday, March 3, 2015 4:39 PM, clay teahouse wrote: Hi all, How do I remove storm generated system files programmatically without stepping over the files that I shouldn't be deleting? I need this to cleanup the files left behind from the aborted topologies. I tried fuser and lsof, but with no luck. For example, fuser shows stormconf.ser and stormcode.ser not in use even in cases where these files are being used. thanksClay
Re: cleaning up storm generated files
Is this also the case if you kill a topology or the machine goes down? Or do I need to write a jvm hook to handle this? But in any case, is there a way to know which files are in use by storm (as I said fuser and lsof are not of no help here). thanks. On Wed, Mar 4, 2015 at 11:34 AM, Bobby Evans wrote: > Storm is supposed to be cleaning them up itself. The supervisor is > supposed to know when a topology is no longer needed and remove them after > a specific timeout. If this is not happening for you there may be a bug in > the code. Could you file a JIRA and post the JIRA number here. If you > could include the version number of storm and any logs you have from the > supervisor that would be helpful. I cannot guarantee how quickly the > community can get to this though. > - Bobby > > > On Tuesday, March 3, 2015 4:39 PM, clay teahouse < > clayteaho...@gmail.com> wrote: > > > Hi all, > How do I remove storm generated system files programmatically without > stepping over the files that I shouldn't be deleting? I need this to > cleanup the files left behind from the aborted topologies. I tried fuser > and lsof, but with no luck. For example, fuser shows stormconf.ser and > stormcode.ser not in use even in cases where these files are being used. > thanksClay > > > >
Re: cleaning up storm generated files
Even if the supervisor is killed, when it comes back up it should do the cleanup. The way it knows is that it will download from zookeeper the list of workers that it is supposed to be running, and then compare that with what it knows about. If a worker is detected that it does not know about it will kill the worker, and clean up the files. This happens when you see a log message like "Shutting down and clearing state for id..." in the supervisor. It is possible that we got an error and the local state of the supervisor got off. The only way to know what is not needed is to look at zookeeper and do what the supervisor does to determine what should be running. - Bobby On Wednesday, March 4, 2015 11:46 AM, clay teahouse wrote: Is this also the case if you kill a topology or the machine goes down? Or do I need to write a jvm hook to handle this? But in any case, is there a way to know which files are in use by storm (as I said fuser and lsof are not of no help here). thanks. On Wed, Mar 4, 2015 at 11:34 AM, Bobby Evans wrote: > Storm is supposed to be cleaning them up itself. The supervisor is > supposed to know when a topology is no longer needed and remove them after > a specific timeout. If this is not happening for you there may be a bug in > the code. Could you file a JIRA and post the JIRA number here. If you > could include the version number of storm and any logs you have from the > supervisor that would be helpful. I cannot guarantee how quickly the > community can get to this though. > - Bobby > > > On Tuesday, March 3, 2015 4:39 PM, clay teahouse < > clayteaho...@gmail.com> wrote: > > > Hi all, > How do I remove storm generated system files programmatically without > stepping over the files that I shouldn't be deleting? I need this to > cleanup the files left behind from the aborted topologies. I tried fuser > and lsof, but with no luck. For example, fuser shows stormconf.ser and > stormcode.ser not in use even in cases where these files are being used. > thanksClay > > > >
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on a diff in the pull request: https://github.com/apache/storm/pull/456#discussion_r25820615 --- Diff: bin/storm --- @@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server", jvmopts=[], extrajars=[], args=[] os.spawnvp(os.P_WAIT, JAVA_CMD, all_args) else: # handling whitespaces in JAVA_CMD -sub.call(all_args) +try: +ret = sub.check_output(all_args, stderr=sub.STDOUT) +print(ret) --- End diff -- Because `check_output` absolutes all outout from command. It is necessary to `print` alternatively to see stdout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user Lewuathe commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-77264323 @miguno Thank you for comment. I tested the error handling and exit code of current storm command. And I found when any exception has occurred `check_output` returns non-zero code(just only 1). So with this patch, storm command can detect the occurrence of exception. But error exit code always be 1, so in order to specify what exception is occurred, we must parse stdout and stderr. This is what I was trying to do with this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347692#comment-14347692
]
ASF GitHub Bot commented on STORM-695:
--
Github user Lewuathe commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-77264323
@miguno Thank you for comment. I tested the error handling and exit code of
current storm command. And I found when any exception has occurred
`check_output` returns non-zero code(just only 1). So with this patch, storm
command can detect the occurrence of exception. But error exit code always be
1, so in order to specify what exception is occurred, we must parse stdout and
stderr. This is what I was trying to do with this PR.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:kill wait-am
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347674#comment-14347674
]
ASF GitHub Bot commented on STORM-695:
--
Github user Lewuathe commented on a diff in the pull request:
https://github.com/apache/storm/pull/456#discussion_r25820615
--- Diff: bin/storm ---
@@ -183,7 +209,16 @@ def exec_storm_class(klass, jvmtype="-server",
jvmopts=[], extrajars=[], args=[]
os.spawnvp(os.P_WAIT, JAVA_CMD, all_args)
else:
# handling whitespaces in JAVA_CMD
-sub.call(all_args)
+try:
+ret = sub.check_output(all_args, stderr=sub.STDOUT)
+print(ret)
--- End diff --
Because `check_output` absolutes all outout from command. It is necessary
to `print` alternatively to see stdout.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
[jira] [Assigned] (STORM-477) Incorrectly set JAVA_HOME is not detected
[ https://issues.apache.org/jira/browse/STORM-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Ostashchuk reassigned STORM-477: - Assignee: Oleg Ostashchuk > Incorrectly set JAVA_HOME is not detected > - > > Key: STORM-477 > URL: https://issues.apache.org/jira/browse/STORM-477 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.2-incubating > Environment: Rhel6 >Reporter: Paul Poulosky >Assignee: Oleg Ostashchuk >Priority: Minor > Labels: newbie > > If JAVA_HOME is incorrectly set in a user's environment when launching storm, > > it fails with an error message that is confusing to end users. > Traceback (most recent call last): > File "/home/y/bin/storm", line 485, in > main() > File "/home/y/bin/storm", line 482, in main > (COMMANDS.get(COMMAND, unknown_command))(*ARGS) > File "/home/y/bin/storm", line 225, in listtopos > extrajars=[USER_CONF_DIR, STORM_DIR + "/bin"]) > File "/home/y/bin/storm", line 153, in exec_storm_class > ] + jvmopts + [klass] + list(args) > File "/home/y/bin/storm", line 97, in confvalue > p = sub.Popen(command, stdout=sub.PIPE) > File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__ > errread, errwrite) > File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child > raise child_exception > It would be nice if this were either detected and a proper error message > printed, or if it warned and fell back to the java found in PATH. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-532:Supervisor should restart worker imm...
Github user caofangkun commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-77296982 @kishorvpatil I have resolved merge conflicts , please review the PR again. Thank you . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-532) Supervisor should restart worker immediately, if the worker process does not exist any more
[ https://issues.apache.org/jira/browse/STORM-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348036#comment-14348036 ] ASF GitHub Bot commented on STORM-532: -- Github user caofangkun commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-77296982 @kishorvpatil I have resolved merge conflicts , please review the PR again. Thank you . > Supervisor should restart worker immediately, if the worker process does not > exist any more > > > Key: STORM-532 > URL: https://issues.apache.org/jira/browse/STORM-532 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > > For now > if the worker process does not exist any more > Supervisor will have to wait a few seconds for worker heartbeart timeout and > restart worker . > If supervisor knows the worker processid and check if the process exists in > the sync-processes thread ,may need less time to restart worker. > 1: record worker process id in the worker local heartbeart > 2: in supervisor sync-processes ,get process id from worker local heartbeat > and check if the process exits > 3: if not restart it immediately -- This message was sent by Atlassian JIRA (v6.3.4#6332)
