[jira] [Commented] (STORM-702) Apache Exhibitor support
[ https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358036#comment-14358036 ] ASF GitHub Bot commented on STORM-702: -- Github user atdixon commented on the pull request: https://github.com/apache/storm/pull/432#issuecomment-78415948 Note: I've added a short bit on design options and why I recommend this implementation in a comment attached to the JIRA ticket. > Apache Exhibitor support > > > Key: STORM-702 > URL: https://issues.apache.org/jira/browse/STORM-702 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Dixon >Priority: Critical > > Storm connects to Zookeeper via an explicit list of zookeeper hosts. > Apache Exhibitor offers zookeeper cluster discovery and management features > allowing for dynamically resizing zookeeper clusters, restarting and > replacing zk machines in the cluster, etc. > Curator supports creating zookeeper client connections using exhibitor hosts. > Storm should allow (optionally) for connection to its Zookeeper clusters > through Exhibitor. > Here is the github pull request that prompted this ticket, and that > introduces optional exhibitor configuration and exhibitor support in Storm: > https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-702) Apache Exhibitor support
[ https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358034#comment-14358034 ] Aaron Dixon edited comment on STORM-702 at 3/12/15 3:02 AM: A couple of notes on some of the code/design decisions in the current pull request. By supporting exhibitor, there will necessarily be two ways to configure Storm's connection to a Zookeeper cluster. Either you will configure a direct connection to zookeeper hosts or you will configure a connection to exhibitor host(s). I see two ways to offer this up from a configuration perspective: (1) You have *separate* connection properties for exhibitor servers and port that parallel those configuration options for zookeeper. This is how I chose to implement it in the current pull request. The properties are "storm.exhibitor.servers", "storm.exhibitor.port", "storm.exhibitor.poll.millis", and a few more.) In this case, when the exhibitor servers/port are provided, the "storm.zookeeper.servers" and "storm.zookeeper.port" configuration properties are not required but if present are used as backups in the exhibitor-configured client. This of course is backwards-compatible. All existing storm configurations that configure zookeeper directly will work as usual. Exhibitor is only used if configured. (2) Another design option would be to try to overload the existing "storm.zookeeper.servers" and "storm.zookeeper.port" properties by using a scheme for hosts. So for example you might indicate exhibitor servers via a configuration like so: storm.zookeeper.servers: - "exhibitor://127.0.0.1" I think this is less desirable for a number of reasons. I think it is less explicit, less conventional. Also there will be always still be a need for "storm.exhibitor.*" properties to configure specific connection details unique to an exhibitor connection. was (Author: atdixon): A couple of notes on some of the code/design decisions in the current pull request. By supporting exhibitor, there will necessarily be two ways to configure Storm's connection to a Zookeeper cluster. Either you will configure a direct connection to zookeeper hosts or you will configure a connection to exhibitor host(s). I see two ways to offer this up from a configuration perspective: (1) You have *separate* connection properties for exhibitor servers and port that parallel those configuration options for zookeeper. This is how I chose to implement it in the current pull request. The properties are "storm.exhibitor.servers", "storm.exhibitor.port", "storm.exhibitor.poll.millis", and a few more.) In this case, when the exhibitor servers/port are provided, the "storm.zookeeper.servers" and "storm.zookeeper.port" configuration properties are not required but if present are used as backups in the exhibitor-configured client. This of course is backwards-compatible. All existing storm configurations that configure zookeeper directly will work as usual. Exhibitor is only used if configured. (2) Another design option would be to try to overload the existing "storm.zookeeper.servers" and "storm.zookeeper.port" properties by using a scheme for hosts. So for example you might indicate exhibitor servers via a configuration like so: storm.zookeeper.servers: - "exhibitor://127.0.0.1" I think this is less desirable for a number of reasons. I think it is less explicit, less conventional. Also there will be a need for "storm.exhibitor.*" properties to configure specific connection details unique to an exhibitor connection. > Apache Exhibitor support > > > Key: STORM-702 > URL: https://issues.apache.org/jira/browse/STORM-702 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Dixon >Priority: Critical > > Storm connects to Zookeeper via an explicit list of zookeeper hosts. > Apache Exhibitor offers zookeeper cluster discovery and management features > allowing for dynamically resizing zookeeper clusters, restarting and > replacing zk machines in the cluster, etc. > Curator supports creating zookeeper client connections using exhibitor hosts. > Storm should allow (optionally) for connection to its Zookeeper clusters > through Exhibitor. > Here is the github pull request that prompted this ticket, and that > introduces optional exhibitor configuration and exhibitor support in Storm: > https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-702: Exhibitor support
Github user atdixon commented on the pull request: https://github.com/apache/storm/pull/432#issuecomment-78415948 Note: I've added a short bit on design options and why I recommend this implementation in a comment attached to the JIRA ticket. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-702) Apache Exhibitor support
[ https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358034#comment-14358034 ] Aaron Dixon commented on STORM-702: --- A couple of notes on some of the code/design decisions in the current pull request. By supporting exhibitor, there will be necessarily be two ways to configure Storm's connection to a Zookeeper cluster. Either you will configure a direct connection to zookeeper hosts or you will configure a connection to exhibitor host(s). I see two ways to offer this up from a configuration perspective: (1) You have *separate* connection properties for exhibitor servers and port that parallel those configuration options for zookeeper. This is how I chose to implement it in the current pull request. The properties are "storm.exhibitor.servers", "storm.exhibitor.port", "storm.exhibitor.poll.millis", and a few more.) In this case, when the exhibitor servers/port are provided, the "storm.zookeeper.servers" and "storm.zookeeper.port" configuration properties are not required but if present are used as backups in the exhibitor-configured client. This of course is backwards-compatible. All existing storm configurations that configure zookeeper directly will work as usual. Exhibitor is only used if configured. (2) Another design option would be to try to overload the existing "storm.zookeeper.servers" and "storm.zookeeper.port" properties by using a scheme for hosts. So for example you might indicate exhibitor servers via a configuration like so: storm.zookeeper.servers: - "exhibitor://127.0.0.1" I think this is less desirable for a number of reasons. I think it is less explicit, less conventional. Also there will be a need for "storm.exhibitor.*" properties to configure specific connection details unique to an exhibitor connection. > Apache Exhibitor support > > > Key: STORM-702 > URL: https://issues.apache.org/jira/browse/STORM-702 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Dixon >Priority: Critical > > Storm connects to Zookeeper via an explicit list of zookeeper hosts. > Apache Exhibitor offers zookeeper cluster discovery and management features > allowing for dynamically resizing zookeeper clusters, restarting and > replacing zk machines in the cluster, etc. > Curator supports creating zookeeper client connections using exhibitor hosts. > Storm should allow (optionally) for connection to its Zookeeper clusters > through Exhibitor. > Here is the github pull request that prompted this ticket, and that > introduces optional exhibitor configuration and exhibitor support in Storm: > https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-702) Apache Exhibitor support
[ https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358034#comment-14358034 ] Aaron Dixon edited comment on STORM-702 at 3/12/15 3:01 AM: A couple of notes on some of the code/design decisions in the current pull request. By supporting exhibitor, there will necessarily be two ways to configure Storm's connection to a Zookeeper cluster. Either you will configure a direct connection to zookeeper hosts or you will configure a connection to exhibitor host(s). I see two ways to offer this up from a configuration perspective: (1) You have *separate* connection properties for exhibitor servers and port that parallel those configuration options for zookeeper. This is how I chose to implement it in the current pull request. The properties are "storm.exhibitor.servers", "storm.exhibitor.port", "storm.exhibitor.poll.millis", and a few more.) In this case, when the exhibitor servers/port are provided, the "storm.zookeeper.servers" and "storm.zookeeper.port" configuration properties are not required but if present are used as backups in the exhibitor-configured client. This of course is backwards-compatible. All existing storm configurations that configure zookeeper directly will work as usual. Exhibitor is only used if configured. (2) Another design option would be to try to overload the existing "storm.zookeeper.servers" and "storm.zookeeper.port" properties by using a scheme for hosts. So for example you might indicate exhibitor servers via a configuration like so: storm.zookeeper.servers: - "exhibitor://127.0.0.1" I think this is less desirable for a number of reasons. I think it is less explicit, less conventional. Also there will be a need for "storm.exhibitor.*" properties to configure specific connection details unique to an exhibitor connection. was (Author: atdixon): A couple of notes on some of the code/design decisions in the current pull request. By supporting exhibitor, there will be necessarily be two ways to configure Storm's connection to a Zookeeper cluster. Either you will configure a direct connection to zookeeper hosts or you will configure a connection to exhibitor host(s). I see two ways to offer this up from a configuration perspective: (1) You have *separate* connection properties for exhibitor servers and port that parallel those configuration options for zookeeper. This is how I chose to implement it in the current pull request. The properties are "storm.exhibitor.servers", "storm.exhibitor.port", "storm.exhibitor.poll.millis", and a few more.) In this case, when the exhibitor servers/port are provided, the "storm.zookeeper.servers" and "storm.zookeeper.port" configuration properties are not required but if present are used as backups in the exhibitor-configured client. This of course is backwards-compatible. All existing storm configurations that configure zookeeper directly will work as usual. Exhibitor is only used if configured. (2) Another design option would be to try to overload the existing "storm.zookeeper.servers" and "storm.zookeeper.port" properties by using a scheme for hosts. So for example you might indicate exhibitor servers via a configuration like so: storm.zookeeper.servers: - "exhibitor://127.0.0.1" I think this is less desirable for a number of reasons. I think it is less explicit, less conventional. Also there will be a need for "storm.exhibitor.*" properties to configure specific connection details unique to an exhibitor connection. > Apache Exhibitor support > > > Key: STORM-702 > URL: https://issues.apache.org/jira/browse/STORM-702 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Dixon >Priority: Critical > > Storm connects to Zookeeper via an explicit list of zookeeper hosts. > Apache Exhibitor offers zookeeper cluster discovery and management features > allowing for dynamically resizing zookeeper clusters, restarting and > replacing zk machines in the cluster, etc. > Curator supports creating zookeeper client connections using exhibitor hosts. > Storm should allow (optionally) for connection to its Zookeeper clusters > through Exhibitor. > Here is the github pull request that prompted this ticket, and that > introduces optional exhibitor configuration and exhibitor support in Storm: > https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-703) RedisMapState with hash key can cause network overload
[ https://issues.apache.org/jira/browse/STORM-703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357837#comment-14357837 ] ASF GitHub Bot commented on STORM-703: -- Github user dkatten commented on the pull request: https://github.com/apache/storm/pull/462#issuecomment-78398611 STORM-703 https://issues.apache.org/jira/browse/STORM-703 > RedisMapState with hash key can cause network overload > -- > > Key: STORM-703 > URL: https://issues.apache.org/jira/browse/STORM-703 > Project: Apache Storm > Issue Type: Bug > Components: external >Affects Versions: 0.10.0 >Reporter: Dave Katten > > When RedisMapState is constructed to use a hash key (and store values under a > hash in redis, rather than as key/value pairs in the top namespace), a > multiGet actually requests ALL of the state in that hash, instead of only the > keys in the batch. > As the size of the hash grows in redis, this becomes an inordinate amount of > traffic, and can cause interfaces to fall over. > * Solution > Instead of calling jedis.hgetall(hash), call jedis.hmget(hash, stringKeys[]). > Also, remove the buildValuesFromMap function, as it is no longer needed. > See PR#462 on github. > https://github.com/apache/storm/pull/462 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: With hash key option for RedisMapState, only g...
Github user dkatten commented on the pull request: https://github.com/apache/storm/pull/462#issuecomment-78398611 STORM-703 https://issues.apache.org/jira/browse/STORM-703 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-703) RedisMapState with hash key can cause network overload
Dave Katten created STORM-703: - Summary: RedisMapState with hash key can cause network overload Key: STORM-703 URL: https://issues.apache.org/jira/browse/STORM-703 Project: Apache Storm Issue Type: Bug Components: external Affects Versions: 0.10.0 Reporter: Dave Katten When RedisMapState is constructed to use a hash key (and store values under a hash in redis, rather than as key/value pairs in the top namespace), a multiGet actually requests ALL of the state in that hash, instead of only the keys in the batch. As the size of the hash grows in redis, this becomes an inordinate amount of traffic, and can cause interfaces to fall over. * Solution Instead of calling jedis.hgetall(hash), call jedis.hmget(hash, stringKeys[]). Also, remove the buildValuesFromMap function, as it is no longer needed. See PR#462 on github. https://github.com/apache/storm/pull/462 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: With hash key option for RedisMapState, only g...
Github user dkatten commented on the pull request: https://github.com/apache/storm/pull/462#issuecomment-78397688 I'll file it. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357827#comment-14357827
]
ASF GitHub Bot commented on STORM-695:
--
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-78397516
IMHO I agree @revans2, parsing stdout by regular expression would become
easily broken by later commits.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
> )]
> (transition-name! nimbus storm-name [:kill wait-amt] true)
> ))
> {code}
> As you can see the current implementation does not pass success/failure
> information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-78397516 IMHO I agree @revans2, parsing stdout by regular expression would become easily broken by later commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-691) [storm-redis] Add basic lookup / persist bolts
[ https://issues.apache.org/jira/browse/STORM-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357817#comment-14357817 ] ASF GitHub Bot commented on STORM-691: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/451#issuecomment-78396550 @revans2 Thanks for reviewing! It's updated. Could you please check again? > [storm-redis] Add basic lookup / persist bolts > -- > > Key: STORM-691 > URL: https://issues.apache.org/jira/browse/STORM-691 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Currently storm-redis provides AbstractRedisBolt for normal (not Trident) > Bolt. > Jedis is easy to use so it may be enough, but we can also provide > implementations of AbstractRedisBolt for simple usage. > eg. store (key, value) pair, get key's value > Since Redis has various data types and commands, we can't cover whole things, > but seems like below things could be considered. > || Type || Read || Write || > | STRING | GET (key) | SET (key, value) | > | HASH | HGET (key, field) | HSET (key, field, value) | > | LIST | LPOP (key) | RPUSH (key, value) | > | SET | SCARD (key) | SADD (key, member) | > | SORTED SET | ZSCORE (key, member) | ZADD (key, score, member) | > | HLL (HyperLogLog) | PFCOUNT (key) | PFADD (key, element) | > Btw, since we will normally get key & value from tuple (as most external > module did), HASH, SET, SORTED SET needs additional key to process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-691 Add basic lookup / persist bolts
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/451#issuecomment-78396550 @revans2 Thanks for reviewing! It's updated. Could you please check again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-689) SimpleACLAuthorizer should provide a way to restrict who can submit topologies
[ https://issues.apache.org/jira/browse/STORM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357803#comment-14357803 ] ASF GitHub Bot commented on STORM-689: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/445#issuecomment-78395668 Sorry it took so long for the review. It looks good, and I agree having groups of allowed users would be a good step. > SimpleACLAuthorizer should provide a way to restrict who can submit topologies > -- > > Key: STORM-689 > URL: https://issues.apache.org/jira/browse/STORM-689 > Project: Apache Storm > Issue Type: Improvement >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani >Priority: Trivial > > SimpleACLAuthorizer currently allows anyone with a valid kerberos ticket to > submit topologies. There are cases where storm admins want to allow only > selected users to submit topologies. I am proposing nimbus.users config > option if its added to storm.yaml only the listed users can deploy the storm > topologies. > cc [~revans2] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/445#issuecomment-78395668 Sorry it took so long for the review. It looks good, and I agree having groups of allowed users would be a good step. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-689) SimpleACLAuthorizer should provide a way to restrict who can submit topologies
[
https://issues.apache.org/jira/browse/STORM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357798#comment-14357798
]
ASF GitHub Bot commented on STORM-689:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/445#discussion_r26266419
--- Diff:
storm-core/src/jvm/backtype/storm/security/auth/authorizer/SimpleACLAuthorizer.java
---
@@ -96,7 +101,10 @@ public boolean permit(ReqContext context, String
operation, Map topology_conf) {
}
if (_userCommands.contains(operation)) {
-return true;
+if (_nimbusUsers.size() > 0 && _nimbusUsers.contains(user))
+return true;
+else if (_nimbusUsers.size() == 0)
+return true;
--- End diff --
I personally think
```
return _nimbusUsers.size() == 0 || _nimbusUsers.contains(user);
```
is more readable than what is here, but this is really minor.
> SimpleACLAuthorizer should provide a way to restrict who can submit topologies
> --
>
> Key: STORM-689
> URL: https://issues.apache.org/jira/browse/STORM-689
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
>Priority: Trivial
>
> SimpleACLAuthorizer currently allows anyone with a valid kerberos ticket to
> submit topologies. There are cases where storm admins want to allow only
> selected users to submit topologies. I am proposing nimbus.users config
> option if its added to storm.yaml only the listed users can deploy the storm
> topologies.
> cc [~revans2]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/445#discussion_r26266419
--- Diff:
storm-core/src/jvm/backtype/storm/security/auth/authorizer/SimpleACLAuthorizer.java
---
@@ -96,7 +101,10 @@ public boolean permit(ReqContext context, String
operation, Map topology_conf) {
}
if (_userCommands.contains(operation)) {
-return true;
+if (_nimbusUsers.size() > 0 && _nimbusUsers.contains(user))
+return true;
+else if (_nimbusUsers.size() == 0)
+return true;
--- End diff --
I personally think
```
return _nimbusUsers.size() == 0 || _nimbusUsers.contains(user);
```
is more readable than what is here, but this is really minor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Resolved] (STORM-696) Single Namespace Test Launching
[ https://issues.apache.org/jira/browse/STORM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-696. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~knusbaum] > Single Namespace Test Launching > --- > > Key: STORM-696 > URL: https://issues.apache.org/jira/browse/STORM-696 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum >Priority: Minor > Fix For: 0.10.0 > > > As a dev I would like to be able to run unit tests in a particular namespace. > Right now, we have to run the full test suite through maven. If we want to > test particular namespaces, we must do so from a REPL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-696) Single Namespace Test Launching
[ https://issues.apache.org/jira/browse/STORM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357792#comment-14357792 ] ASF GitHub Bot commented on STORM-696: -- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/450#issuecomment-78394408 Sweet! > Single Namespace Test Launching > --- > > Key: STORM-696 > URL: https://issues.apache.org/jira/browse/STORM-696 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum >Priority: Minor > > As a dev I would like to be able to run unit tests in a particular namespace. > Right now, we have to run the full test suite through maven. If we want to > test particular namespaces, we must do so from a REPL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-696) Single Namespace Test Launching
[ https://issues.apache.org/jira/browse/STORM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357793#comment-14357793 ] ASF GitHub Bot commented on STORM-696: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/450 > Single Namespace Test Launching > --- > > Key: STORM-696 > URL: https://issues.apache.org/jira/browse/STORM-696 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum >Priority: Minor > > As a dev I would like to be able to run unit tests in a particular namespace. > Right now, we have to run the full test suite through maven. If we want to > test particular namespaces, we must do so from a REPL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-696] Single Namespace Test Launching
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/450 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-696) Single Namespace Test Launching
[ https://issues.apache.org/jira/browse/STORM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357788#comment-14357788 ] ASF GitHub Bot commented on STORM-696: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/450#issuecomment-78394267 Its a bit hacky, but it works! +++1. I'll check this in. > Single Namespace Test Launching > --- > > Key: STORM-696 > URL: https://issues.apache.org/jira/browse/STORM-696 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum >Priority: Minor > > As a dev I would like to be able to run unit tests in a particular namespace. > Right now, we have to run the full test suite through maven. If we want to > test particular namespaces, we must do so from a REPL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-696] Single Namespace Test Launching
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/450#issuecomment-78394408 Sweet! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-696] Single Namespace Test Launching
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/450#issuecomment-78394267 Its a bit hacky, but it works! +++1. I'll check this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-691) [storm-redis] Add basic lookup / persist bolts
[ https://issues.apache.org/jira/browse/STORM-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357778#comment-14357778 ] ASF GitHub Bot commented on STORM-691: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/451#issuecomment-78393165 The code looks fine to me. My only comment is that it would be nice to have a default in most of the case statements, mostly as defensive programming so if something gets off we wold get an error instead of falling through and ignoring the tuples. > [storm-redis] Add basic lookup / persist bolts > -- > > Key: STORM-691 > URL: https://issues.apache.org/jira/browse/STORM-691 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Minor > > Currently storm-redis provides AbstractRedisBolt for normal (not Trident) > Bolt. > Jedis is easy to use so it may be enough, but we can also provide > implementations of AbstractRedisBolt for simple usage. > eg. store (key, value) pair, get key's value > Since Redis has various data types and commands, we can't cover whole things, > but seems like below things could be considered. > || Type || Read || Write || > | STRING | GET (key) | SET (key, value) | > | HASH | HGET (key, field) | HSET (key, field, value) | > | LIST | LPOP (key) | RPUSH (key, value) | > | SET | SCARD (key) | SADD (key, member) | > | SORTED SET | ZSCORE (key, member) | ZADD (key, score, member) | > | HLL (HyperLogLog) | PFCOUNT (key) | PFADD (key, element) | > Btw, since we will normally get key & value from tuple (as most external > module did), HASH, SET, SORTED SET needs additional key to process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-691 Add basic lookup / persist bolts
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/451#issuecomment-78393165 The code looks fine to me. My only comment is that it would be nice to have a default in most of the case statements, mostly as defensive programming so if something gets off we wold get an error instead of falling through and ignoring the tuples. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357767#comment-14357767
]
ASF GitHub Bot commented on STORM-695:
--
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/456#issuecomment-78392307
Why do we want to distinguish between different errors in the exit code?
I would rather see the commands themselves handle this instead of the
python code, the coupling between the script and the java code is too high for
me to feel comfortable relying on this functionality. If I have a java program
that catches the exception and outputs the error differently this code will not
work. What is more the original problem still exists in those cases, and storm
will exit with a 0 exit code on error.
> storm CLI tool reports zero exit code on error scenario, take 2
> ---
>
> Key: STORM-695
> URL: https://issues.apache.org/jira/browse/STORM-695
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.3
>Reporter: Michael Noll
>Assignee: Kai Sasaki
>Priority: Minor
>
> Commands such as "storm kill non-existing-topology" will return an exit code
> of zero, indicating success when in fact the command failed.
> h3. How to reproduce
> Here is but one example where the {{storm}} CLI tool violates shell best
> practices:
> {code}
> # Let's kill a topology that is in fact not running in the cluster.
> $ storm kill i-do-not-exist-topo
>
> Exception in thread "main" NotAliveException(msg:i-do-not-exist-topo is not
> alive)
> # Print the exit code of last command.
> $ echo $?
> 0 # <<< but since the kill command failed this should be non-zero!
> {code}
> Another example is the "storm jar" command. If you attempt to submit a
> topology that has the same name as an existing, running topology, the "storm
> jar" command will not submit the topology -- instead it will print an
> exception (think: "the topology FooName is already running"), which is ok,
> but it will then exit with a return code of zero, which indicates success
> (which is wrong).
> h3. Impact
> This bug prevents automated deployment tools such as Ansible or Puppet as
> well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
> properly because Storm violates shell conventions by not returning non-zero
> exit codes in case of failures.
> h3. How to fix
> From what I understand the solution is two-fold:
> # The various Storm commands that are being called by the {{storm}} script
> must return proper exit codes.
> # The {{storm}} script must store these exit codes and return itself with the
> respective exit code of the Storm command it actually ran.
> For example, here's the current code that implements the "storm kill" command:
> {code}
> # In: bin/storm
> def kill(*args):
> """Syntax: [storm kill topology-name [-w wait-time-secs]]
> Kills the topology with the name topology-name. Storm will
> first deactivate the topology's spouts for the duration of
> the topology's message timeout to allow all messages currently
> being processed to finish processing. Storm will then shutdown
> the workers and clean up their state. You can override the length
> of time Storm waits between deactivation and shutdown with the -w flag.
> """
> exec_storm_class(
> "backtype.storm.command.kill_topology",
> args=args,
> jvmtype="-client",
> extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
> {code}
> which in turn calls the following code in {{kill_topology.clj}}:
> {code}
> ;; In: backtype.storm.command.kill-topology
> (ns backtype.storm.command.kill-topology
> (:use [clojure.tools.cli :only [cli]])
> (:use [backtype.storm thrift config log])
> (:import [backtype.storm.generated KillOptions])
> (:gen-class))
> (defn -main [& args]
> (let [[{wait :wait} [name] _] (cli args ["-w" "--wait" :default nil
> :parse-fn #(Integer/parseInt %)])
> opts (KillOptions.)]
> (if wait (.set_wait_secs opts wait))
> (with-configured-nimbus-connection nimbus
> (.killTopologyWithOpts nimbus name opts)
> (log-message "Killed topology: " name)
> )))
> {code}
> which in turn calls the following code in {{nimbus.clj}}:
> {code}
> ;; In: backtype.storm.daemon.nimbus
> (^void killTopologyWithOpts [this ^String storm-name ^KillOptions
> options]
> (check-storm-active! nimbus storm-name true)
> (let [topology-conf (try-read-storm-conf-from-name conf storm-name
> nimbus)]
> (check-authorization! nimbus storm-name topology-conf
> "killTopology"))
> (let [wait-amt (if (.is_set_wait_secs options)
> (.get_wait_secs options)
>
[GitHub] storm pull request: [STORM-695] storm CLI tool reports zero exit c...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/456#issuecomment-78392307 Why do we want to distinguish between different errors in the exit code? I would rather see the commands themselves handle this instead of the python code, the coupling between the script and the java code is too high for me to feel comfortable relying on this functionality. If I have a java program that catches the exception and outputs the error differently this code will not work. What is more the original problem still exists in those cases, and storm will exit with a 0 exit code on error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: With hash key option for RedisMapState, only g...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/462#issuecomment-78390669 The change looks good to me. I am +1 for the change. @dkatten if you want to file a JIRA under STORM at issues.apache.org that would be great. If you would rather I can file it for you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-477) Incorrectly set JAVA_HOME is not detected
[
https://issues.apache.org/jira/browse/STORM-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357745#comment-14357745
]
ASF GitHub Bot commented on STORM-477:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/457#discussion_r26263810
--- Diff: bin/storm ---
@@ -82,6 +82,8 @@ CONFFILE = ""
JAR_JVM_OPTS = shlex.split(os.getenv('STORM_JAR_JVM_OPTS', ''))
JAVA_HOME = os.getenv('JAVA_HOME', None)
JAVA_CMD = 'java' if not JAVA_HOME else os.path.join(JAVA_HOME, 'bin',
'java')
+if JAVA_HOME and not os.path.exists(JAVA_CMD):
+print "WARNING: Check JAVA_HOME. Could not find java at %s." %
JAVA_HOME
--- End diff --
I personally would rather have storm exit with an error here instead of
just a warning.
> Incorrectly set JAVA_HOME is not detected
> -
>
> Key: STORM-477
> URL: https://issues.apache.org/jira/browse/STORM-477
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.9.2-incubating
> Environment: Rhel6
>Reporter: Paul Poulosky
>Assignee: Oleg Ostashchuk
>Priority: Minor
> Labels: newbie
>
> If JAVA_HOME is incorrectly set in a user's environment when launching storm,
>
> it fails with an error message that is confusing to end users.
> Traceback (most recent call last):
> File "/home/y/bin/storm", line 485, in
> main()
> File "/home/y/bin/storm", line 482, in main
> (COMMANDS.get(COMMAND, unknown_command))(*ARGS)
> File "/home/y/bin/storm", line 225, in listtopos
> extrajars=[USER_CONF_DIR, STORM_DIR + "/bin"])
> File "/home/y/bin/storm", line 153, in exec_storm_class
> ] + jvmopts + [klass] + list(args)
> File "/home/y/bin/storm", line 97, in confvalue
> p = sub.Popen(command, stdout=sub.PIPE)
> File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
> errread, errwrite)
> File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
> raise child_exception
> It would be nice if this were either detected and a proper error message
> printed, or if it warned and fell back to the java found in PATH.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-477] Add warning for invalid JAVA_HOME
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/457#discussion_r26263810
--- Diff: bin/storm ---
@@ -82,6 +82,8 @@ CONFFILE = ""
JAR_JVM_OPTS = shlex.split(os.getenv('STORM_JAR_JVM_OPTS', ''))
JAVA_HOME = os.getenv('JAVA_HOME', None)
JAVA_CMD = 'java' if not JAVA_HOME else os.path.join(JAVA_HOME, 'bin',
'java')
+if JAVA_HOME and not os.path.exists(JAVA_CMD):
+print "WARNING: Check JAVA_HOME. Could not find java at %s." %
JAVA_HOME
--- End diff --
I personally would rather have storm exit with an error here instead of
just a warning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: added missing @Override annotations
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/460#issuecomment-78388527 @krichter722 thanks for the pull request, if you could remove the changes you made to the generated thrift code the rest seems fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Client (Netty): improving logging to help trou...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/463#issuecomment-78388285 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-702) Apache Exhibitor support
[ https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357723#comment-14357723 ] ASF GitHub Bot commented on STORM-702: -- Github user atdixon commented on the pull request: https://github.com/apache/storm/pull/432#issuecomment-78386437 Done, I've added the ticket number to the title. Here is the ticket: https://issues.apache.org/jira/browse/STORM-702 > Apache Exhibitor support > > > Key: STORM-702 > URL: https://issues.apache.org/jira/browse/STORM-702 > Project: Apache Storm > Issue Type: Improvement >Reporter: Aaron Dixon >Priority: Critical > > Storm connects to Zookeeper via an explicit list of zookeeper hosts. > Apache Exhibitor offers zookeeper cluster discovery and management features > allowing for dynamically resizing zookeeper clusters, restarting and > replacing zk machines in the cluster, etc. > Curator supports creating zookeeper client connections using exhibitor hosts. > Storm should allow (optionally) for connection to its Zookeeper clusters > through Exhibitor. > Here is the github pull request that prompted this ticket, and that > introduces optional exhibitor configuration and exhibitor support in Storm: > https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-702: Exhibitor support
Github user atdixon commented on the pull request: https://github.com/apache/storm/pull/432#issuecomment-78386437 Done, I've added the ticket number to the title. Here is the ticket: https://issues.apache.org/jira/browse/STORM-702 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-702) Apache Exhibitor support
Aaron Dixon created STORM-702: - Summary: Apache Exhibitor support Key: STORM-702 URL: https://issues.apache.org/jira/browse/STORM-702 Project: Apache Storm Issue Type: Improvement Reporter: Aaron Dixon Priority: Critical Storm connects to Zookeeper via an explicit list of zookeeper hosts. Apache Exhibitor offers zookeeper cluster discovery and management features allowing for dynamically resizing zookeeper clusters, restarting and replacing zk machines in the cluster, etc. Curator supports creating zookeeper client connections using exhibitor hosts. Storm should allow (optionally) for connection to its Zookeeper clusters through Exhibitor. Here is the github pull request that prompted this ticket, and that introduces optional exhibitor configuration and exhibitor support in Storm: https://github.com/apache/storm/pull/432 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-583: Initiali check-in for storm-eventhu...
Github user shanyu commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-78383657 @ptgoetz It has been a while, is the IP clearance done? Thx --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357700#comment-14357700 ] ASF GitHub Bot commented on STORM-583: -- Github user shanyu commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-78383657 @ptgoetz It has been a while, is the IP clearance done? Thx > Add spout and bolt implementation for Azure Eventhubs > - > > Key: STORM-583 > URL: https://issues.apache.org/jira/browse/STORM-583 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 >Reporter: shanyu zhao >Assignee: shanyu zhao > Fix For: 0.10.0 > > Attachments: Storm-EventHubsDesign.docx > > > Add spout and bolt implementations for Azure Eventhubs - a messaging service > that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to > add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Exhibitor support
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/432#issuecomment-78378359 I like the idea of this, and I am +1 on the concept, but we really need a JIRA for tracking this. Could you please file a JIRA and include the JIRA number in the title of this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-670) [storm-kafka] Restore Java 1.6 compatibility
[ https://issues.apache.org/jira/browse/STORM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357658#comment-14357658 ] ASF GitHub Bot commented on STORM-670: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/431 > [storm-kafka] Restore Java 1.6 compatibility > > > Key: STORM-670 > URL: https://issues.apache.org/jira/browse/STORM-670 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 0.10.0 >Reporter: P. Taylor Goetz >Assignee: P. Taylor Goetz > Fix For: 0.10.0 > > > java.lang.Long.compare(Long, Long) is only available in Java 1.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-670: restore java 1.6 compatibility (sto...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/431 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-670) [storm-kafka] Restore Java 1.6 compatibility
[ https://issues.apache.org/jira/browse/STORM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357654#comment-14357654 ] ASF GitHub Bot commented on STORM-670: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/431#issuecomment-78377210 +1 > [storm-kafka] Restore Java 1.6 compatibility > > > Key: STORM-670 > URL: https://issues.apache.org/jira/browse/STORM-670 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 0.10.0 >Reporter: P. Taylor Goetz >Assignee: P. Taylor Goetz > > java.lang.Long.compare(Long, Long) is only available in Java 1.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-670: restore java 1.6 compatibility (sto...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/431#issuecomment-78377210 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (STORM-693) KafkaBolt exception handling improvement
[ https://issues.apache.org/jira/browse/STORM-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rick Kellogg closed STORM-693. -- Resolution: Fixed Fix Version/s: 0.9.4 Confirmed change is within master branch. > KafkaBolt exception handling improvement > > > Key: STORM-693 > URL: https://issues.apache.org/jira/browse/STORM-693 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 0.9.3 >Reporter: Rick Kellogg >Assignee: Parth Brahmbhatt > Fix For: 0.9.4 > > > Within the KafkaBolt execute method, an error message is logged if any sort > of error occurs communicating with Kafka. Unfortunately the input is still > acknowledged. > Upon review of the HdfsBolt & HiveBolt, I believe the exception handling > block should include the following two lines: > this.collector.reportError(ex); > this.collector.fail(input); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0
Sorry for the late reply. I am +1 on releasing 0.9.4 . For 0.10.0 I would like to include this https://issues.apache.org/jira/browse/STORM-617 Thanks, Harsha On March 11, 2015 at 1:52:02 PM, P. Taylor Goetz (ptgo...@gmail.com) wrote: Sorry for the late reply. I am +1 on releasing 0.9.4 . For 0.10.0 I would like include this https://issues.apache.org/jira/browse/STORM-617 Thanks, Harsha
[jira] [Resolved] (STORM-539) Storm hive bolt and trident state
[ https://issues.apache.org/jira/browse/STORM-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani resolved STORM-539. -- Resolution: Fixed > Storm hive bolt and trident state > -- > > Key: STORM-539 > URL: https://issues.apache.org/jira/browse/STORM-539 > Project: Apache Storm > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > > Hive 0.13 provides streaming api capability to write to bucketed orc table. > Storm should've have connector(bolt,trident) to write data directly into hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-669) Replace links with ones to latest api document
[ https://issues.apache.org/jira/browse/STORM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-669. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks again [~lewuathe], I merged this into master. > Replace links with ones to latest api document > -- > > Key: STORM-669 > URL: https://issues.apache.org/jira/browse/STORM-669 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0, 0.9.3-rc2 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Minor > Labels: documentaion, javadoc > Fix For: 0.10.0 > > > Replace links to old api document with new ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-669) Replace links with ones to latest api document
[ https://issues.apache.org/jira/browse/STORM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357580#comment-14357580 ] ASF GitHub Bot commented on STORM-669: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/425 > Replace links with ones to latest api document > -- > > Key: STORM-669 > URL: https://issues.apache.org/jira/browse/STORM-669 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0, 0.9.3-rc2 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Minor > Labels: documentaion, javadoc > > Replace links to old api document with new ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-669) Replace links with ones to latest api document
[ https://issues.apache.org/jira/browse/STORM-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357578#comment-14357578 ] ASF GitHub Bot commented on STORM-669: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/425#issuecomment-78368904 +1 > Replace links with ones to latest api document > -- > > Key: STORM-669 > URL: https://issues.apache.org/jira/browse/STORM-669 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0, 0.9.3-rc2 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Minor > Labels: documentaion, javadoc > > Replace links to old api document with new ones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-669] Replace links with ones to latest ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/425 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-669] Replace links with ones to latest ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/425#issuecomment-78368904 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0
Thanks for the feedback Richard, Parth, and Jungtaek. I’ve made a note of your suggestions for the releases. Does anyone else have any thoughts (esp. committers)? Should we move forward with releasing? -Taylor On Mar 5, 2015, at 5:00 PM, 임정택 wrote: > storm-redis (scheduled to be released at 0.10.0) has one bugfix and one > essential feature PRs. > > - bugfix: https://issues.apache.org/jira/browse/STORM-690 > -- It fixes connection pool issue. > - feature: https://issues.apache.org/jira/browse/STORM-691 > -- It provides basic lookup / persist bolts so I believe it's necessary. > > Furthermore, I'd like to continue to support various data types with > storm-redis Trident, after STORM-691 is merged to master. > > Thanks! > > Regards > Jungtaek Lim (HeartSaVioR) > > > 2015-03-06 2:37 GMT+09:00 P. Taylor Goetz : > >> I’d like to start a discussion for releasing 0.9.4 (maintenance release) >> and 0.10.0 (security release). >> >> 0.9.4 is basically a branch of 0.9.3 with two important bug fixes: >> >>STORM-329: fix cascading Storm failure by improving reconnection >> strategy and buffering messages >>STORM-130: Supervisor getting killed due to >> java.io.FileNotFoundException: File '../stormconf.ser' does not exist. >> >> Both are long-standing bugs that have proven problematic for many users. >> I’d be in favor of releasing 0.9.4 with just those two fixes, but I’m >> interested in finding out if anyone thinks there are additional patches to >> master that should be considered for 0.9.4. >> >> 0.10.0 is a much larger release in terms of changes. In addition to the >> changes above, it includes all the new security features and numerous fixes >> and enhancements (see the CHANGELOG in the master branch for a full list). >> >> Do we feel 0.10.0 is ready for release? If not what outstanding >> bugs/patches should we consider before releasing? >> >> I’m fine holding off on a 0.10.0 release if we feel there is additional >> work to be done, but I’d like to at least move forward with 0.9.4 release. >> >> Thoughts? >> >> -Taylor >> > > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior signature.asc Description: Message signed with OpenPGP using GPGMail
[jira] [Resolved] (STORM-690) [storm-redis] Broken Jedis Connection should be returned as broken
[ https://issues.apache.org/jira/browse/STORM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-690. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~kabhwan], I merged this into master. > [storm-redis] Broken Jedis Connection should be returned as broken > --- > > Key: STORM-690 > URL: https://issues.apache.org/jira/browse/STORM-690 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > Fix For: 0.10.0 > > > While using JedisPool, any broken Jedis instances should be returned as > broken. > (Thanks to Apache Commons Pool 2, we can delegate verifying instances to > JedisPool itself by configuration, but performance will be decreased a bit.) > We can try-with-finally with Jedis / JedisCluster instance, but Storm should > be run at least JDK 6 so we can't apply it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-690) [storm-redis] Broken Jedis Connection should be returned as broken
[ https://issues.apache.org/jira/browse/STORM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357570#comment-14357570 ] ASF GitHub Bot commented on STORM-690: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/444 > [storm-redis] Broken Jedis Connection should be returned as broken > --- > > Key: STORM-690 > URL: https://issues.apache.org/jira/browse/STORM-690 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > While using JedisPool, any broken Jedis instances should be returned as > broken. > (Thanks to Apache Commons Pool 2, we can delegate verifying instances to > JedisPool itself by configuration, but performance will be decreased a bit.) > We can try-with-finally with Jedis / JedisCluster instance, but Storm should > be run at least JDK 6 so we can't apply it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-690) [storm-redis] Broken Jedis Connection should be returned as broken
[ https://issues.apache.org/jira/browse/STORM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357563#comment-14357563 ] ASF GitHub Bot commented on STORM-690: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/444#issuecomment-78366813 The change looks fine to me +1. > [storm-redis] Broken Jedis Connection should be returned as broken > --- > > Key: STORM-690 > URL: https://issues.apache.org/jira/browse/STORM-690 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > While using JedisPool, any broken Jedis instances should be returned as > broken. > (Thanks to Apache Commons Pool 2, we can delegate verifying instances to > JedisPool itself by configuration, but performance will be decreased a bit.) > We can try-with-finally with Jedis / JedisCluster instance, but Storm should > be run at least JDK 6 so we can't apply it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-690 Return Jedis into JedisPool with mar...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/444 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-690 Return Jedis into JedisPool with mar...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/444#issuecomment-78366813 The change looks fine to me +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-663) Create javadocs for BoltDeclarer
[ https://issues.apache.org/jira/browse/STORM-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357531#comment-14357531 ] ASF GitHub Bot commented on STORM-663: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/424#issuecomment-78363934 +1 looks good > Create javadocs for BoltDeclarer > > > Key: STORM-663 > URL: https://issues.apache.org/jira/browse/STORM-663 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 0.9.3 >Reporter: Karl Richter >Assignee: Kai Sasaki > Fix For: 0.10.0 > > > There's no documentation of grouping stream usage in the javadocs of > BoltDeclarerhttp://storm.apache.org/javadoc/apidocs/backtype/storm/topology/BoltDeclarer.html. > It might be sufficient to a links to sites or HTML anchors of the full text > documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-663) Create javadocs for BoltDeclarer
[ https://issues.apache.org/jira/browse/STORM-663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-663. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~lewuathe], I merged this into master. Keep up the good work. > Create javadocs for BoltDeclarer > > > Key: STORM-663 > URL: https://issues.apache.org/jira/browse/STORM-663 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 0.9.3 >Reporter: Karl Richter >Assignee: Kai Sasaki > Fix For: 0.10.0 > > > There's no documentation of grouping stream usage in the javadocs of > BoltDeclarerhttp://storm.apache.org/javadoc/apidocs/backtype/storm/topology/BoltDeclarer.html. > It might be sufficient to a links to sites or HTML anchors of the full text > documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-663) Create javadocs for BoltDeclarer
[ https://issues.apache.org/jira/browse/STORM-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357530#comment-14357530 ] ASF GitHub Bot commented on STORM-663: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/424 > Create javadocs for BoltDeclarer > > > Key: STORM-663 > URL: https://issues.apache.org/jira/browse/STORM-663 > Project: Apache Storm > Issue Type: Documentation >Affects Versions: 0.9.3 >Reporter: Karl Richter >Assignee: Kai Sasaki > > There's no documentation of grouping stream usage in the javadocs of > BoltDeclarerhttp://storm.apache.org/javadoc/apidocs/backtype/storm/topology/BoltDeclarer.html. > It might be sufficient to a links to sites or HTML anchors of the full text > documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-663] Create javadocs for BoltDeclarer
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/424 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-663] Create javadocs for BoltDeclarer
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/424#issuecomment-78363934 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.
[
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357525#comment-14357525
]
ASF GitHub Bot commented on STORM-634:
--
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/414#discussion_r26251585
--- Diff: storm-core/src/storm.thrift ---
@@ -193,6 +193,7 @@ struct ExecutorStats {
1: required map> emitted;
2: required map> transferred;
3: required ExecutorSpecificStats specific;
+ 4: required double rate;
--- End diff --
Older client newer server works, newer client with older server brakes
because rate is required behavior. I think this is acceptable and would hate to
change a required param to optional to support this.
> Storm should support rolling upgrade/downgrade of storm cluster.
>
>
> Key: STORM-634
> URL: https://issues.apache.org/jira/browse/STORM-634
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>
> Currently when a new version of storm is released in order to upgrade
> existing storm clusters users need to backup their existing topologies , kill
> all the topologies , perform the upgrade and resubmit all the topologies.
> This is painful and results in downtime which may not be acceptable for
> "Always alive" production systems.
> Storm should support a rolling upgrade/downgrade deployment process to avoid
> these downtimes and to make the transition to a different version effortless.
> Based on my initial attempt the primary issue seem to be the java
> serialization used to serialize java classes like StormBase, Assignment,
> WorkerHeartbeat which is then stored in zookeeper. When deserializing if the
> serial versions do not match the deserialization fails resulting in processes
> just getting killed indefinitely. We need to change the Utils/serialize and
> Utils/deserialize so it can support non java serialization mechanism like
> json.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/414#discussion_r26251585
--- Diff: storm-core/src/storm.thrift ---
@@ -193,6 +193,7 @@ struct ExecutorStats {
1: required map> emitted;
2: required map> transferred;
3: required ExecutorSpecificStats specific;
+ 4: required double rate;
--- End diff --
Older client newer server works, newer client with older server brakes
because rate is required behavior. I think this is acceptable and would hate to
change a required param to optional to support this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-541) Build produces maven warnings
[
https://issues.apache.org/jira/browse/STORM-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357512#comment-14357512
]
ASF GitHub Bot commented on STORM-541:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/422#discussion_r26250879
--- Diff: pom.xml ---
@@ -521,12 +521,6 @@
test
-org.clojars.runa
-conjure
-${conjure.version}
--- End diff --
The code no longer compiles because this dependency was removed.
dependencyManagement in the pom does not add in a dependency, it just tells
maven if a project needs this dependency use this particular version. Without
this here the remaining dependency in storm-core does not know which version to
use.
> Build produces maven warnings
> -
>
> Key: STORM-541
> URL: https://issues.apache.org/jira/browse/STORM-541
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.3-rc2
> Environment: Ubuntu 14.04 on POWER
>Reporter: jez wain
>Assignee: caofangkun
>Priority: Minor
> Labels: patch
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> The maven compilation task produces the following warnings because version
> numbers are missing from some plugins in the pom.xml file. I have created a
> patch file.
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:maven-shade-clojure-transformer:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:storm-core:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:storm-starter:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:storm-kafka:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:storm-hdfs:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for com.github.ptgoetz:storm-hbase:jar:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661,
> column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:storm:pom:0.9.3-rc2-SNAPSHOT
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ line 661,
> column 21
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> PATCH:
> diff --git a/pom.xml b/pom.xml
> index b63d332..37a5acd 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -656,10 +656,12 @@
>
> org.apache.maven.plugins
> maven-javadoc-plugin
> +2.9
>
>
> org.apache.maven.plugins
> maven-surefire-report-plugin
> + 2.16
>
>
> ${project.build.directory}/test-reports
> @@ -694,6 +696,7 @@
>
> org.apache.maven.plugins
> maven-javadoc-plu
[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.
[ https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357505#comment-14357505 ] ASF GitHub Bot commented on STORM-634: -- Github user Parth-Brahmbhatt commented on a diff in the pull request: https://github.com/apache/storm/pull/414#discussion_r26250717 --- Diff: storm-core/src/jvm/backtype/storm/serialization/Test.java --- @@ -0,0 +1,17 @@ +package backtype.storm.serialization; --- End diff -- yes, was testing if thrift support serialization of Null, it does not. Removed. > Storm should support rolling upgrade/downgrade of storm cluster. > > > Key: STORM-634 > URL: https://issues.apache.org/jira/browse/STORM-634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > Currently when a new version of storm is released in order to upgrade > existing storm clusters users need to backup their existing topologies , kill > all the topologies , perform the upgrade and resubmit all the topologies. > This is painful and results in downtime which may not be acceptable for > "Always alive" production systems. > Storm should support a rolling upgrade/downgrade deployment process to avoid > these downtimes and to make the transition to a different version effortless. > Based on my initial attempt the primary issue seem to be the java > serialization used to serialize java classes like StormBase, Assignment, > WorkerHeartbeat which is then stored in zookeeper. When deserializing if the > serial versions do not match the deserialization fails resulting in processes > just getting killed indefinitely. We need to change the Utils/serialize and > Utils/deserialize so it can support non java serialization mechanism like > json. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.
[
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357511#comment-14357511
]
ASF GitHub Bot commented on STORM-634:
--
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/414#discussion_r26250794
--- Diff: storm-core/src/storm.thrift ---
@@ -243,6 +244,55 @@ struct SubmitOptions {
2: optional Credentials creds;
}
+struct SupervisorInfo {
+1: required i64 time_secs;
+2: required string hostname;
+3: optional string assignment_id;
+4: optional list used_ports;
+5: optional list meta;
+6: optional map scheduler_meta;
+7: optional i64 uptime_secs;
+}
+struct NodeInfo {
+1: required string node;
+2: required set port;
+}
+
+struct Assignment {
+1: required string master_code_dir;
+2: optional map node_host = {};
+3: optional map, NodeInfo> executor_node_port = {};
+4: optional map, i64> executor_start_time_secs = {};
+}
+
+enum TopologyStatus {
+ACTIVE = 1,
+INACTIVE = 2,
+REBALANCING = 3,
+KILLED = 4
+}
+
+union TopologyActionOptions {
+1: optional KillOptions kill_options;
+2: optional RebalanceOptions rebalance_options;
+}
+
+struct StormBase {
+1: required string name;
+2: required TopologyStatus status;
+3: required i32 num_workers;
+4: optional map component_executors;
+5: optional i32 launch_time_secs;
+6: optional string owner;
+7: optional TopologyActionOptions topology_action_options;
+8: optional TopologyStatus prev_status;//currently only used during
rebalance action.
+}
+
+struct ZKWorkerHeartbeat {
--- End diff --
I agree this is a bad naming choice. Updated to ClusterWorkerHeartbeat.
> Storm should support rolling upgrade/downgrade of storm cluster.
>
>
> Key: STORM-634
> URL: https://issues.apache.org/jira/browse/STORM-634
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>
> Currently when a new version of storm is released in order to upgrade
> existing storm clusters users need to backup their existing topologies , kill
> all the topologies , perform the upgrade and resubmit all the topologies.
> This is painful and results in downtime which may not be acceptable for
> "Always alive" production systems.
> Storm should support a rolling upgrade/downgrade deployment process to avoid
> these downtimes and to make the transition to a different version effortless.
> Based on my initial attempt the primary issue seem to be the java
> serialization used to serialize java classes like StormBase, Assignment,
> WorkerHeartbeat which is then stored in zookeeper. When deserializing if the
> serial versions do not match the deserialization fails resulting in processes
> just getting killed indefinitely. We need to change the Utils/serialize and
> Utils/deserialize so it can support non java serialization mechanism like
> json.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/414#discussion_r26250794
--- Diff: storm-core/src/storm.thrift ---
@@ -243,6 +244,55 @@ struct SubmitOptions {
2: optional Credentials creds;
}
+struct SupervisorInfo {
+1: required i64 time_secs;
+2: required string hostname;
+3: optional string assignment_id;
+4: optional list used_ports;
+5: optional list meta;
+6: optional map scheduler_meta;
+7: optional i64 uptime_secs;
+}
+struct NodeInfo {
+1: required string node;
+2: required set port;
+}
+
+struct Assignment {
+1: required string master_code_dir;
+2: optional map node_host = {};
+3: optional map, NodeInfo> executor_node_port = {};
+4: optional map, i64> executor_start_time_secs = {};
+}
+
+enum TopologyStatus {
+ACTIVE = 1,
+INACTIVE = 2,
+REBALANCING = 3,
+KILLED = 4
+}
+
+union TopologyActionOptions {
+1: optional KillOptions kill_options;
+2: optional RebalanceOptions rebalance_options;
+}
+
+struct StormBase {
+1: required string name;
+2: required TopologyStatus status;
+3: required i32 num_workers;
+4: optional map component_executors;
+5: optional i32 launch_time_secs;
+6: optional string owner;
+7: optional TopologyActionOptions topology_action_options;
+8: optional TopologyStatus prev_status;//currently only used during
rebalance action.
+}
+
+struct ZKWorkerHeartbeat {
--- End diff --
I agree this is a bad naming choice. Updated to ClusterWorkerHeartbeat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...
Github user Parth-Brahmbhatt commented on a diff in the pull request: https://github.com/apache/storm/pull/414#discussion_r26250717 --- Diff: storm-core/src/jvm/backtype/storm/serialization/Test.java --- @@ -0,0 +1,17 @@ +package backtype.storm.serialization; --- End diff -- yes, was testing if thrift support serialization of Null, it does not. Removed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (STORM-657) make the shutdown-worker sleep time before kill -9 configurable
[ https://issues.apache.org/jira/browse/STORM-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-657. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~caofangkun], I merged this into master. Keep up the good work. > make the shutdown-worker sleep time before kill -9 configurable > --- > > Key: STORM-657 > URL: https://issues.apache.org/jira/browse/STORM-657 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.9.2-incubating, 0.10.0 >Reporter: Vincent Russell >Assignee: caofangkun >Priority: Minor > Fix For: 0.10.0 > > > This is a continuation of STORM-183: Supervisor/worker shutdown hook should > be called in distributed mode. > It would be nice to be able to configure how many seconds (or millis) to > sleep for before shutting down the worker process in the shutdown-worker > function in supervisor.clj -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-657) make the shutdown-worker sleep time before kill -9 configurable
[ https://issues.apache.org/jira/browse/STORM-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357494#comment-14357494 ] ASF GitHub Bot commented on STORM-657: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/421#issuecomment-78360046 +1 > make the shutdown-worker sleep time before kill -9 configurable > --- > > Key: STORM-657 > URL: https://issues.apache.org/jira/browse/STORM-657 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.9.2-incubating, 0.10.0 >Reporter: Vincent Russell >Assignee: caofangkun >Priority: Minor > > This is a continuation of STORM-183: Supervisor/worker shutdown hook should > be called in distributed mode. > It would be nice to be able to configure how many seconds (or millis) to > sleep for before shutting down the worker process in the shutdown-worker > function in supervisor.clj -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-657) make the shutdown-worker sleep time before kill -9 configurable
[ https://issues.apache.org/jira/browse/STORM-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357499#comment-14357499 ] ASF GitHub Bot commented on STORM-657: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/421 > make the shutdown-worker sleep time before kill -9 configurable > --- > > Key: STORM-657 > URL: https://issues.apache.org/jira/browse/STORM-657 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.9.2-incubating, 0.10.0 >Reporter: Vincent Russell >Assignee: caofangkun >Priority: Minor > > This is a continuation of STORM-183: Supervisor/worker shutdown hook should > be called in distributed mode. > It would be nice to be able to configure how many seconds (or millis) to > sleep for before shutting down the worker process in the shutdown-worker > function in supervisor.clj -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-657:make the shutdown-worker sleep time ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-656) Document "external" modules and "Committer Sponsors"
[ https://issues.apache.org/jira/browse/STORM-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-656: -- Assignee: P. Taylor Goetz > Document "external" modules and "Committer Sponsors" > > > Key: STORM-656 > URL: https://issues.apache.org/jira/browse/STORM-656 > Project: Apache Storm > Issue Type: Documentation > Components: external >Reporter: P. Taylor Goetz >Assignee: P. Taylor Goetz > Fix For: 0.10.0 > > > Document the purpose of the "external" module collection and the concept of > "Committer Sponsor." > The more important point is to document the fact those listed as "Committer > Sponsors" have no special authority, roles, responsibilities, etc. and that > the Committers/PMC Members collectively bear all responsibility for the > codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-656) Document "external" modules and "Committer Sponsors"
[ https://issues.apache.org/jira/browse/STORM-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357483#comment-14357483 ] ASF GitHub Bot commented on STORM-656: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/416 > Document "external" modules and "Committer Sponsors" > > > Key: STORM-656 > URL: https://issues.apache.org/jira/browse/STORM-656 > Project: Apache Storm > Issue Type: Documentation > Components: external >Reporter: P. Taylor Goetz > Fix For: 0.10.0 > > > Document the purpose of the "external" module collection and the concept of > "Committer Sponsor." > The more important point is to document the fact those listed as "Committer > Sponsors" have no special authority, roles, responsibilities, etc. and that > the Committers/PMC Members collectively bear all responsibility for the > codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-656) Document "external" modules and "Committer Sponsors"
[ https://issues.apache.org/jira/browse/STORM-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-656. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~ptgoetz], I merged this into master. > Document "external" modules and "Committer Sponsors" > > > Key: STORM-656 > URL: https://issues.apache.org/jira/browse/STORM-656 > Project: Apache Storm > Issue Type: Documentation > Components: external >Reporter: P. Taylor Goetz >Assignee: P. Taylor Goetz > Fix For: 0.10.0 > > > Document the purpose of the "external" module collection and the concept of > "Committer Sponsor." > The more important point is to document the fact those listed as "Committer > Sponsors" have no special authority, roles, responsibilities, etc. and that > the Committers/PMC Members collectively bear all responsibility for the > codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-656: Document "external" modules and "Co...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/416 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-638) UI should show up process-id of the Worker to which an Executor is assigned
[ https://issues.apache.org/jira/browse/STORM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357442#comment-14357442 ] ASF GitHub Bot commented on STORM-638: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/396#issuecomment-78354003 @revans2 not quite comfortable with exposing pid on UI. UI is supposed to be available for all the users. In non-secure env it can be run without any filter so I am against exposing pid on the UI. > UI should show up process-id of the Worker to which an Executor is assigned > > > Key: STORM-638 > URL: https://issues.apache.org/jira/browse/STORM-638 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: with process id.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-638) UI should show up process-id of the Worker to which an Executor is assigned
[ https://issues.apache.org/jira/browse/STORM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357381#comment-14357381 ] ASF GitHub Bot commented on STORM-638: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/396#issuecomment-78343992 I'm not convinced having the PID on the UI is really that helpful, but it does provide more information for possible automation or monitoring, so I am OK with it. > UI should show up process-id of the Worker to which an Executor is assigned > > > Key: STORM-638 > URL: https://issues.apache.org/jira/browse/STORM-638 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: with process id.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-638:UI should show up process-id of the ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/396#issuecomment-78343992 I'm not convinced having the PID on the UI is really that helpful, but it does provide more information for possible automation or monitoring, so I am OK with it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-638) UI should show up process-id of the Worker to which an Executor is assigned
[
https://issues.apache.org/jira/browse/STORM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357375#comment-14357375
]
ASF GitHub Bot commented on STORM-638:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/396#discussion_r26242669
--- Diff: storm-core/src/storm.thrift ---
@@ -205,7 +205,8 @@ struct ExecutorSummary {
2: required string component_id;
3: required string host;
4: required i32 port;
- 5: required i32 uptime_secs;
+ 5: required i32 process_id;
--- End diff --
You cannot change a thrift ID like this. It breaks backwards compatibility
too much. Please make sure that you add new tags instead of changing existing
ones. It is also usually preferable to make the new tag optional, so you can
have forward and backwards compatibility.
> UI should show up process-id of the Worker to which an Executor is assigned
>
>
> Key: STORM-638
> URL: https://issues.apache.org/jira/browse/STORM-638
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: caofangkun
>Assignee: caofangkun
>Priority: Minor
> Attachments: with process id.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-638:UI should show up process-id of the ...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/396#discussion_r26242669
--- Diff: storm-core/src/storm.thrift ---
@@ -205,7 +205,8 @@ struct ExecutorSummary {
2: required string component_id;
3: required string host;
4: required i32 port;
- 5: required i32 uptime_secs;
+ 5: required i32 process_id;
--- End diff --
You cannot change a thrift ID like this. It breaks backwards compatibility
too much. Please make sure that you add new tags instead of changing existing
ones. It is also usually preferable to make the new tag optional, so you can
have forward and backwards compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-532) Supervisor should restart worker immediately, if the worker process does not exist any more
[ https://issues.apache.org/jira/browse/STORM-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357360#comment-14357360 ] ASF GitHub Bot commented on STORM-532: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-78341357 For the most part it seems fine. My manual attempts to kill workers all are foiled very quickly, but I would like to not have the supervisor be so noisy with the ps command. > Supervisor should restart worker immediately, if the worker process does not > exist any more > > > Key: STORM-532 > URL: https://issues.apache.org/jira/browse/STORM-532 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > > For now > if the worker process does not exist any more > Supervisor will have to wait a few seconds for worker heartbeart timeout and > restart worker . > If supervisor knows the worker processid and check if the process exists in > the sync-processes thread ,may need less time to restart worker. > 1: record worker process id in the worker local heartbeart > 2: in supervisor sync-processes ,get process id from worker local heartbeat > and check if the process exits > 3: if not restart it immediately -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-532:Supervisor should restart worker imm...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-78341357 For the most part it seems fine. My manual attempts to kill workers all are foiled very quickly, but I would like to not have the supervisor be so noisy with the ps command. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-532:Supervisor should restart worker imm...
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/296#discussion_r26240829 --- Diff: storm-core/src/clj/backtype/storm/util.clj --- @@ -392,6 +392,15 @@ (.addArgument command a)) (.execute (DefaultExecutor.) command))) +(defn exists-process? + [process-id] + (let [line (if on-windows? (str "cmd /c \"tasklist /FI \"PID eq " process-id "\" | findstr " process-id "\"" ) + (str "ps -p " process-id))] +(try-cause + (exec-command! line) --- End diff -- For me the supervisor is echoing the output of the ps command a lot. It is a pain and makes debugging things difficult in some cases. Could we do something to not have them printed except on an error? It would also be nice if we could have something that is linux specific that makes it so we don't need to run ps, but instead use /proc/ directly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-532) Supervisor should restart worker immediately, if the worker process does not exist any more
[ https://issues.apache.org/jira/browse/STORM-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357342#comment-14357342 ] ASF GitHub Bot commented on STORM-532: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/296#discussion_r26240829 --- Diff: storm-core/src/clj/backtype/storm/util.clj --- @@ -392,6 +392,15 @@ (.addArgument command a)) (.execute (DefaultExecutor.) command))) +(defn exists-process? + [process-id] + (let [line (if on-windows? (str "cmd /c \"tasklist /FI \"PID eq " process-id "\" | findstr " process-id "\"" ) + (str "ps -p " process-id))] +(try-cause + (exec-command! line) --- End diff -- For me the supervisor is echoing the output of the ps command a lot. It is a pain and makes debugging things difficult in some cases. Could we do something to not have them printed except on an error? It would also be nice if we could have something that is linux specific that makes it so we don't need to run ps, but instead use /proc/ directly? > Supervisor should restart worker immediately, if the worker process does not > exist any more > > > Key: STORM-532 > URL: https://issues.apache.org/jira/browse/STORM-532 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > > For now > if the worker process does not exist any more > Supervisor will have to wait a few seconds for worker heartbeart timeout and > restart worker . > If supervisor knows the worker processid and check if the process exists in > the sync-processes thread ,may need less time to restart worker. > 1: record worker process id in the worker local heartbeart > 2: in supervisor sync-processes ,get process id from worker local heartbeat > and check if the process exits > 3: if not restart it immediately -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-469) Storm UI Last Error Detail Insufficient for debugging
[ https://issues.apache.org/jira/browse/STORM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-469. --- Resolution: Fixed Fix Version/s: 0.10.0 Thanks [~rekhajoshm], I merged your pull request into master. It is as good of a solution as we can probably do at this time. Keep up the good work. > Storm UI Last Error Detail Insufficient for debugging > - > > Key: STORM-469 > URL: https://issues.apache.org/jira/browse/STORM-469 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.2-incubating > Environment: java 1.7 update 67, debian wheezy and squeeze >Reporter: Jason Kania >Assignee: Rekha Joshi > Fix For: 0.10.0 > > > The error text that is captured in the Storm UI is insufficient to debug the > issue as it gets cut off and does not link to the actual error to allow a > person to get more detail. This means that the debugging is unnecessarily > blind. This prevents diagnosis of the error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-469) Storm UI Last Error Detail Insufficient for debugging
[ https://issues.apache.org/jira/browse/STORM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357311#comment-14357311 ] ASF GitHub Bot commented on STORM-469: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/388 > Storm UI Last Error Detail Insufficient for debugging > - > > Key: STORM-469 > URL: https://issues.apache.org/jira/browse/STORM-469 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.2-incubating > Environment: java 1.7 update 67, debian wheezy and squeeze >Reporter: Jason Kania >Assignee: Rekha Joshi > > The error text that is captured in the Storm UI is insufficient to debug the > issue as it gets cut off and does not link to the actual error to allow a > person to get more detail. This means that the debugging is unnecessarily > blind. This prevents diagnosis of the error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-469
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/388#issuecomment-78327431 +1 I think the change is a good start to this problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-596: remove config topology.receiver.buf...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/363#issuecomment-78316822 This is technically a non-backwards compatible change, but I think it is OK. I am +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-681) Auto insert license header with genthrift.sh
[ https://issues.apache.org/jira/browse/STORM-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357268#comment-14357268 ] ASF GitHub Bot commented on STORM-681: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/436#issuecomment-78316257 I just had the one comment. As for the date in the generated files I see two options. 1) We remove it with a sed script. It is optional, and thrift is not following the formatting convention for it anyways 2) We eventually move to generating the code for the build each time. I don't think either of these should be something we worry about now. The thrift code changes rarely enough that I am fine with ignoring it until it is a problem. > Auto insert license header with genthrift.sh > > > Key: STORM-681 > URL: https://issues.apache.org/jira/browse/STORM-681 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0, 0.9.3-rc2 >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Minor > Labels: documentaion, thrift > > Current genthrift.sh does not insert license headers into generated source > codes. These java codes and python codes should have license headers. > And documentation about this command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-596) "topology.receiver.buffer.size" has no effect
[ https://issues.apache.org/jira/browse/STORM-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357273#comment-14357273 ] ASF GitHub Bot commented on STORM-596: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/363#issuecomment-78316822 This is technically a non-backwards compatible change, but I think it is OK. I am +1. > "topology.receiver.buffer.size" has no effect > -- > > Key: STORM-596 > URL: https://issues.apache.org/jira/browse/STORM-596 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0, 0.9.3-rc2 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > > https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/messaging/loader.clj#L27 > backtype.storm.messaging.loader#mk-receive-thread accepts max-buffer-size as > an input but the value isn't used within the function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-681] Auto insert license header with ge...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/436#issuecomment-78316257 I just had the one comment. As for the date in the generated files I see two options. 1) We remove it with a sed script. It is optional, and thrift is not following the formatting convention for it anyways 2) We eventually move to generating the code for the build each time. I don't think either of these should be something we worry about now. The thrift code changes rarely enough that I am fine with ignoring it until it is a problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-681) Auto insert license header with genthrift.sh
[
https://issues.apache.org/jira/browse/STORM-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357257#comment-14357257
]
ASF GitHub Bot commented on STORM-681:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/436#discussion_r26235414
--- Diff: storm-core/src/genthrift.sh ---
@@ -17,6 +17,16 @@
rm -rf gen-javabean gen-py py
rm -rf jvm/backtype/storm/generated
thrift7 --gen java:beans,hashcode,nocamel --gen py:utf8strings storm.thrift
+for file in gen-javabean/backtype/storm/generated/* ; do
+ cat java_license_header.txt ${file} > ${file}.tmp
+ mv ${file}.tmp ${file}
+done
+cat py_license_header.txt gen-py/__init__.py > gen-py/__init__.py.tmp
+mv gen-py/__init__.py.tmp gen-py/__init__.py
+for file in gen-py/storm/* ; do
+ cat py_license_header.txt ${file} > ${file}.tmp
+ mv ${file}.tmp ${file}
--- End diff --
Could we include a `-f` on the mv command? I know a lot of us have mv
configured as an alias to `mv -i` to avoid unexpected overwrites and this would
cause problems.
> Auto insert license header with genthrift.sh
>
>
> Key: STORM-681
> URL: https://issues.apache.org/jira/browse/STORM-681
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0, 0.9.3-rc2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Labels: documentaion, thrift
>
> Current genthrift.sh does not insert license headers into generated source
> codes. These java codes and python codes should have license headers.
> And documentation about this command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.
[ https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357254#comment-14357254 ] ASF GitHub Bot commented on STORM-634: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/414#issuecomment-78313719 For the most part it looks good. I had a few minor comments. My only real concern is about maintaining backwards compatibility for clients. I didn't see much that would prevent this, but I would feel more comfortable if it were tested. > Storm should support rolling upgrade/downgrade of storm cluster. > > > Key: STORM-634 > URL: https://issues.apache.org/jira/browse/STORM-634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > Currently when a new version of storm is released in order to upgrade > existing storm clusters users need to backup their existing topologies , kill > all the topologies , perform the upgrade and resubmit all the topologies. > This is painful and results in downtime which may not be acceptable for > "Always alive" production systems. > Storm should support a rolling upgrade/downgrade deployment process to avoid > these downtimes and to make the transition to a different version effortless. > Based on my initial attempt the primary issue seem to be the java > serialization used to serialize java classes like StormBase, Assignment, > WorkerHeartbeat which is then stored in zookeeper. When deserializing if the > serial versions do not match the deserialization fails resulting in processes > just getting killed indefinitely. We need to change the Utils/serialize and > Utils/deserialize so it can support non java serialization mechanism like > json. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.
[ https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357247#comment-14357247 ] ASF GitHub Bot commented on STORM-634: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/414#discussion_r26234849 --- Diff: storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java --- @@ -17,11 +17,7 @@ */ package backtype.storm.serialization; --- End diff -- This is really minor, but it might also be nice to deprecate this, and migrate the core deserialization over to Utils, so we are not calling a deprecated class. > Storm should support rolling upgrade/downgrade of storm cluster. > > > Key: STORM-634 > URL: https://issues.apache.org/jira/browse/STORM-634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > Currently when a new version of storm is released in order to upgrade > existing storm clusters users need to backup their existing topologies , kill > all the topologies , perform the upgrade and resubmit all the topologies. > This is painful and results in downtime which may not be acceptable for > "Always alive" production systems. > Storm should support a rolling upgrade/downgrade deployment process to avoid > these downtimes and to make the transition to a different version effortless. > Based on my initial attempt the primary issue seem to be the java > serialization used to serialize java classes like StormBase, Assignment, > WorkerHeartbeat which is then stored in zookeeper. When deserializing if the > serial versions do not match the deserialization fails resulting in processes > just getting killed indefinitely. We need to change the Utils/serialize and > Utils/deserialize so it can support non java serialization mechanism like > json. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/414#discussion_r26234849 --- Diff: storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java --- @@ -17,11 +17,7 @@ */ package backtype.storm.serialization; --- End diff -- This is really minor, but it might also be nice to deprecate this, and migrate the core deserialization over to Utils, so we are not calling a deprecated class. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-699) storm-jdbc should support customer insert queries.
[
https://issues.apache.org/jira/browse/STORM-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357222#comment-14357222
]
ASF GitHub Bot commented on STORM-699:
--
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/458#discussion_r26232904
--- Diff: external/storm-jdbc/README.md ---
@@ -13,14 +13,39 @@ public interface JdbcMapper extends Serializable {
}
```
-The `getColumns()` method defines how a storm tuple maps to a list of
columns representing a row in a database.
+The `getColumns()` method defines how a storm tuple maps to a list of
columns representing a row in a database.
+**The order of the returned list is important. The place holders in the
supplied queries are resolved in the same order as returned list.**
+For example if the user supplied insert query is `insert into table
user(user_id, user_name, create_date) values (?,?, now())` the 1st item
--- End diff --
I had an extra "table" in it, fixed.
> storm-jdbc should support customer insert queries.
> --
>
> Key: STORM-699
> URL: https://issues.apache.org/jira/browse/STORM-699
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
>
> Currently storm-jdbc insert bolt/state only supports to specify a table name
> and constructs a query of the form "insert into tablename values(?,?,?)"
> based on table's schema. This fails to support use cases like "insert into as
> select * from" or special cases like Phoenix that has a jdbc driver but only
> supports "upsert into".
> We should add a way so the users can specify their own custom query for the
> insert bolt. This was already pointed out by [~revans2] during the PR review
> and we now have concrete cases that will be benefited by this feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-699) storm-jdbc should support customer insert queries.
[
https://issues.apache.org/jira/browse/STORM-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357224#comment-14357224
]
ASF GitHub Bot commented on STORM-699:
--
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/458#discussion_r26232936
--- Diff: external/storm-jdbc/README.md ---
@@ -38,34 +63,30 @@ hikariConfigMap.put("dataSource.password","password");
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
```
-The mapper initialized in the example above assumes a storm tuple has
value for all the columns.
-If your storm tuple only has fields for a subset of columns i.e. if some
of the columns in your table have default values
-and you want to only insert values for columns with no default values you
can enforce the behavior by initializing the
-`SimpleJdbcMapper` with explicit columnschema. For example, if you have a
user_details table
-`create table if not exists user_details (user_id integer, user_name
varchar(100), dept_name varchar(100), create_time TIMESTAMP DEFAULT
CURRENT_TIMESTAMP);`
-In this table the create_time column has a default value. To ensure only
the columns with no default values are inserted
-you can initialize the `jdbcMapper` as below:
+The mapper initialized in the example above assumes a storm tuple has
value for all the columns of the table you intend to insert data into and its
`getColumn`
+method will return the columns in the order in which Jdbc connection
instance's `connection.getMetaData().getColumns();` method returns them.
+**If you specified your own insert query to `JdbcInsertBolt` you must
initialize `SimpleJdbcMapper` with explicit columnschema such that the schema
has columns in the same order as your insert queries.**
+For example if your insert query is `Insert into user (user_id, user_name)
values (?,?)` then your `SimpleJdbcMapper` should be initialized with the
following statements:
```java
List columnSchema = Lists.newArrayList(
new Column("user_id", java.sql.Types.INTEGER),
-new Column("user_name", java.sql.Types.VARCHAR));
-JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(columnSchema);
+new Column("user_name", java.sql.Types.VARCHAR),
--- End diff --
missed a closing bracket and ; fixed.
> storm-jdbc should support customer insert queries.
> --
>
> Key: STORM-699
> URL: https://issues.apache.org/jira/browse/STORM-699
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
>
> Currently storm-jdbc insert bolt/state only supports to specify a table name
> and constructs a query of the form "insert into tablename values(?,?,?)"
> based on table's schema. This fails to support use cases like "insert into as
> select * from" or special cases like Phoenix that has a jdbc driver but only
> supports "upsert into".
> We should add a way so the users can specify their own custom query for the
> insert bolt. This was already pointed out by [~revans2] during the PR review
> and we now have concrete cases that will be benefited by this feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-699: storm-jdbc should support custom in...
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/storm/pull/458#discussion_r26232936
--- Diff: external/storm-jdbc/README.md ---
@@ -38,34 +63,30 @@ hikariConfigMap.put("dataSource.password","password");
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
```
-The mapper initialized in the example above assumes a storm tuple has
value for all the columns.
-If your storm tuple only has fields for a subset of columns i.e. if some
of the columns in your table have default values
-and you want to only insert values for columns with no default values you
can enforce the behavior by initializing the
-`SimpleJdbcMapper` with explicit columnschema. For example, if you have a
user_details table
-`create table if not exists user_details (user_id integer, user_name
varchar(100), dept_name varchar(100), create_time TIMESTAMP DEFAULT
CURRENT_TIMESTAMP);`
-In this table the create_time column has a default value. To ensure only
the columns with no default values are inserted
-you can initialize the `jdbcMapper` as below:
+The mapper initialized in the example above assumes a storm tuple has
value for all the columns of the table you intend to insert data into and its
`getColumn`
+method will return the columns in the order in which Jdbc connection
instance's `connection.getMetaData().getColumns();` method returns them.
+**If you specified your own insert query to `JdbcInsertBolt` you must
initialize `SimpleJdbcMapper` with explicit columnschema such that the schema
has columns in the same order as your insert queries.**
+For example if your insert query is `Insert into user (user_id, user_name)
values (?,?)` then your `SimpleJdbcMapper` should be initialized with the
following statements:
```java
List columnSchema = Lists.newArrayList(
new Column("user_id", java.sql.Types.INTEGER),
-new Column("user_name", java.sql.Types.VARCHAR));
-JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(columnSchema);
+new Column("user_name", java.sql.Types.VARCHAR),
--- End diff --
missed a closing bracket and ; fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
