date:20150312


[ 
https://issues.apache.org/jira/browse/STORM-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357075#comment-14357075
 ] 

ASF GitHub Bot commented on STORM-625:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/461#issuecomment-78291337
  
+1, but since some of the code came from me another committer is going to 
need to give a +1 too.


 Netty context never forgets about a Client.
 ---

 Key: STORM-625
 URL: https://issues.apache.org/jira/browse/STORM-625
 Project: Apache Storm
  Issue Type: Bug
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
  Labels: newbie

 The Netty Context 
 (https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/messaging/netty/Context.java)
  keeps track of all connections so that it can close them when it is closed 
 in term.  But it never removes any of them from its list until it is 
 terminated.  If workers are rescheduled to new locations this can cause the 
 old connections to be leaked.
 We should have the Connections remove themselves from the list when they are 
 closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: New Committer/PMC Member: Thomas Becker

2015-03-12 Thread Michael G. Noll

Welcome aboard, Thomas! :-)

--Michael



 On 10.03.2015, at 19:07, P. Taylor Goetz ptgo...@apache.org wrote:
 
 Please join me in welcoming Thomas Becker (wurstmeister) as a new Apache 
 Storm Committer/PMC member.
 
 Thomas has made significant contributions to the Storm codebase, particularly 
 in the area of Kafka integration, and shown dedication to advancing the Storm 
 community.
 
 Congratulations and welcome Thomas!
 
 -Taylor

[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78291244
  
@revans2  pinging for a review. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-638:UI should show up process-id of the ...

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/396#discussion_r26242766
  
--- Diff: storm-core/src/ui/public/templates/component-page-template.html 
---
@@ -211,6 +216,7 @@
 td{{uptime}}/td
 td{{host}}/td
 tda href={{workerLogLink}}{{port}}/a/td
+td{{processId}}
--- End diff --

Can we close the td tag?  I know it is not required but all the other ones 
are closed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-638) UI should show up process-id of the Worker to which an Executor is assigned


[ 
https://issues.apache.org/jira/browse/STORM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357376#comment-14357376
 ] 

ASF GitHub Bot commented on STORM-638:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/396#discussion_r26242766
  
--- Diff: storm-core/src/ui/public/templates/component-page-template.html 
---
@@ -211,6 +216,7 @@
 td{{uptime}}/td
 td{{host}}/td
 tda href={{workerLogLink}}{{port}}/a/td
+td{{processId}}
--- End diff --

Can we close the td tag?  I know it is not required but all the other ones 
are closed.


 UI should show up  process-id of the Worker to which an Executor is assigned
 

 Key: STORM-638
 URL: https://issues.apache.org/jira/browse/STORM-638
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 0.10.0
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
 Attachments: with process id.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...

2015-03-12 Thread Parth-Brahmbhatt

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78300710
  
+1. I think most users would much rather define a nimbus.groups m e can 
probably file a separate jira for that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-689) SimpleACLAuthorizer should provide a way to restrict who can submit topologies


[ 
https://issues.apache.org/jira/browse/STORM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357139#comment-14357139
 ] 

ASF GitHub Bot commented on STORM-689:
--

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78300710
  
+1. I think most users would much rather define a nimbus.groups m e can 
probably file a separate jira for that.


 SimpleACLAuthorizer should provide a way to restrict who can submit topologies
 --

 Key: STORM-689
 URL: https://issues.apache.org/jira/browse/STORM-689
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
Priority: Trivial

 SimpleACLAuthorizer currently allows anyone with a valid kerberos ticket to 
 submit topologies. There are cases where storm admins want to allow only 
 selected users to submit topologies. I am proposing nimbus.users config 
 option if its added to storm.yaml only the listed users can deploy the storm 
 topologies. 
 cc [~revans2]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/414#discussion_r26231202
  
--- Diff: storm-core/src/storm.thrift ---
@@ -243,6 +244,55 @@ struct SubmitOptions {
   2: optional Credentials creds;
 }
 
+struct SupervisorInfo {
+1: required i64 time_secs;
+2: required string hostname;
+3: optional string assignment_id;
+4: optional listi64 used_ports;
+5: optional listi64 meta;
+6: optional mapstring, string scheduler_meta;
+7: optional i64 uptime_secs;
+}
+struct NodeInfo {
+1: required string node;
+2: required seti64 port;
+}
+
+struct Assignment {
+1: required string master_code_dir;
+2: optional mapstring, string node_host = {};
+3: optional maplisti64, NodeInfo executor_node_port = {};
+4: optional maplisti64, i64 executor_start_time_secs = {};
+}
+
+enum TopologyStatus {
+ACTIVE = 1,
+INACTIVE = 2,
+REBALANCING = 3,
+KILLED = 4
+}
+
+union TopologyActionOptions {
+1: optional KillOptions kill_options;
+2: optional RebalanceOptions rebalance_options;
+}
+
+struct StormBase {
+1: required string name;
+2: required TopologyStatus status;
+3: required i32 num_workers;
+4: optional mapstring, i32 component_executors;
+5: optional i32 launch_time_secs;
+6: optional string owner;
+7: optional TopologyActionOptions topology_action_options;
+8: optional TopologyStatus prev_status;//currently only used during 
rebalance action.
+}
+
+struct ZKWorkerHeartbeat {
--- End diff --

I'd rather that this is not called ZKWorkerHeartbeat.  ZK is a storage 
solution that could change in the future, I'd rather call it a 
ClusterWorkerHeartbeat or something that describes the abstraction rather than 
the storage location.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/414#discussion_r26229576
  
--- Diff: storm-core/src/clj/backtype/storm/converter.clj ---
@@ -0,0 +1,200 @@
+(ns backtype.storm.converter
--- End diff --

I think it is OK for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.


[ 
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357176#comment-14357176
 ] 

ASF GitHub Bot commented on STORM-634:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/414#discussion_r26230947
  
--- Diff: storm-core/src/storm.thrift ---
@@ -193,6 +193,7 @@ struct ExecutorStats {
   1: required mapstring, mapstring, i64 emitted;
   2: required mapstring, mapstring, i64 transferred;
   3: required ExecutorSpecificStats specific;
+  4: required double rate;
--- End diff --

Having this be required technically breaks compatibility.  I'm not sure if 
this is a problem or not.  ExecutorStats is exposed to the end user, but really 
only as a consumer. So it makes it impossible for a newer client to interact 
with an older nimbus instance.  I think this is OK, but we should be very 
careful about these kinds of things going forward. 

Have you tested using older clients with a newer nimbus instance?


 Storm should support rolling upgrade/downgrade of storm cluster.
 

 Key: STORM-634
 URL: https://issues.apache.org/jira/browse/STORM-634
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt

 Currently when a new version of storm is released in order to upgrade 
 existing storm clusters users need to backup their existing topologies , kill 
 all the topologies , perform the upgrade and resubmit all the topologies. 
 This is painful and results in downtime which may not be acceptable for 
 Always alive  production systems.
 Storm should support a rolling  upgrade/downgrade deployment process to avoid 
 these downtimes and to make the transition to a different version effortless. 
 Based on my initial attempt the primary issue seem to be the java 
 serialization used to serialize java classes like StormBase, Assignment, 
 WorkerHeartbeat which is then stored in zookeeper. When deserializing if the 
 serial versions do not match the deserialization fails resulting in processes 
 just getting killed indefinitely. We need to change the Utils/serialize and 
 Utils/deserialize so it can support non java serialization mechanism like 
 json. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-469

2015-03-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/388


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-541:Clean duplicate dependences in poms

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/422#discussion_r26250879
  
--- Diff: pom.xml ---
@@ -521,12 +521,6 @@
 scopetest/scope
 /dependency
 dependency
-groupIdorg.clojars.runa/groupId
-artifactIdconjure/artifactId
-version${conjure.version}/version
--- End diff --

The code no longer compiles because this dependency was removed.  
dependencyManagement in the pom does not add in a dependency, it just tells 
maven if a project needs this dependency use this particular version.  Without 
this here the remaining dependency in storm-core does not know which version to 
use.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-657:make the shutdown-worker sleep time ...

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/421#issuecomment-78360046
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: [DISCUSS] Release Storm 0.9.4 / 0.10.0

2015-03-12 Thread Richard Kellogg

Suggest we pull in STORM-559 as well.  It is strictly an update to 
documentation and has already been merged to trunk.

-Original Message-
From: P. Taylor Goetz [mailto:ptgo...@gmail.com] 
Sent: Wednesday, March 11, 2015 4:51 PM
To: dev@storm.apache.org
Subject: Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

Thanks for the feedback Richard, Parth, and Jungtaek.  I’ve made a note of your 
suggestions for the releases.

Does anyone else have any thoughts (esp. committers)? Should we move forward 
with releasing?

-Taylor

On Mar 5, 2015, at 5:00 PM, 임정택 kabh...@gmail.com wrote:

 storm-redis (scheduled to be released at 0.10.0) has one bugfix and 
 one essential feature PRs.

 - bugfix: https://issues.apache.org/jira/browse/STORM-690
 -- It fixes connection pool issue.
 - feature: https://issues.apache.org/jira/browse/STORM-691
 -- It provides basic lookup / persist bolts so I believe it's necessary.

 Furthermore, I'd like to continue to support various data types with 
 storm-redis Trident, after STORM-691 is merged to master.

 Thanks!

 Regards
 Jungtaek Lim (HeartSaVioR)

 2015-03-06 2:37 GMT+09:00 P. Taylor Goetz ptgo...@gmail.com:

 I’d like to start a discussion for releasing 0.9.4 (maintenance 
 release) and 0.10.0 (security release).

 0.9.4 is basically a branch of 0.9.3 with two important bug fixes:

STORM-329: fix cascading Storm failure by improving 
 reconnection strategy and buffering messages
STORM-130: Supervisor getting killed due to
 java.io.FileNotFoundException: File '../stormconf.ser' does not exist.

 Both are long-standing bugs that have proven problematic for many users.
 I’d be in favor of releasing 0.9.4 with just those two fixes, but I’m 
 interested in finding out if anyone thinks there are additional 
 patches to master that should be considered for 0.9.4.

 0.10.0 is a much larger release in terms of changes. In addition to 
 the changes above, it includes all the new security features and 
 numerous fixes and enhancements (see the CHANGELOG in the master branch for 
 a full list).

 Do we feel 0.10.0 is ready for release? If not what outstanding 
 bugs/patches should we consider before releasing?

 I’m fine holding off on a 0.10.0 release if we feel there is 
 additional work to be done, but I’d like to at least move forward with 0.9.4 
 release.

 Thoughts?

 -Taylor

 --
 Name : 임 정택
 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter 
 : http://twitter.com/heartsavior LinkedIn : 
 http://www.linkedin.com/in/heartsavior

[GitHub] storm pull request: STORM-702: Exhibitor support

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/432#issuecomment-78750701
  
@atdixon @revans2 with this patch can a user who doesn't want use exhibitor 
avoid it by not having the config or the user must have the config in their 
storm.yaml.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-541:Clean duplicate dependences in poms

2015-03-12 Thread caofangkun

Github user caofangkun commented on a diff in the pull request:

https://github.com/apache/storm/pull/422#discussion_r26361833
  
--- Diff: pom.xml ---
@@ -521,12 +521,6 @@
 scopetest/scope
 /dependency
 dependency
-groupIdorg.clojars.runa/groupId
-artifactIdconjure/artifactId
-version${conjure.version}/version
--- End diff --

Thank you @revans2 
Fixed. Please review this PR again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-541) Build produces maven warnings


[ 
https://issues.apache.org/jira/browse/STORM-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359836#comment-14359836
 ] 

ASF GitHub Bot commented on STORM-541:
--

Github user caofangkun commented on a diff in the pull request:

https://github.com/apache/storm/pull/422#discussion_r26361833
  
--- Diff: pom.xml ---
@@ -521,12 +521,6 @@
 scopetest/scope
 /dependency
 dependency
-groupIdorg.clojars.runa/groupId
-artifactIdconjure/artifactId
-version${conjure.version}/version
--- End diff --

Thank you @revans2 
Fixed. Please review this PR again.


 Build produces maven warnings
 -

 Key: STORM-541
 URL: https://issues.apache.org/jira/browse/STORM-541
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 0.10.0, 0.9.3-rc2
 Environment: Ubuntu 14.04 on POWER
Reporter: jez wain
Assignee: caofangkun
Priority: Minor
  Labels: patch
   Original Estimate: 10m
  Remaining Estimate: 10m

 The maven compilation task produces the following warnings because version 
 numbers are missing from some plugins in the pom.xml file. I have created a 
 patch file.
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:maven-shade-clojure-transformer:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:storm-core:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:storm-starter:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:storm-kafka:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:storm-hdfs:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for com.github.ptgoetz:storm-hbase:jar:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ 
 org.apache.storm:storm:0.9.3-rc2-SNAPSHOT, /home/jez/storm/pom.xml, line 661, 
 column 21
 [WARNING]
 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.storm:storm:pom:0.9.3-rc2-SNAPSHOT
 [WARNING] 'reporting.plugins.plugin.version' for 
 org.apache.maven.plugins:maven-surefire-report-plugin is missing. @ line 661, 
 column 21
 [WARNING]
 [WARNING] It is highly recommended to fix these problems because they 
 threaten the stability of your build.
 [WARNING]
 [WARNING] For this reason, future Maven versions might no longer support 
 building such malformed projects.
 [WARNING]
 PATCH:
 diff --git a/pom.xml b/pom.xml
 index b63d332..37a5acd 100644
 --- a/pom.xml
 +++ b/pom.xml
 @@ -656,10 +656,12 @@
  plugin
  groupIdorg.apache.maven.plugins/groupId
  artifactIdmaven-javadoc-plugin/artifactId
 +version2.9/version
  /plugin
  plugin
  groupIdorg.apache.maven.plugins/groupId
  artifactIdmaven-surefire-report-plugin/artifactId
 + version2.16/version
  configuration
  reportsDirectories
  file${project.build.directory}/test-reports/file
 @@ -694,6 +696,7 @@
  plugin
  groupIdorg.apache.maven.plugins/groupId
  artifactIdmaven-javadoc-plugin/artifactId
 +version2.9/version

[jira] [Commented] (STORM-689) SimpleACLAuthorizer should provide a way to restrict who can submit topologies


[ 
https://issues.apache.org/jira/browse/STORM-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359973#comment-14359973
 ] 

ASF GitHub Bot commented on STORM-689:
--

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78814683
  
+1


 SimpleACLAuthorizer should provide a way to restrict who can submit topologies
 --

 Key: STORM-689
 URL: https://issues.apache.org/jira/browse/STORM-689
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
Priority: Trivial

 SimpleACLAuthorizer currently allows anyone with a valid kerberos ticket to 
 submit topologies. There are cases where storm admins want to allow only 
 selected users to submit topologies. I am proposing nimbus.users config 
 option if its added to storm.yaml only the listed users can deploy the storm 
 topologies. 
 cc [~revans2]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-704) Apply Travis CI

[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359987#comment-14359987
]

Jungtaek Lim commented on STORM-704:

It's first success build of Apache Storm from Travis CI.
[https://travis-ci.org/HeartSaVioR/storm/jobs/54201436]

Btw, as I described it earlier, if storm-core is broken we cannot see what
tests fail. (It just prints 'clojure failed'.)
If we resolve it everything should be OK.

Apply Travis CI
---

Key: STORM-704
URL: https://issues.apache.org/jira/browse/STORM-704
Project: Apache Storm
Issue Type: Improvement
Environment: Travis CI
Reporter: Jungtaek Lim
Assignee: Jungtaek Lim
Priority: Minor

Now Apache Storm takes advantage of Github, we can apply Travis CI to some
more advantages.
- Build matrix
-- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7,
oraclejdk8), and it can be tested separately.
- Build automatically
-- pushed new commits, any new PRs
- Integrated with Github
-- Contributors can see his/her PR breaks compilation / test in some minutes.
-- If he/she adds commits to PR, Travis builds it automatically and update
build result.
Please see [https://travis-ci.org/xetorthio/jedis] for example.
There're some hurdles applying Travis CI to Apache Storm project, but we can
overcome these and finally get great CI.
Current hurdles
- asfgit should manage Travis CI setup for the first time
-- other Apache projects already did it by requesting it to INFRA
--- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
- Travis CI restricts stdout with 4M which is too small for Storm maven
output.
-- alternative way : filter Storm's INFO message by {code}mvn clean test -U |
egrep -v [0-9]+ \[.+\] INFO{code}
--- inspired by [https://github.com/apache/tajo/pull/8]
- In storm-core, we can't see tests failure information on stdout cause it
just prints 'clojure failed'.
-- We need to find a way to upload surefire / clojure tests report files to
somewhere, and uploaded files should be visible easily.
--- Travis supports uploading artifacts to S3 by
[https://github.com/travis-ci/artifacts], but S3 is not free.
-- I'm not familiar with Clojure, but can we print tests summary with clojure
tests as same as Java junit tests?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-702) Apache Exhibitor support


[ 
https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359806#comment-14359806
 ] 

ASF GitHub Bot commented on STORM-702:
--

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/432#issuecomment-78750701
  
@atdixon @revans2 with this patch can a user who doesn't want use exhibitor 
avoid it by not having the config or the user must have the config in their 
storm.yaml.


 Apache Exhibitor support
 

 Key: STORM-702
 URL: https://issues.apache.org/jira/browse/STORM-702
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Aaron Dixon
Priority: Critical

 Storm connects to Zookeeper via an explicit list of zookeeper hosts.
 Apache Exhibitor offers zookeeper cluster discovery and management features 
 allowing for dynamically resizing zookeeper clusters, restarting and 
 replacing zk machines in the cluster, etc.
 Curator supports creating zookeeper client connections using exhibitor hosts.
 Storm should allow (optionally) for connection to its Zookeeper clusters 
 through Exhibitor.
 Here is the github pull request that prompted this ticket, and that 
 introduces  optional exhibitor configuration and exhibitor support in Storm: 
 https://github.com/apache/storm/pull/432



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-702) Apache Exhibitor support


[ 
https://issues.apache.org/jira/browse/STORM-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359840#comment-14359840
 ] 

ASF GitHub Bot commented on STORM-702:
--

Github user atdixon commented on the pull request:

https://github.com/apache/storm/pull/432#issuecomment-78764060
  
The config is optional. If not present, everything works as usual.

On Thu, Mar 12, 2015 at 9:18 PM, Harsha notificati...@github.com wrote:

 @atdixon @revans2 with this patch can a user who doesn't want use 
exhibitor avoid it by not having the config or the user must have the config in 
their storm.yaml.
 ---
 Reply to this email directly or view it on GitHub:
 https://github.com/apache/storm/pull/432#issuecomment-78750701


 Apache Exhibitor support
 

 Key: STORM-702
 URL: https://issues.apache.org/jira/browse/STORM-702
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Aaron Dixon
Priority: Critical

 Storm connects to Zookeeper via an explicit list of zookeeper hosts.
 Apache Exhibitor offers zookeeper cluster discovery and management features 
 allowing for dynamically resizing zookeeper clusters, restarting and 
 replacing zk machines in the cluster, etc.
 Curator supports creating zookeeper client connections using exhibitor hosts.
 Storm should allow (optionally) for connection to its Zookeeper clusters 
 through Exhibitor.
 Here is the github pull request that prompted this ticket, and that 
 introduces  optional exhibitor configuration and exhibitor support in Storm: 
 https://github.com/apache/storm/pull/432



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-638:UI should show up process-id of the ...

2015-03-12 Thread caofangkun

Github user caofangkun commented on a diff in the pull request:

https://github.com/apache/storm/pull/396#discussion_r26362269
  
--- Diff: storm-core/src/storm.thrift ---
@@ -205,7 +205,8 @@ struct ExecutorSummary {
   2: required string component_id;
   3: required string host;
   4: required i32 port;
-  5: required i32 uptime_secs;
+  5: required i32 process_id;
--- End diff --

Fixed. Thank you. Please review this again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-638) UI should show up process-id of the Worker to which an Executor is assigned


[ 
https://issues.apache.org/jira/browse/STORM-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359852#comment-14359852
 ] 

ASF GitHub Bot commented on STORM-638:
--

Github user caofangkun commented on a diff in the pull request:

https://github.com/apache/storm/pull/396#discussion_r26362269
  
--- Diff: storm-core/src/storm.thrift ---
@@ -205,7 +205,8 @@ struct ExecutorSummary {
   2: required string component_id;
   3: required string host;
   4: required i32 port;
-  5: required i32 uptime_secs;
+  5: required i32 process_id;
--- End diff --

Fixed. Thank you. Please review this again.


 UI should show up  process-id of the Worker to which an Executor is assigned
 

 Key: STORM-638
 URL: https://issues.apache.org/jira/browse/STORM-638
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 0.10.0
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
 Attachments: with process id.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...

2015-03-12 Thread vesense

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78814683
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-650) Storm-Kafka Refactoring and Improvements

2015-03-12 Thread Sriharsha Chintalapani (JIRA)

[
https://issues.apache.org/jira/browse/STORM-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358844#comment-14358844
]

Sriharsha Chintalapani commented on STORM-650:
--

Thanks for summarizing [~wurstmeister] . There are lot of things here instead
of getting this in as one patch can we do it progressively.
We've a PR up for STORM-631 is it conflicting with where we want to go with
kafka-connector If not we should get it in and open up additional jiras for
further patches.
[~parth.brahmbhatt] [~ptgoetz] [~revans2] any thoughts . We are getting lot of
queries around Kafka spout config IMO it definitely need refactoring I don't
think this parent JIRA hold back on any progress.

Storm-Kafka Refactoring and Improvements

Key: STORM-650
URL: https://issues.apache.org/jira/browse/STORM-650
Project: Apache Storm
Issue Type: Improvement
Components: storm-kafka
Reporter: P. Taylor Goetz

This is intended to be a parent/umbrella JIRA covering a number of
efforts/suggestions aimed at improving the storm-kafka module. The goal is to
facilitate communication and collaboration by providing a central point for
discussion and coordination.
The first phase should be to identify and agree upon a list of high-level
points we would like to address. Once that is complete, we can move on to
implementation/design discussions, followed by an implementation plan,
division of labor, etc.
A non-exhaustive, initial list of items follows. New/additional thoughts can
be proposed in the comments.
* Improve API for Specifying the Kafka Starting point
Configuring the kafka spout's starting position (e.g. forceFromStart=true) is
a common source of confusion. This should be refactored to provide an easy to
understand, unambiguous API for configuring this property.
* Use Kafka APIs Instead of Internal ZK Metadata (STORM-590)
Currently the Kafka spout relies on reading Kafka's internal metadata from
zookeeper. This should be refactored to use the Kafka Consumer API to protect
against changes to the internal metadata format stored in ZK.
* Improve Error Handling
There are a number of failure scenarios with the kafka spout that users may
want to react to differently based on their use case. Add a failure handler
API that allows users to implement and/or plug in alternative failure
handling implementations. It is assumed that default (sane) implementations
would be included and configured by default.
* Configuration/General Refactoring (BrokerHosts, etc.) (STORM-631)
(need to flesh this out better) Reduce unnecessary marker
interfaces/instance of checks. Unify configuration of core storm/trident
spout implementations.
* Kafka Spout doesn't pick up from the beginning of the queue unless
forceFromStart specified (STORM-563)
Discussion Items:
* How important is backward compatibility?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

2015-03-12 Thread T B

+1 for releasing 0.9.4

Thomas

On 11 March 2015 at 21:39, Richard Kellogg rmkell...@comcast.net wrote:

 Suggest we pull in STORM-559 as well.  It is strictly an update to
 documentation and has already been merged to trunk.

 -Original Message-
 From: P. Taylor Goetz [mailto:ptgo...@gmail.com]
 Sent: Wednesday, March 11, 2015 4:51 PM
 To: dev@storm.apache.org
 Subject: Re: [DISCUSS] Release Storm 0.9.4 / 0.10.0

 Thanks for the feedback Richard, Parth, and Jungtaek.  I’ve made a note of
 your suggestions for the releases.

 Does anyone else have any thoughts (esp. committers)? Should we move
 forward with releasing?

 -Taylor

 On Mar 5, 2015, at 5:00 PM, 임정택 kabh...@gmail.com wrote:

  storm-redis (scheduled to be released at 0.10.0) has one bugfix and
  one essential feature PRs.
 
  - bugfix: https://issues.apache.org/jira/browse/STORM-690
  -- It fixes connection pool issue.
  - feature: https://issues.apache.org/jira/browse/STORM-691
  -- It provides basic lookup / persist bolts so I believe it's necessary.
 
  Furthermore, I'd like to continue to support various data types with
  storm-redis Trident, after STORM-691 is merged to master.
 
  Thanks!
 
  Regards
  Jungtaek Lim (HeartSaVioR)
 
 
  2015-03-06 2:37 GMT+09:00 P. Taylor Goetz ptgo...@gmail.com:
 
  I’d like to start a discussion for releasing 0.9.4 (maintenance
  release) and 0.10.0 (security release).
 
  0.9.4 is basically a branch of 0.9.3 with two important bug fixes:
 
 STORM-329: fix cascading Storm failure by improving
  reconnection strategy and buffering messages
 STORM-130: Supervisor getting killed due to
  java.io.FileNotFoundException: File '../stormconf.ser' does not exist.
 
  Both are long-standing bugs that have proven problematic for many users.
  I’d be in favor of releasing 0.9.4 with just those two fixes, but I’m
  interested in finding out if anyone thinks there are additional
  patches to master that should be considered for 0.9.4.
 
  0.10.0 is a much larger release in terms of changes. In addition to
  the changes above, it includes all the new security features and
  numerous fixes and enhancements (see the CHANGELOG in the master branch
 for a full list).
 
  Do we feel 0.10.0 is ready for release? If not what outstanding
  bugs/patches should we consider before releasing?
 
  I’m fine holding off on a 0.10.0 release if we feel there is
  additional work to be done, but I’d like to at least move forward with
 0.9.4 release.
 
  Thoughts?
 
  -Taylor
 
 
 
 
  --
  Name : 임 정택
  Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter
  : http://twitter.com/heartsavior LinkedIn :
  http://www.linkedin.com/in/heartsavior

[GitHub] storm pull request: STORM-446: Allow superusers to impersonate oth...

2015-03-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/448


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-446) secure Impersonation in storm


[ 
https://issues.apache.org/jira/browse/STORM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358985#comment-14358985
 ] 

ASF GitHub Bot commented on STORM-446:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/448


 secure Impersonation in storm
 -

 Key: STORM-446
 URL: https://issues.apache.org/jira/browse/STORM-446
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Sriharsha Chintalapani
Assignee: Parth Brahmbhatt
  Labels: Security

 Storm security adds features of authenticating with kerberos and than uses 
 that principal and TGT as way to authorize user operations, topology 
 operation. Currently Storm UI user needs to be part of nimbus.admins to get 
 details on user submitted topologies. Ideally storm ui needs to take 
 authenticated user  principal to submit requests to nimbus which will than 
 authorize the user rather than storm UI user. This feature will also benefit 
 superusers to impersonate other users to submit topologies in a secured way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...

2015-03-12 Thread ptgoetz

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/414#issuecomment-78618488
  
@revans2 I agree that it would be great to test forward/backward 
compatibility, but testing that in an automated way would be really difficult 
since it would likely require the introduction of what in previous versions of 
storm would have been a breaking change (clojure structures, etc.). I think 
older client/newer server compatibility is acceptable.

I think this is a big step toward better supporting upgrades, rolling 
upgrades, and continuity of operations for Storm deployments. Nice work 
@Parth-Brahmbhatt, and @revans2 for the thorough review.

+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-615. Add REST API to upload topology.

GitHub user harshach opened a pull request:

https://github.com/apache/storm/pull/464

STORM-615. Add REST API to upload topology.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harshach/incubator-storm STORM-615-V2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #464


commit d18508e6defa4d790002d2512c2905aa6a76
Author: Sriharsha Chintalapani m...@harsha.io
Date:   2015-03-12T20:58:31Z

STORM-615. Add REST API to upload topology.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-615) Add REST API to upload topology


[ 
https://issues.apache.org/jira/browse/STORM-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359397#comment-14359397
 ] 

ASF GitHub Bot commented on STORM-615:
--

GitHub user harshach opened a pull request:

https://github.com/apache/storm/pull/464

STORM-615. Add REST API to upload topology.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harshach/incubator-storm STORM-615-V2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #464


commit d18508e6defa4d790002d2512c2905aa6a76
Author: Sriharsha Chintalapani m...@harsha.io
Date:   2015-03-12T20:58:31Z

STORM-615. Add REST API to upload topology.




 Add REST API to upload topology
 ---

 Key: STORM-615
 URL: https://issues.apache.org/jira/browse/STORM-615
 Project: Apache Storm
  Issue Type: Bug
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani

 Add REST api /api/v1/submitTopology to upload topology jars and config using 
 REST api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.

[
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359420#comment-14359420
]

ASF GitHub Bot commented on STORM-634:
--

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/414#issuecomment-78618488

@revans2 I agree that it would be great to test forward/backward
compatibility, but testing that in an automated way would be really difficult
since it would likely require the introduction of what in previous versions of
storm would have been a breaking change (clojure structures, etc.). I think
older client/newer server compatibility is acceptable.

I think this is a big step toward better supporting upgrades, rolling
upgrades, and continuity of operations for Storm deployments. Nice work
@Parth-Brahmbhatt, and @revans2 for the thorough review.

Storm should support rolling upgrade/downgrade of storm cluster.

Key: STORM-634
URL: https://issues.apache.org/jira/browse/STORM-634
Project: Apache Storm
Issue Type: Improvement
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt

Currently when a new version of storm is released in order to upgrade
existing storm clusters users need to backup their existing topologies , kill
all the topologies , perform the upgrade and resubmit all the topologies.
This is painful and results in downtime which may not be acceptable for
Always alive production systems.
Storm should support a rolling upgrade/downgrade deployment process to avoid
these downtimes and to make the transition to a different version effortless.
Based on my initial attempt the primary issue seem to be the java
serialization used to serialize java classes like StormBase, Assignment,
WorkerHeartbeat which is then stored in zookeeper. When deserializing if the
serial versions do not match the deserialization fails resulting in processes
just getting killed indefinitely. We need to change the Utils/serialize and
Utils/deserialize so it can support non java serialization mechanism like
json.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...

2015-03-12 Thread Parth-Brahmbhatt

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/414#issuecomment-78574474
  
@revans2 All comments addressed and I really appreciate the time you are 
taking to review this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-496) task.clj missing debug for logging spout and bolt emit values

2015-03-12 Thread Sriharsha Chintalapani (JIRA)


 [ 
https://issues.apache.org/jira/browse/STORM-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani resolved STORM-496.
--
Resolution: Fixed

 task.clj missing debug for logging spout and bolt emit values
 -

 Key: STORM-496
 URL: https://issues.apache.org/jira/browse/STORM-496
 Project: Apache Storm
  Issue Type: Bug
Reporter: Sriharsha Chintalapani
Assignee: Sriharsha Chintalapani
  Labels: newbie, security

 task.clj in security branch missing debug config for logging spout , bolt 
 emit values.
 https://github.com/apache/incubator-storm/blob/security/storm-core/src/clj/backtype/storm/daemon/task.clj
 https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L130
 https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L133
 https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/daemon/task.clj#L151



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.


[ 
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359192#comment-14359192
 ] 

ASF GitHub Bot commented on STORM-634:
--

Github user Parth-Brahmbhatt commented on a diff in the pull request:

https://github.com/apache/storm/pull/414#discussion_r26334368
  
--- Diff: 
storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
 ---
@@ -17,11 +17,7 @@
  */
 package backtype.storm.serialization;
--- End diff --

Done. As part of this I also deleted ThriftBridgeSerializer which depended 
on the now deprecated DefaultSerializationDelegate. So we now using 
ThriftSerializationDelegate as the default.


 Storm should support rolling upgrade/downgrade of storm cluster.
 

 Key: STORM-634
 URL: https://issues.apache.org/jira/browse/STORM-634
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt

 Currently when a new version of storm is released in order to upgrade 
 existing storm clusters users need to backup their existing topologies , kill 
 all the topologies , perform the upgrade and resubmit all the topologies. 
 This is painful and results in downtime which may not be acceptable for 
 Always alive  production systems.
 Storm should support a rolling  upgrade/downgrade deployment process to avoid 
 these downtimes and to make the transition to a different version effortless. 
 Based on my initial attempt the primary issue seem to be the java 
 serialization used to serialize java classes like StormBase, Assignment, 
 WorkerHeartbeat which is then stored in zookeeper. When deserializing if the 
 serial versions do not match the deserialization fails resulting in processes 
 just getting killed indefinitely. We need to change the Utils/serialize and 
 Utils/deserialize so it can support non java serialization mechanism like 
 json. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-634: Storm serialization changed to thri...

2015-03-12 Thread Parth-Brahmbhatt

Github user Parth-Brahmbhatt commented on a diff in the pull request:

https://github.com/apache/storm/pull/414#discussion_r26334368
  
--- Diff: 
storm-core/src/jvm/backtype/storm/serialization/DefaultSerializationDelegate.java
 ---
@@ -17,11 +17,7 @@
  */
 package backtype.storm.serialization;
--- End diff --

Done. As part of this I also deleted ThriftBridgeSerializer which depended 
on the now deprecated DefaultSerializationDelegate. So we now using 
ThriftSerializationDelegate as the default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-634) Storm should support rolling upgrade/downgrade of storm cluster.


[ 
https://issues.apache.org/jira/browse/STORM-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359194#comment-14359194
 ] 

ASF GitHub Bot commented on STORM-634:
--

Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/414#issuecomment-78574474
  
@revans2 All comments addressed and I really appreciate the time you are 
taking to review this.



 Storm should support rolling upgrade/downgrade of storm cluster.
 

 Key: STORM-634
 URL: https://issues.apache.org/jira/browse/STORM-634
 Project: Apache Storm
  Issue Type: Improvement
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt

 Currently when a new version of storm is released in order to upgrade 
 existing storm clusters users need to backup their existing topologies , kill 
 all the topologies , perform the upgrade and resubmit all the topologies. 
 This is painful and results in downtime which may not be acceptable for 
 Always alive  production systems.
 Storm should support a rolling  upgrade/downgrade deployment process to avoid 
 these downtimes and to make the transition to a different version effortless. 
 Based on my initial attempt the primary issue seem to be the java 
 serialization used to serialize java classes like StormBase, Assignment, 
 WorkerHeartbeat which is then stored in zookeeper. When deserializing if the 
 serial versions do not match the deserialization fails resulting in processes 
 just getting killed indefinitely. We need to change the Utils/serialize and 
 Utils/deserialize so it can support non java serialization mechanism like 
 json. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-689. SimpleACLAuthorizer should provide ...

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/445#issuecomment-78663596
  
@Parth-Brahmbhatt @revans2 Thanks for the review. I updated the PR to 
include nimbus.groups as well and added doc under SECURITY.md. Please take a 
look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-670) [storm-kafka] Restore Java 1.6 compatibility

2015-03-12 Thread Robert Joseph Evans (JIRA)


 [ 
https://issues.apache.org/jira/browse/STORM-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans resolved STORM-670.
---
   Resolution: Fixed
Fix Version/s: 0.10.0

Thanks [~ptgoetz],

I merged this into master.

 [storm-kafka] Restore Java 1.6 compatibility
 

 Key: STORM-670
 URL: https://issues.apache.org/jira/browse/STORM-670
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-kafka
Affects Versions: 0.10.0
Reporter: P. Taylor Goetz
Assignee: P. Taylor Goetz
 Fix For: 0.10.0


 java.lang.Long.compare(Long, Long) is only available in Java 1.7



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-682: supervisor should handle worker sta...

2015-03-12 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/437


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.


[ 
https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359579#comment-14359579
 ] 

ASF GitHub Bot commented on STORM-682:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/437


 Supervisor local worker state corrupted and failing to start.
 -

 Key: STORM-682
 URL: https://issues.apache.org/jira/browse/STORM-682
 Project: Apache Storm
  Issue Type: Bug
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt

 If supervisor's cleanup of a worker fails to delete some heartbeat files the 
 local state of the supervisors get corrupted.The only way to recover the 
 supervisor from this state is to delete the local state folder where 
 supervisor stores all worker information.This fix can get very cumbersome if 
 it happens on multiple worker nodes.
 The root cause of the issue is the order in which worker heartbeat versioned 
 store files are created vs the deletion order of those files. LocalState.put 
 first creates a data file X and then marks a success by creating a file 
 X.version.  During get it first checks for all *.version files , tries to 
 find the largest value of X and then issues a read against X. See the below 
 pseudo code
 {code:java}
 start_supervisor() {
 workerIds = `ls local-state/workers`
 for each workerId in workerIds
  versions =  `ls local-state/workers/workerId/heartbeats/*.version`
  latest_version = max(versions)
  read  local-state/workers/workerId/heartbeats/latest_version [Note there 
 is no .version extension] 
 }
 {code}
 During cleanup it first tries to delete file X and then X.version. If X gets 
 deleted  but X.version fails to delete the supervisor fails to start with 
 FileNotFoundException in the code above. 
 We propose to change the deletion order so the .version files get deleted 
 before the data file and catch any IOException when reading worker heartbeats 
 to avoid supervisor failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-704) Apply Travis CI

Jungtaek Lim created STORM-704:
--

 Summary: Apply Travis CI
 Key: STORM-704
 URL: https://issues.apache.org/jira/browse/STORM-704
 Project: Apache Storm
  Issue Type: Improvement
 Environment: Travis CI
Reporter: Jungtaek Lim
Priority: Minor


Now Apache Storm takes advantage of Github, we can apply Travis CI to some more 
advantages.

- Build matrix
-- Travis CI supports various JDK versions (openjdk6, openjdk7, oraclejdk7, 
oraclejdk8), and it can be tested separately.
- Build automatically
-- pushed new commits, any new PRs
- Integrated with Github
-- Contributors can see his/her PR breaks compilation / test in some minutes.
-- If he/she adds commits to PR, Travis builds it automatically and update 
build result.

Please see [https://travis-ci.org/xetorthio/jedis] for example.

There're some hurdles applying Travis CI to Apache Storm project, but we can 
overcome these and finally get great CI.

Current hurdles
- asfgit should manage Travis CI setup for the first time
-- other Apache projects already did it by requesting it to INFRA
--- ex. [https://issues.apache.org/jira/browse/INFRA-6161]
- Travis CI restricts stdout with 4M which is too small for Storm maven output.
-- alternative way : filter Storm's INFO message by {code}mvn clean test -U | 
egrep -v [0-9]+ \[.+\] INFO{code} 
--- inspired by [https://github.com/apache/tajo/pull/8]
- In storm-core, we can't see tests failure information on stdout cause it just 
prints 'clojure failed'.
-- We need to find a way to upload surefire / clojure tests report files to 
somewhere, and uploaded files should be visible easily.
--- Travis supports uploading artifacts to S3 by 
[https://github.com/travis-ci/artifacts], but S3 is not free.
-- I'm not familiar with Clojure, but can we print tests summary with clojure 
tests as same as Java junit tests?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-704) Apply Travis CI

[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359730#comment-14359730
]

Jungtaek Lim edited comment on STORM-704 at 3/13/15 1:12 AM:
-

I'm trying this now.
https://travis-ci.org/HeartSaVioR/storm

Any helps are greatly appreciated!

was (Author: kabhwan):
I'm trying this now.

Apply Travis CI
---

Key: STORM-704
URL: https://issues.apache.org/jira/browse/STORM-704
Project: Apache Storm
Issue Type: Improvement
Environment: Travis CI
Reporter: Jungtaek Lim
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-704) Apply Travis CI

[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jungtaek Lim reassigned STORM-704:
--

Assignee: Jungtaek Lim

Apply Travis CI
---

Key: STORM-704
URL: https://issues.apache.org/jira/browse/STORM-704
Project: Apache Storm
Issue Type: Improvement
Environment: Travis CI
Reporter: Jungtaek Lim
Assignee: Jungtaek Lim
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-704) Apply Travis CI

[
https://issues.apache.org/jira/browse/STORM-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359730#comment-14359730
]

Jungtaek Lim commented on STORM-704:

I'm trying this now.

Apply Travis CI
---

Key: STORM-704
URL: https://issues.apache.org/jira/browse/STORM-704
Project: Apache Storm
Issue Type: Improvement
Environment: Travis CI
Reporter: Jungtaek Lim
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-625: don't leak netty clients when worke...