[jira] [Created] (STORM-1053) Correct storm-kafka readme file because of using new kafka-clients api

2015-09-18 Thread Xin Wang (JIRA) 
Xin Wang created STORM-1053:
---

 Summary: Correct storm-kafka readme file because of using new 
kafka-clients api
 Key: STORM-1053
 URL: https://issues.apache.org/jira/browse/STORM-1053
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-kafka
Reporter: Xin Wang
Assignee: Xin Wang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1053) Correct storm-kafka readme file because of using new kafka-clients api

2015-09-18 Thread Xin Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xin Wang updated STORM-1053:

Priority: Minor  (was: Major)

> Correct storm-kafka readme file because of using new kafka-clients api
> --
>
> Key: STORM-1053
> URL: https://issues.apache.org/jira/browse/STORM-1053
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1053] Correct storm-kafka readme file b...

2015-09-18 Thread vesense 
GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/744

[STORM-1053] Correct storm-kafka readme file because of using new 
kafka-clients api

correct readme file: new producer link & configs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm patch-7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #744


commit be7f2ad1d9b6be1a92fb2a8775822b5752703c89
Author: Xin Wang 
Date:   2015-09-18T07:19:44Z

correct storm-kafka readme file

correct readme file: link & configs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1053) Correct storm-kafka readme file because of using new kafka-clients api

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14805118#comment-14805118
 ] 

ASF GitHub Bot commented on STORM-1053:
---

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/744

[STORM-1053] Correct storm-kafka readme file because of using new 
kafka-clients api

correct readme file: new producer link & configs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm patch-7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/744.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #744


commit be7f2ad1d9b6be1a92fb2a8775822b5752703c89
Author: Xin Wang 
Date:   2015-09-18T07:19:44Z

correct storm-kafka readme file

correct readme file: link & configs




> Correct storm-kafka readme file because of using new kafka-clients api
> --
>
> Key: STORM-1053
> URL: https://issues.apache.org/jira/browse/STORM-1053
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Exception using custom 0.9.6 RPM

2015-09-18 Thread abe oppenheim 
Hi,

We installed a custom RPM built from the 0.9.x branch in an attempt to fix
this bug:

https://issues.apache.org/jira/browse/STORM-763

But when running on the new installation, we see the below stacktrace.

Any recommendations of how to fix the issue we are seeing, or if/when a
stable release of 0.9.6 will be available?

thanks!

Connection failed
Netty-Client-myserver.mydomain.com/10.22.16.170:6701java.lang.NullPointerException
:
null
at backtype.storm.messaging.netty.Client.flushMessages(Client.java:310)
~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
backtype.storm.messaging.netty.Client.notifyInterestChanged(Client.java:439)
~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
backtype.storm.messaging.netty.StormClientHandler.channelInterestChanged(StormClientHandler.java:36)
~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:200)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.Channels.fireChannelInterestChanged(Channels.java:358)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:310)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:204)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:186)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[na:1.7.0_11]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[na:1.7.0_11]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]

[GitHub] storm pull request: STORM-1012 STORM-967 STORM-922 STORM-1042 Shad...

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/741#issuecomment-141475150
  
@HeartSaVioR Great I will upmerge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1012) Shade Jackson dependency

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875726#comment-14875726
 ] 

ASF GitHub Bot commented on STORM-1012:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/741#issuecomment-141475150
  
@HeartSaVioR Great I will upmerge.


> Shade Jackson dependency
> 
>
> Key: STORM-1012
> URL: https://issues.apache.org/jira/browse/STORM-1012
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.11.0
>
>
> Shading jackson dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread HeartSaVioR 
Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/742#issuecomment-141480621
  
@schonfeld 
Could you post a new PR which points to master, too?
Thanks in advance!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875757#comment-14875757
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/742#issuecomment-141480621
  
@schonfeld 
Could you post a new PR which points to master, too?
Thanks in advance!


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Exception using custom 0.9.6 RPM

2015-09-18 Thread P. Taylor Goetz 
Hi Abe,

We should have a 0.9.6 release within a week or so.

-Taylor

> On Sep 18, 2015, at 10:53 AM, abe oppenheim  wrote:
> 
> Hi,
> 
> We installed a custom RPM built from the 0.9.x branch in an attempt to fix
> this bug:
> 
> https://issues.apache.org/jira/browse/STORM-763
> 
> But when running on the new installation, we see the below stacktrace.
> 
> Any recommendations of how to fix the issue we are seeing, or if/when a
> stable release of 0.9.6 will be available?
> 
> thanks!
> 
> Connection failed
> Netty-Client-myserver.mydomain.com/10.22.16.170:6701java.lang.NullPointerException
> :
> null
> at backtype.storm.messaging.netty.Client.flushMessages(Client.java:310)
> ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> backtype.storm.messaging.netty.Client.notifyInterestChanged(Client.java:439)
> ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> backtype.storm.messaging.netty.StormClientHandler.channelInterestChanged(StormClientHandler.java:36)
> ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:200)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.Channels.fireChannelInterestChanged(Channels.java:358)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:310)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:204)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:186)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> [na:1.7.0_11]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> [na:1.7.0_11]
> at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]



signature.asc
Description: Message signed with OpenPGP using GPGMail

[jira] [Commented] (STORM-1012) Shade Jackson dependency

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875778#comment-14875778
 ] 

ASF GitHub Bot commented on STORM-1012:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/741#issuecomment-141483057
  
Upmerged so we should be good to go on this pull request.


> Shade Jackson dependency
> 
>
> Key: STORM-1012
> URL: https://issues.apache.org/jira/browse/STORM-1012
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
> Fix For: 0.11.0
>
>
> Shading jackson dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1012 STORM-967 STORM-922 STORM-1042 Shad...

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/741#issuecomment-141483057
  
Upmerged so we should be good to go on this pull request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Disruptor Queue Batching (DON'T MERGE YET)

2015-09-18 Thread revans2 
Github user revans2 closed the pull request at:

https://github.com/apache/storm/pull/704


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Disruptor Queue Batching (DON'T MERGE YET)

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/704#issuecomment-141484458
  
I have done a lot of testing on this, and we need batching, but this is not 
the correct way to do it.  I have a proof of concept with some micro 
benchmarks, but I am going to probably do a pull request for a fixed version 
shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141491743
  
@zhuoliu looks like you missed adding in a file, We also need to upmerge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875828#comment-14875828
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141491743
  
@zhuoliu looks like you missed adding in a file, We also need to upmerge.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39871444
  
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -602,6 +640,7 @@
 (log-message "Activating spout " component-id ":" 
(keys task-datas))
 (fast-list-iter [^ISpout spout spouts] (.activate 
spout)))

+;; (log-message "Spout executor " (:executor-id 
executor-data) " found throttle-on, now suspends sending tuples")
--- End diff --

Can we remove this line? it looks like it is no longer needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875833#comment-14875833
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39871444
  
--- Diff: storm-core/src/clj/backtype/storm/daemon/executor.clj ---
@@ -602,6 +640,7 @@
 (log-message "Activating spout " component-id ":" 
(keys task-datas))
 (fast-list-iter [^ISpout spout spouts] (.activate 
spout)))

+;; (log-message "Spout executor " (:executor-id 
executor-data) " found throttle-on, now suspends sending tuples")
--- End diff --

Can we remove this line? it looks like it is no longer needed.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875854#comment-14875854
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872212
  
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -1024,6 +1024,48 @@
 
 
 /**
+ * Whether to enable backpressure in for a certain topology
+ */
+public static final String TOPOLOGY_BACKPRESSURE_ENABLE = 
"topology.backpressure.enable";
+public static final Object TOPOLOGY_BACKPRESSURE_ENABLE_SCHEMA = 
Boolean.class;
+
+/**
+ * This signifies the tuple congestion in a worker's out-going queue.
+ * When the used ratio of a worker's outgoing queue is higher than the 
high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple 
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+public static final String 
BACKPRESSURE_WORKER_HIGH_WATERMARK="backpressure.worker.high.watermark";
+public static final Object BACKPRESSURE_WORKER_HIGH_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies a state that a worker has left the congestion.
+ * If the used ratio of a worker's outgoing queue is lower than the 
low watermark,
+ * it notifies the worker to check whether all its executors have also 
left congestion,
+ * if yes, it will unset the worker's backpressure flag on the 
Zookeeper
+ */
+public static final String 
BACKPRESSURE_WORKER_LOW_WATERMARK="backpressure.worker.low.watermark";
+public static final Object BACKPRESSURE_WORKER_LOW_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies the tuple congestion in an executor's receiving 
queue.
+ * When the used ratio of an executor's receiving queue is higher than 
the high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple 
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+public static final String 
BACKPRESSURE_EXECUTOR_HIGH_WATERMARK="backpressure.executor.high.watermark";
+public static final Object BACKPRESSURE_EXECUTOR_HIGH_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies a state that an executor has left the congestion.
+ * If the used ratio of an execuotr's receive queue is lower than the 
low watermark,
+ * it may notify the worker to check whether all its executors have 
also left congestion,
+ * if yes, the worker's backpressure flag will be unset on the 
Zookeeper
+ */
+public static final String 
BACKPRESSURE_EXECUTOR_LOW_WATERMARK="backpressure.executor.low.watermark";
+public static final Object BACKPRESSURE_EXECUTOR_LOW_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
--- End diff --

Looking through the code it looks like these two configs are never used.

BACKPRESSURE_EXECUTOR_HIGH_WATERMARK and 
BACKPRESSURE_EXECUTOR_LOW_WATERMARK.  I don't see a reason to have different 
configs for different queues.  Perhaps we can just remove these configs. and 
make the description for the other configs more generic.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atla

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872212
  
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -1024,6 +1024,48 @@
 
 
 /**
+ * Whether to enable backpressure in for a certain topology
+ */
+public static final String TOPOLOGY_BACKPRESSURE_ENABLE = 
"topology.backpressure.enable";
+public static final Object TOPOLOGY_BACKPRESSURE_ENABLE_SCHEMA = 
Boolean.class;
+
+/**
+ * This signifies the tuple congestion in a worker's out-going queue.
+ * When the used ratio of a worker's outgoing queue is higher than the 
high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple 
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+public static final String 
BACKPRESSURE_WORKER_HIGH_WATERMARK="backpressure.worker.high.watermark";
+public static final Object BACKPRESSURE_WORKER_HIGH_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies a state that a worker has left the congestion.
+ * If the used ratio of a worker's outgoing queue is lower than the 
low watermark,
+ * it notifies the worker to check whether all its executors have also 
left congestion,
+ * if yes, it will unset the worker's backpressure flag on the 
Zookeeper
+ */
+public static final String 
BACKPRESSURE_WORKER_LOW_WATERMARK="backpressure.worker.low.watermark";
+public static final Object BACKPRESSURE_WORKER_LOW_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies the tuple congestion in an executor's receiving 
queue.
+ * When the used ratio of an executor's receiving queue is higher than 
the high watermark,
+ * the backpressure scheme, if enabled, should slow down the tuple 
sending speed of
+ * the spouts until reaching the low watermark.
+ */
+public static final String 
BACKPRESSURE_EXECUTOR_HIGH_WATERMARK="backpressure.executor.high.watermark";
+public static final Object BACKPRESSURE_EXECUTOR_HIGH_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
+
+/**
+ * This signifies a state that an executor has left the congestion.
+ * If the used ratio of an execuotr's receive queue is lower than the 
low watermark,
+ * it may notify the worker to check whether all its executors have 
also left congestion,
+ * if yes, the worker's backpressure flag will be unset on the 
Zookeeper
+ */
+public static final String 
BACKPRESSURE_EXECUTOR_LOW_WATERMARK="backpressure.executor.low.watermark";
+public static final Object BACKPRESSURE_EXECUTOR_LOW_WATERMARK_SCHEMA 
=ConfigValidation.PositiveNumberValidator;
--- End diff --

Looking through the code it looks like these two configs are never used.

BACKPRESSURE_EXECUTOR_HIGH_WATERMARK and 
BACKPRESSURE_EXECUTOR_LOW_WATERMARK.  I don't see a reason to have different 
configs for different queues.  Perhaps we can just remove these configs. and 
make the description for the other configs more generic.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872538
  
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -138,7 +150,21 @@ private void consumeBatchToCursor(long cursor, 
EventHandler handler) {
 //TODO: only set this if the consumer cursor has changed?
 _consumer.set(cursor);
 }
-
+
+public void registerBackpressureCallback(DisruptorBackpressureCallback 
cb) {
+this._cb = cb;
+}
+
+static public void notifyBackpressureChecker(Object trigger) {
--- End diff --

I don't think this code really belongs here.  It would make more since to 
move it to the WorkerBackpressureThread itself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875862#comment-14875862
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872538
  
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -138,7 +150,21 @@ private void consumeBatchToCursor(long cursor, 
EventHandler handler) {
 //TODO: only set this if the consumer cursor has changed?
 _consumer.set(cursor);
 }
-
+
+public void registerBackpressureCallback(DisruptorBackpressureCallback 
cb) {
+this._cb = cb;
+}
+
+static public void notifyBackpressureChecker(Object trigger) {
--- End diff --

I don't think this code really belongs here.  It would make more since to 
move it to the WorkerBackpressureThread itself.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875866#comment-14875866
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872683
  
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -35,6 +35,7 @@
 import java.util.HashMap;
 import java.util.Map;
 import backtype.storm.metric.api.IStatefulObject;
+import org.jgrapht.graph.DirectedSubgraph;
--- End diff --

Where did this come from?


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/700#discussion_r39872683
  
--- Diff: storm-core/src/jvm/backtype/storm/utils/DisruptorQueue.java ---
@@ -35,6 +35,7 @@
 import java.util.HashMap;
 import java.util.Map;
 import backtype.storm.metric.api.IStatefulObject;
+import org.jgrapht.graph.DirectedSubgraph;
--- End diff --

Where did this come from?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141499766
  
For the most part things look really good.  I would also love to see 
something added to Examples.  Like a Word Count that just goes as fast as it 
can.  That way we can see before, with acking disabled it crashes, after with 
acking disabled it stays up and can run really well.  I can help with code for 
that if you want.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875903#comment-14875903
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141499766
  
For the most part things look really good.  I would also love to see 
something added to Examples.  Like a Word Count that just goes as fast as it 
can.  That way we can see before, with acking disabled it crashes, after with 
acking disabled it stays up and can run really well.  I can help with code for 
that if you want.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
GitHub user schonfeld opened a pull request:

https://github.com/apache/storm/pull/745

[STORM-1051] Fix for flushMessagse NPE

STORM-763 introduced a situation in which flushMessages could receive batch 
= null, in which case, an NPE is thrown because we weren't validating != null

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schonfeld/storm master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #745


commit 8b845392e920d753eb24fcffead0263416bc6227
Author: Michael Schonfeld 
Date:   2015-09-18T16:31:32Z

[STORM-1051] Fix for flushMessagse NPE




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld closed the pull request at:

https://github.com/apache/storm/pull/742


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875907#comment-14875907
 ] 

ASF GitHub Bot commented on STORM-1051:
---

GitHub user schonfeld opened a pull request:

https://github.com/apache/storm/pull/745

[STORM-1051] Fix for flushMessagse NPE

STORM-763 introduced a situation in which flushMessages could receive batch 
= null, in which case, an NPE is thrown because we weren't validating != null

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/schonfeld/storm master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/745.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #745


commit 8b845392e920d753eb24fcffead0263416bc6227
Author: Michael Schonfeld 
Date:   2015-09-18T16:31:32Z

[STORM-1051] Fix for flushMessagse NPE




> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875910#comment-14875910
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld closed the pull request at:

https://github.com/apache/storm/pull/742


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875909#comment-14875909
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/742#issuecomment-141500177
  
Closing in favor of PR #745 


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/742#issuecomment-141500177
  
Closing in favor of PR #745 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Netty Zlib Encoding Decoding

2015-09-18 Thread Onur Yalazı 

Hello,

I'm very new to storm world and the list, so Hello from Turkey.

Because of a recent incident we had to increase our openstack network 
bandwidth soft limits from 1gb/s to 2gb/s.


And of course even though the problem resides in our tuples' size and 
topology size, I thought if storm's netty was using ZlibEncoding. 
Looking into backtype.storm.messaging.netty.StormServerPipelineFactory 
and client counterpart, pipline seems to not have any compression handlers.


Is it an intentional decision to not include compression in the 
pipeline? I know it would need more processing power and reduce topology 
performance, but I would like to know it it was considered before, and 
if not raise the issue.


Thank you.

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876011#comment-14876011
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user zhuoliu commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141514572
  
Thanks, Bobby @revans2 . I addressed all the comments. 
Actually I have such examples already written and tested. See:
 
https://github.com/zhuoliu/storm/blob/888/examples/storm-starter/src/jvm/storm/starter/WordCountTopology2.java
I can check them in to master if needed. 


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread zhuoliu 
Github user zhuoliu commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141514572
  
Thanks, Bobby @revans2 . I addressed all the comments. 
Actually I have such examples already written and tested. See:
 
https://github.com/zhuoliu/storm/blob/888/examples/storm-starter/src/jvm/storm/starter/WordCountTopology2.java
I can check them in to master if needed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Netty Zlib Encoding Decoding

2015-09-18 Thread Bobby Evans 
Compression was just not something that we really though about all that much.  
The fastest route is probably to replace the tuple serializer with one that can 
handle compression.  We did something similar for encryption.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/security/serialization/BlowfishTupleSerializer.java
But compression is generic enough it might be nice to make it a part of the 
Real TupleSerializer.
https://github.com/EsotericSoftware/kryo#compression-and-encryption

I would also suggest that you look at Snappy and LZO or LZ4 for your 
compression.  As they tend to be much faster and still get good compression 
ratios. - Bobby 


 On Friday, September 18, 2015 11:51 AM, Onur Yalazı 
 wrote:
   

 Hello,

I'm very new to storm world and the list, so Hello from Turkey.

Because of a recent incident we had to increase our openstack network 
bandwidth soft limits from 1gb/s to 2gb/s.

And of course even though the problem resides in our tuples' size and 
topology size, I thought if storm's netty was using ZlibEncoding. 
Looking into backtype.storm.messaging.netty.StormServerPipelineFactory 
and client counterpart, pipline seems to not have any compression handlers.

Is it an intentional decision to not include compression in the 
pipeline? I know it would need more processing power and reduce topology 
performance, but I would like to know it it was considered before, and 
if not raise the issue.

Thank you.

Re: Netty Zlib Encoding Decoding

2015-09-18 Thread Onur Yalazı 

Hello Bobby,

If it's possible to enable a TupleSerializer while submitting 
topologies, without touching Storm's internals it would be really wise 
and a fast way to implement compression. Actually I have no idea if it's 
possible this way.


Other than that, If I got the Netty's gist right, to enable Zlib 
compression one needs only 2 lines of code for each pipeline to enable 
it. It's first class citizen of netty. Looking into 
https://github.com/netty/netty/tree/master/codec/src/main/java/io/netty/handler/codec/compression 
a good chunk of compression algorithms are also available.



From 
https://github.com/netty/netty/blob/ed4a89082bb29b9e7d869c5d25d6b9ea8fc9d25b/example/src/main/java/io/netty/example/factorial/FactorialClientInitializer.java:



 //  Enable stream compression (you can remove these two if 
unnecessary)

pipeline.addLast(ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP));
pipeline.addLast(ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP));



On 09/18/2015 08:32 PM, Bobby Evans wrote:

Compression was just not something that we really though about all that much.  
The fastest route is probably to replace the tuple serializer with one that can 
handle compression.  We did something similar for encryption.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/security/serialization/BlowfishTupleSerializer.java
But compression is generic enough it might be nice to make it a part of the 
Real TupleSerializer.
https://github.com/EsotericSoftware/kryo#compression-and-encryption

I would also suggest that you look at Snappy and LZO or LZ4 for your 
compression.  As they tend to be much faster and still get good compression 
ratios. - Bobby


  On Friday, September 18, 2015 11:51 AM, Onur Yalazı 
 wrote:



  Hello,

I'm very new to storm world and the list, so Hello from Turkey.

Because of a recent incident we had to increase our openstack network
bandwidth soft limits from 1gb/s to 2gb/s.

And of course even though the problem resides in our tuples' size and
topology size, I thought if storm's netty was using ZlibEncoding.
Looking into backtype.storm.messaging.netty.StormServerPipelineFactory
and client counterpart, pipline seems to not have any compression handlers.

Is it an intentional decision to not include compression in the
pipeline? I know it would need more processing power and reduce topology
performance, but I would like to know it it was considered before, and
if not raise the issue.

Thank you.

Re: Netty Zlib Encoding Decoding

2015-09-18 Thread Bobby Evans 
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/Config.java#L212-L217
 topology.tuple.serializer

- Bobby 


 On Friday, September 18, 2015 12:54 PM, Onur Yalazı 
 wrote:
   

 Hello Bobby,

If it's possible to enable a TupleSerializer while submitting 
topologies, without touching Storm's internals it would be really wise 
and a fast way to implement compression. Actually I have no idea if it's 
possible this way.

Other than that, If I got the Netty's gist right, to enable Zlib 
compression one needs only 2 lines of code for each pipeline to enable 
it. It's first class citizen of netty. Looking into 
https://github.com/netty/netty/tree/master/codec/src/main/java/io/netty/handler/codec/compression
 
a good chunk of compression algorithms are also available.


From 
https://github.com/netty/netty/blob/ed4a89082bb29b9e7d869c5d25d6b9ea8fc9d25b/example/src/main/java/io/netty/example/factorial/FactorialClientInitializer.java:


      //  Enable stream compression (you can remove these two if 
unnecessary)
pipeline.addLast(ZlibCodecFactory.newZlibEncoder(ZlibWrapper.GZIP));
pipeline.addLast(ZlibCodecFactory.newZlibDecoder(ZlibWrapper.GZIP));



On 09/18/2015 08:32 PM, Bobby Evans wrote:
> Compression was just not something that we really though about all that much. 
>  The fastest route is probably to replace the tuple serializer with one that 
> can handle compression.  We did something similar for encryption.
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/security/serialization/BlowfishTupleSerializer.java
> But compression is generic enough it might be nice to make it a part of the 
> Real TupleSerializer.
> https://github.com/EsotericSoftware/kryo#compression-and-encryption
>
> I would also suggest that you look at Snappy and LZO or LZ4 for your 
> compression.  As they tend to be much faster and still get good compression 
> ratios. - Bobby
>
>
>      On Friday, September 18, 2015 11:51 AM, Onur Yalazı 
> wrote:
>    
>
>  Hello,
>
> I'm very new to storm world and the list, so Hello from Turkey.
>
> Because of a recent incident we had to increase our openstack network
> bandwidth soft limits from 1gb/s to 2gb/s.
>
> And of course even though the problem resides in our tuples' size and
> topology size, I thought if storm's netty was using ZlibEncoding.
> Looking into backtype.storm.messaging.netty.StormServerPipelineFactory
> and client counterpart, pipline seems to not have any compression handlers.
>
> Is it an intentional decision to not include compression in the
> pipeline? I know it would need more processing power and reduce topology
> performance, but I would like to know it it was considered before, and
> if not raise the issue.
>
> Thank you.
>
>
>   

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread erikdw 
Github user erikdw commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141532453
  
@revans2 & @ptgoetz I know y'all are busy, but is there anything I can do 
to get this minor (yah, "minor", I know, right?) change reviewed?  The Mesos 
Apache project has a "shepherd" process, is there something similar for Storm?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread Parth-Brahmbhatt 
Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141532910
  
+1. sorry for the delay.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread Parth-Brahmbhatt 
Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141532980
  
will wait for 24 hours as per our Bylaws and merge it in tomorrow. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-893] Resource Aware Scheduling

2015-09-18 Thread jerrypeng 
GitHub user jerrypeng opened a pull request:

https://github.com/apache/storm/pull/746

[STORM-893] Resource Aware Scheduling

I have created a initial open source implementation of the Resource Aware 
Scheduler as described in the paper I published: 

web.engr.illinois.edu/~bpeng/files/r-storm.pdf

The paper describes the general architecture, concepts, and algorithms used.

I have written an example topology that demonstrates how to use the API I 
have written to specify resource requirements in your topology.  Currently the 
user can only specify two types of resources: Memory and CPU.  We plan on 
adding support for more resources in the future.

Currently, there is no built in enforcement of resource usage in worker 
processes, however, we plan to add that functionality via CGroups.

People that worked on the implementation at Yahoo with me:

Bobby Evans (Yahoo & Storm PMC)
Derek Dagit (Yahoo & Storm PMC)
Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)
Sanket Chintapalli (Yahoo)
Reza Fravier (Yahoo & UIUC)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerrypeng/storm opensource_ras

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #746


commit e014804a68b9c6887c64b9d03a17929863b80909
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:50:44Z

[STORM-893] - Resource Aware Scheduler implementation
[STORM-894] - Basic Resource Aware Scheduling implementation.

commit 0f3f237f158218cbe46ffc59670f300875eb7950
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:02:44Z

Added functionality for users to limit the amount of memory resources 
allocated to a worker (JVM) process when scheduling with resource aware 
scheduler. This allows users to potentially spread executors more evenly across 
workers.
Also refactored some code

commit 28ee867c5a27c220be562689e61f4bb959a1aa62
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:56:23Z

regenerated thrift

commit 0256f6baff63e8c394ebfddbb68d38de5893b053
Author: Boyang Jerry Peng 
Date:   2015-09-18T18:23:46Z

adding miscellaneous things




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-893) Resource Aware Scheduling

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876154#comment-14876154
 ] 

ASF GitHub Bot commented on STORM-893:
--

GitHub user jerrypeng opened a pull request:

https://github.com/apache/storm/pull/746

[STORM-893] Resource Aware Scheduling

I have created a initial open source implementation of the Resource Aware 
Scheduler as described in the paper I published: 

web.engr.illinois.edu/~bpeng/files/r-storm.pdf

The paper describes the general architecture, concepts, and algorithms used.

I have written an example topology that demonstrates how to use the API I 
have written to specify resource requirements in your topology.  Currently the 
user can only specify two types of resources: Memory and CPU.  We plan on 
adding support for more resources in the future.

Currently, there is no built in enforcement of resource usage in worker 
processes, however, we plan to add that functionality via CGroups.

People that worked on the implementation at Yahoo with me:

Bobby Evans (Yahoo & Storm PMC)
Derek Dagit (Yahoo & Storm PMC)
Kyle Nusbaum (Yahoo & Storm PMC)
Liu Zhuo (Yahoo)
Sanket Chintapalli (Yahoo)
Reza Fravier (Yahoo & UIUC)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerrypeng/storm opensource_ras

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/746.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #746


commit e014804a68b9c6887c64b9d03a17929863b80909
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:50:44Z

[STORM-893] - Resource Aware Scheduler implementation
[STORM-894] - Basic Resource Aware Scheduling implementation.

commit 0f3f237f158218cbe46ffc59670f300875eb7950
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:02:44Z

Added functionality for users to limit the amount of memory resources 
allocated to a worker (JVM) process when scheduling with resource aware 
scheduler. This allows users to potentially spread executors more evenly across 
workers.
Also refactored some code

commit 28ee867c5a27c220be562689e61f4bb959a1aa62
Author: Boyang Jerry Peng 
Date:   2015-09-17T20:56:23Z

regenerated thrift

commit 0256f6baff63e8c394ebfddbb68d38de5893b053
Author: Boyang Jerry Peng 
Date:   2015-09-18T18:23:46Z

adding miscellaneous things




> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141541329
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141544473
  
@zhuoliu I looked over the code, and I ran some tests with a word count 
program that does not sleep, and there are not enough split sentence bolts.  
This kept the workers from crashing, which I verified happened without it.  It 
was able to keep a throughput similar to setting a decent value for max spout 
pending, but with a larger latency.

I am +1 for merging this in.  Great work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876201#comment-14876201
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/700#issuecomment-141544473
  
@zhuoliu I looked over the code, and I ran some tests with a word count 
program that does not sleep, and there are not enough split sentence bolts.  
This kept the workers from crashing, which I verified happened without it.  It 
was able to keep a throughput similar to setting a decent value for max spout 
pending, but with a larger latency.

I am +1 for merging this in.  Great work.


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-886] Automatic Back Pressure (ABP)

2015-09-18 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/700


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-886) Automatic Back Pressure

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876214#comment-14876214
 ] 

ASF GitHub Bot commented on STORM-886:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/700


> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-886) Automatic Back Pressure

2015-09-18 Thread Robert Joseph Evans (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans resolved STORM-886.
---
   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~zhuoliu],

Sorry it took so long to review this.  The code looks really good, and it works 
really well to prevent a topology from crashing because of GC.  Big win for 
storm.

> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread erikdw 
Github user erikdw commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141545439
  
Thanks guys!  Can I help with backporting it to the 0.9.6 train?  I sent a 
PR to that branch, but I feel like that might been counterproductive?  Also, is 
there any chance of getting this into 0.10?  Not sure how the release processes 
are handled, but happy to become more knowledgeable and involved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141545624
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876221#comment-14876221
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141545624
  
+1


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread revans2 
Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141547493
  
@erikdw This is a small enough change that I personally would feel OK with 
pulling this into the 0.10 release branch, but I would want at least one other 
committer to also agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: configurable supervisor log filename

2015-09-18 Thread Parth-Brahmbhatt 
Github user Parth-Brahmbhatt commented on the pull request:

https://github.com/apache/storm/pull/733#issuecomment-141549873
  
+1 on merging into 0.10


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-893) Resource Aware Scheduling

2015-09-18 Thread Boyang Jerry Peng (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Jerry Peng updated STORM-893:

Attachment: resource_aware_scheduler_api.pdf

Document explaining how to use the API in Resource Aware Scheduler

> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: resource_aware_scheduler_api.pdf
>
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-893) Resource Aware Scheduling

2015-09-18 Thread Boyang Jerry Peng (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876237#comment-14876237
 ] 

Boyang Jerry Peng edited comment on STORM-893 at 9/18/15 7:53 PM:
--

uploaded a document explaining how to use the API in Resource Aware Scheduler


was (Author: jerrypeng):
Document explaining how to use the API in Resource Aware Scheduler

> Resource Aware Scheduling
> -
>
> Key: STORM-893
> URL: https://issues.apache.org/jira/browse/STORM-893
> Project: Apache Storm
>  Issue Type: Umbrella
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: resource_aware_scheduler_api.pdf
>
>
> At Yahoo we have been working on resource aware scheduling in storm, based 
> off of some work done in academia.  This rollup ticket is to track the 
> complete project.  With several sub tasks.  Some that are already done and 
> need to be pushed back, and others that we have not started on yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-886) Automatic Back Pressure

2015-09-18 Thread Zhuo Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhuo Liu updated STORM-886:
---
Attachment: aSimpleExampleOfBackpressure.png

> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-886) Automatic Back Pressure

2015-09-18 Thread Zhuo Liu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhuo Liu updated STORM-886:
---
Attachment: (was: an simple example for backpressure.png)

> Automatic Back Pressure
> ---
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Robert Joseph Evans
>Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG 
> since different components may have unmatched tuple processing speed. 
> Currently, the tuples may get dropped if the downstream components can not 
> process as quickly, thereby causing a waste of network bandwidth and 
> processing capability. In addition, it is difficult to tune the 
> max.spout.pending parameter for best backpressure performance. Therefore, an 
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max 
> spout pending.  Instead spouts throttle not only when max.spout.pending is 
> hit, but also if any bolt has gone over a high water mark in their input 
> queue, and has not yet gone below a low water mark again.  There is a lot of 
> room for potential improvement here around control theory and having spouts 
> only respond to downstream bolts backing up, but a simple bang-bang 
> controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876541#comment-14876541
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141580360
  
+1


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread HeartSaVioR 
Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141580360
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1051:

Priority: Critical  (was: Major)

> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread Jungtaek Lim (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876667#comment-14876667
 ] 

Jungtaek Lim commented on STORM-1051:
-

It could occur easily, and the result is shown as strange worker dying. So 
changing priority to critical.

> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Exception using custom 0.9.6 RPM

2015-09-18 Thread 임정택 
The issue that Abe addressed is a kind of side effect of bugfix.
(Bugfix introduces another bug.)

- STORM-763  was
backported to all lines including 0.9.x line.
- STORM-1051 , fixes bug
Abe addressed, is side effect of STORM-763.

We should apply STORM-1051 to all version lines before releasing 0.9.6.


2015-09-19 0:25 GMT+09:00 P. Taylor Goetz :

> Hi Abe,
>
> We should have a 0.9.6 release within a week or so.
>
> -Taylor
>
> > On Sep 18, 2015, at 10:53 AM, abe oppenheim 
> wrote:
> >
> > Hi,
> >
> > We installed a custom RPM built from the 0.9.x branch in an attempt to
> fix
> > this bug:
> >
> > https://issues.apache.org/jira/browse/STORM-763
> >
> > But when running on the new installation, we see the below stacktrace.
> >
> > Any recommendations of how to fix the issue we are seeing, or if/when a
> > stable release of 0.9.6 will be available?
> >
> > thanks!
> >
> > Connection failed
> >
> Netty-Client-myserver.mydomain.com/10.22.16.170:6701java.lang.NullPointerException
> > <
> http://netty-client-myserver.mydomain.com/10.22.16.170:6701java.lang.NullPointerException
> >:
> > null
> > at backtype.storm.messaging.netty.Client.flushMessages(Client.java:310)
> > ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> backtype.storm.messaging.netty.Client.notifyInterestChanged(Client.java:439)
> > ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> backtype.storm.messaging.netty.StormClientHandler.channelInterestChanged(StormClientHandler.java:36)
> > ~[storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.channelInterestChanged(SimpleChannelUpstreamHandler.java:200)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.Channels.fireChannelInterestChanged(Channels.java:358)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:310)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteRequestQueue.poll(AbstractNioChannel.java:204)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:186)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:335)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > [storm-core-0.9.6-SNAPSHOT.jar:0.9.6-SNAPSHOT]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > [na:1.7.0_11]
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread HeartSaVioR 
Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593615
  
Reminder: It should be applied into all version branches.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876683#comment-14876683
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593615
  
Reminder: It should be applied into all version branches.


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593725
  
@HeartSaVioR all branches >= 0.9.6 ... that's when STORM-763 introduced the 
bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread Michael Schonfeld (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876686#comment-14876686
 ] 

Michael Schonfeld commented on STORM-1051:
--

Do I need to attach a patch file? Or is the pull request on GitHub sufficient? 
Also, should I mark this as resolved?

> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876687#comment-14876687
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593725
  
@HeartSaVioR all branches >= 0.9.6 ... that's when STORM-763 introduced the 
bug.


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread HeartSaVioR 
Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593987
  
@schonfeld 
Yes, we maintains three version lines, master (0.11.x), 0.10.x, 0.9.x. It 
should be applied to all three version branches.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876694#comment-14876694
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141593987
  
@schonfeld 
Yes, we maintains three version lines, master (0.11.x), 0.10.x, 0.9.x. It 
should be applied to all three version branches.


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141595905
  
Ah, ok! Here's what it looked like in our logs, just in case someone else 
tries to Google the exception at some point...

```
2015-09-17 15:03:22 b.s.m.n.StormClientHandler [INFO] Connection failed 
Netty-Client-/10.244.114.24:6703
java.lang.NullPointerException
  at backtype.storm.messaging.netty.Client.flushMessages(Client.java:320) 
~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
backtype.storm.messaging.netty.Client.notifyInterestChanged(Client.java:455) 
~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
backtype.storm.messaging.netty.StormClientHandler.channelInterestChanged(StormClientHandler.java:36)
 ~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
 ~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876702#comment-14876702
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141595905
  
Ah, ok! Here's what it looked like in our logs, just in case someone else 
tries to Google the exception at some point...

```
2015-09-17 15:03:22 b.s.m.n.StormClientHandler [INFO] Connection failed 
Netty-Client-/10.244.114.24:6703
java.lang.NullPointerException
  at backtype.storm.messaging.netty.Client.flushMessages(Client.java:320) 
~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
backtype.storm.messaging.netty.Client.notifyInterestChanged(Client.java:455) 
~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
backtype.storm.messaging.netty.StormClientHandler.channelInterestChanged(StormClientHandler.java:36)
 ~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
  at 
org.apache.storm.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:106)
 ~[storm-core-0.10.1-SNAPSHOT.jar:0.10.1-SNAPSHOT]
```


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread Jungtaek Lim (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876713#comment-14876713
 ] 

Jungtaek Lim commented on STORM-1051:
-

We are waiting for more reviewer to review.
In bylaws, we should wait at least 1 day to merge PR into repository.
Please refer https://github.com/apache/storm/blob/master/DEVELOPER.md and 
http://storm.apache.org/documentation/BYLAWS.html.

Issue will be resolved by committer who merges PR, when PR gets merged into 
repository.

> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1005] set storm.local.dir relative valu...

2015-09-18 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/722


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1005) Supervisor do not get running workers after restart.

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1005:

Component/s: (was: Storm-eventhubs)

> Supervisor do not get running workers after restart.
> 
>
> Key: STORM-1005
> URL: https://issues.apache.org/jira/browse/STORM-1005
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Renkai Ge
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> I have 8 supervisors running,2 of them shutdown by themselves.
> After I restart them.One supervisor shows the right number of workers, but 
> another shows the worker number is 0 while the workers are still running on 
> that host.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1005) Supervisor do not get running workers after restart.

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876720#comment-14876720
 ] 

ASF GitHub Bot commented on STORM-1005:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/722


> Supervisor do not get running workers after restart.
> 
>
> Key: STORM-1005
> URL: https://issues.apache.org/jira/browse/STORM-1005
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Renkai Ge
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> I have 8 supervisors running,2 of them shutdown by themselves.
> After I restart them.One supervisor shows the right number of workers, but 
> another shows the worker number is 0 while the workers are still running on 
> that host.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1005) Supervisor do not get running workers after restart.

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1005.
-
   Resolution: Fixed
 Assignee: Renkai Ge
Fix Version/s: 0.10.0

Thanks [~RenkaiGe],
I merged into master and 0.10.x-branch.

> Supervisor do not get running workers after restart.
> 
>
> Key: STORM-1005
> URL: https://issues.apache.org/jira/browse/STORM-1005
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Renkai Ge
>Assignee: Renkai Ge
> Fix For: 0.10.0
>
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> I have 8 supervisors running,2 of them shutdown by themselves.
> After I restart them.One supervisor shows the right number of workers, but 
> another shows the worker number is 0 while the workers are still running on 
> that host.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1005]replace default storm.local.dir va...

2015-09-18 Thread erikdw 
Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/713#discussion_r39913602
  
--- Diff: docs/documentation/Setting-up-a-Storm-cluster.md ---
@@ -52,10 +52,13 @@ storm.zookeeper.servers:
 
 If the port that your Zookeeper cluster uses is different than the 
default, you should set **storm.zookeeper.port** as well.
 
-2) **storm.local.dir**: The Nimbus and Supervisor daemons require a 
directory on the local disk to store small amounts of state (like jars, confs, 
and things like that). You should create that directory on each machine, give 
it proper permissions, and then fill in the directory location using this 
config. For example:
+2) **storm.local.dir**: The Nimbus and Supervisor daemons require a 
directory on the local disk to store small amounts of state (like jars, confs, 
and things like that).
+ You should create that directory on each machine, give it proper 
permissions, and then fill in the directory location using this config.
+ You'd better set it to absolute path rather than relative path,otherwise 
it will cause unexpected behavior when you launch daemons
+ from different directories.For example:
--- End diff --

space between . and For


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1005) Supervisor do not get running workers after restart.

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876755#comment-14876755
 ] 

ASF GitHub Bot commented on STORM-1005:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/713#discussion_r39913602
  
--- Diff: docs/documentation/Setting-up-a-Storm-cluster.md ---
@@ -52,10 +52,13 @@ storm.zookeeper.servers:
 
 If the port that your Zookeeper cluster uses is different than the 
default, you should set **storm.zookeeper.port** as well.
 
-2) **storm.local.dir**: The Nimbus and Supervisor daemons require a 
directory on the local disk to store small amounts of state (like jars, confs, 
and things like that). You should create that directory on each machine, give 
it proper permissions, and then fill in the directory location using this 
config. For example:
+2) **storm.local.dir**: The Nimbus and Supervisor daemons require a 
directory on the local disk to store small amounts of state (like jars, confs, 
and things like that).
+ You should create that directory on each machine, give it proper 
permissions, and then fill in the directory location using this config.
+ You'd better set it to absolute path rather than relative path,otherwise 
it will cause unexpected behavior when you launch daemons
+ from different directories.For example:
--- End diff --

space between . and For


> Supervisor do not get running workers after restart.
> 
>
> Key: STORM-1005
> URL: https://issues.apache.org/jira/browse/STORM-1005
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Renkai Ge
>Assignee: Renkai Ge
> Fix For: 0.10.0
>
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> I have 8 supervisors running,2 of them shutdown by themselves.
> After I restart them.One supervisor shows the right number of workers, but 
> another shows the worker number is 0 while the workers are still running on 
> that host.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1032] Add generics to component configu...

2015-09-18 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/723


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1032) Add generics to component configuration methods

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876761#comment-14876761
 ] 

ASF GitHub Bot commented on STORM-1032:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/723


> Add generics to component configuration methods
> ---
>
> Key: STORM-1032
> URL: https://issues.apache.org/jira/browse/STORM-1032
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Dean de Bree
>Assignee: Dean de Bree
>Priority: Minor
>
> The getComponentConfiguration methods currently return a simple Map object. 
> In fact they should be returning a Map. 
> By changing this, it will make it slightly clear what is being returned by 
> the method and potentially pick up some issues at compile time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1032) Add generics to component configuration methods

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1032.
-
   Resolution: Fixed
Fix Version/s: 0.11.0

Thanks [~ddebree],
I merged into master.

> Add generics to component configuration methods
> ---
>
> Key: STORM-1032
> URL: https://issues.apache.org/jira/browse/STORM-1032
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Dean de Bree
>Assignee: Dean de Bree
>Priority: Minor
> Fix For: 0.11.0
>
>
> The getComponentConfiguration methods currently return a simple Map object. 
> In fact they should be returning a Map. 
> By changing this, it will make it slightly clear what is being returned by 
> the method and potentially pick up some issues at compile time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread erikdw 
Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/745#discussion_r39913836
  
--- Diff: storm-core/src/jvm/backtype/storm/messaging/netty/Client.java ---
@@ -317,7 +317,7 @@ private int iteratorSize(Iterator msgs) {
  * If the write operation fails, then we will close the channel and 
trigger a reconnect.
  */
 private void flushMessages(Channel channel, final MessageBatch batch) {
-if(batch.isEmpty()){
+if(null == batch || batch.isEmpty()){
--- End diff --

style nits:
* `if(` -> `if (`
* `){` -> `) {`

Those are more consistent with the rest of this file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876768#comment-14876768
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/745#discussion_r39913836
  
--- Diff: storm-core/src/jvm/backtype/storm/messaging/netty/Client.java ---
@@ -317,7 +317,7 @@ private int iteratorSize(Iterator msgs) {
  * If the write operation fails, then we will close the channel and 
trigger a reconnect.
  */
 private void flushMessages(Channel channel, final MessageBatch batch) {
-if(batch.isEmpty()){
+if(null == batch || batch.isEmpty()){
--- End diff --

style nits:
* `if(` -> `if (`
* `){` -> `) {`

Those are more consistent with the rest of this file.


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1050) Topologies with same name run on one cluster

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876810#comment-14876810
 ] 

ASF GitHub Bot commented on STORM-1050:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/739


> Topologies with same name run on one cluster
> 
>
> Key: STORM-1050
> URL: https://issues.apache.org/jira/browse/STORM-1050
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.11.0, 0.9.5
>Reporter: chenyuzhao
> Fix For: 0.11.0
>
>
> When a topology is submitted repeatedly very closely, or Nimbus is too busy 
> to response the repeated submission RPC calls from the same topology, there 
> is very high probability that Nimbus will receive the repeatedly calls from 
> one topology, finally several topologies with same name will be running on 
> one storm cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1050:Topologies with same name run on on...

2015-09-18 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/739


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1050) Topologies with same name run on one cluster

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated STORM-1050:

Affects Version/s: 0.9.5
   0.10.0

> Topologies with same name run on one cluster
> 
>
> Key: STORM-1050
> URL: https://issues.apache.org/jira/browse/STORM-1050
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.11.0, 0.9.5
>Reporter: chenyuzhao
> Fix For: 0.11.0
>
>
> When a topology is submitted repeatedly very closely, or Nimbus is too busy 
> to response the repeated submission RPC calls from the same topology, there 
> is very high probability that Nimbus will receive the repeatedly calls from 
> one topology, finally several topologies with same name will be running on 
> one storm cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1050) Topologies with same name run on one cluster

2015-09-18 Thread Jungtaek Lim (JIRA) 

 [ 
https://issues.apache.org/jira/browse/STORM-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1050.
-
   Resolution: Fixed
 Assignee: chenyuzhao
Fix Version/s: (was: 0.11.0)
   0.10.0

Thanks [~danny0405],
I merged into master and 0.10.x-branch.

> Topologies with same name run on one cluster
> 
>
> Key: STORM-1050
> URL: https://issues.apache.org/jira/browse/STORM-1050
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.11.0, 0.9.5
>Reporter: chenyuzhao
>Assignee: chenyuzhao
> Fix For: 0.10.0
>
>
> When a topology is submitted repeatedly very closely, or Nimbus is too busy 
> to response the repeated submission RPC calls from the same topology, there 
> is very high probability that Nimbus will receive the repeatedly calls from 
> one topology, finally several topologies with same name will be running on 
> one storm cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141625088
  
@erikdw better now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread schonfeld 
Github user schonfeld commented on a diff in the pull request:

https://github.com/apache/storm/pull/745#discussion_r39916772
  
--- Diff: storm-core/src/jvm/backtype/storm/messaging/netty/Client.java ---
@@ -317,7 +317,7 @@ private int iteratorSize(Iterator msgs) {
  * If the write operation fails, then we will close the channel and 
trigger a reconnect.
  */
 private void flushMessages(Channel channel, final MessageBatch batch) {
-if(batch.isEmpty()){
+if(null == batch || batch.isEmpty()){
--- End diff --

@erikdw better now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876909#comment-14876909
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141625088
  
@erikdw better now


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876911#comment-14876911
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user schonfeld commented on a diff in the pull request:

https://github.com/apache/storm/pull/745#discussion_r39916772
  
--- Diff: storm-core/src/jvm/backtype/storm/messaging/netty/Client.java ---
@@ -317,7 +317,7 @@ private int iteratorSize(Iterator msgs) {
  * If the write operation fails, then we will close the channel and 
trigger a reconnect.
  */
 private void flushMessages(Channel channel, final MessageBatch batch) {
-if(batch.isEmpty()){
+if(null == batch || batch.isEmpty()){
--- End diff --

@erikdw better now


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1051] Fix for flushMessagse NPE

2015-09-18 Thread erikdw 
Github user erikdw commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141631234
  
@schonfeld: thx!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1051) Netty Client.java's flushMessages produces a NullPointerException

2015-09-18 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/STORM-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876929#comment-14876929
 ] 

ASF GitHub Bot commented on STORM-1051:
---

Github user erikdw commented on the pull request:

https://github.com/apache/storm/pull/745#issuecomment-141631234
  
@schonfeld: thx!


> Netty Client.java's flushMessages produces a NullPointerException
> -
>
> Key: STORM-1051
> URL: https://issues.apache.org/jira/browse/STORM-1051
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Michael Schonfeld
>Assignee: Michael Schonfeld
>Priority: Critical
>
> STORM-763 replaced `return batch != null && !batch.isEmpty();` with 
> `if(batch.isEmpty())`... which means that if batch == null, a 
> NullPointerException is thrown. Problem is, batch is often null, which means 
> that 763 made Storm unusable...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)