[jira] [Resolved] (STORM-2042) Nimbus client connections not closed properly causing connection leaks

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-2042.
-
   Resolution: Fixed
Fix Version/s: (was: 1.x)
   1.0.3
   1.1.0

Thanks [~arunmahadevan], I also merged into master and 1.0.x branch. (Harsha 
merged this into 1.x-branch)

> Nimbus client connections not closed properly causing connection leaks
> --
>
> Key: STORM-2042
> URL: https://issues.apache.org/jira/browse/STORM-2042
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.1.0, 1.0.3
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The nimbus client connections are not closed properly causing connection 
> leaks. After the number of connections exceed nimbus.thrift.threads, a 
> RejectedExecutionException is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1631: [STORM-2042] Nimbus client connections not closed ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1631


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1631: [STORM-2042] Nimbus client connections not closed properl...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1631
  
+1 for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1635: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1635#discussion_r75429055
  
--- Diff: 
external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/ExprCompiler.java
 ---
@@ -67,8 +68,9 @@ public ExprCompiler(PrintWriter pw, JavaTypeFactory 
typeFactory) {
   public String visitInputRef(RexInputRef rexInputRef) {
 String name = reserveName();
 String typeName = javaTypeName(rexInputRef);
+String boxedTypeName = boxedJavaTypeName(rexInputRef);
 pw.print(String.format("%s %s = (%s)(_data.get(%d));\n", typeName, 
name,
-   typeName, rexInputRef.getIndex()));
+   boxedTypeName, rexInputRef.getIndex()));
--- End diff --

Converting Object to primitive type is allowed for Java 7 or upper, but 
Janino throws error while compiling script so I just converted to boxed type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1636: [STORM-2047] Add note to add logviewer hosts to br...

[arunmahadevan]

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/1636

[STORM-2047] Add note to add logviewer hosts to browser whitelist



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2047

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1636


commit 4875f6ebd33d60c9e0ab43c022268915f74cd2eb
Author: Arun Mahadevan 
Date:   2016-08-19T05:18:15Z

[STORM-2047] Add note to add logviewer hosts to browser whitelist




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1635: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1635

STORM-1434 Support the GROUP BY clause in StormSQL

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
  * before: creating codes and compile them
  * after: use Trident features, only creating code block if evaluation is 
needed
* Janino comes in to help evaluating code block in runtime

This change can handle the below sql statements in runtime:

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM 
ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE SUMMARY_ORDERS (ID INT PRIMARY KEY, UNIT_PRICE_AVG 
DOUBLE, SUM_OF_QUANTITY INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO SUMMARY_ORDERS SELECT ID, AVG(UNIT_PRICE) AS UNIT_PRICE_AVG, 
SUM(QUANTITY) AS SUM_OF_QUANTITY FROM ORDERS GROUP BY ID
```

This can be easily backported to 1.x branch since it only touches storm-sql 
module.

When reviewing, please share your idea if you find something to optimize 
further.
I didn't play with Trident hardly so I'm not sure I'm following the best 
practice.

I didn't touch UDF yet since coverage of STORM-1434 would be too broad if 
STORM-1434 contains UDF. Will file an issue to track and plan to address that.

Also STORM-2016 should be addressed soon in order to run the topology.

TODO

- [ ] documentation update

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1434

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1635.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1635


commit fc93fdad03c25967e7b00cc9c2289f46d71fb0a9
Author: Jungtaek Lim 
Date:   2016-08-18T07:32:46Z

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1633
  
Bad branch name. Will resubmit a new pull request with proper branch name 
soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR closed the pull request at:

https://github.com/apache/storm/pull/1633


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-2047) In secure setup the log page can't be viewed

[Arun Mahadevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427634#comment-15427634
 ] 

Arun Mahadevan commented on STORM-2047:
---

[~raghavgautam] The storm security doc 
https://github.com/apache/storm/blob/master/docs/SECURITY.md already explains 
whitelisting the UI servers in the browser. Maybe we can add a note to 
explicitly mention that the hosts where the log viewer runs needs to be 
whitelisted as well.

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
> Attachments: screenshot-1.png
>
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Website update

[Jungtaek Lim]

Thanks guys. I updated the page.
http://storm.apache.org/talksAndVideos.html

- Jungtaek Lim (HeartSaVIoR)


2016년 8월 12일 (금) 오후 6:12, Harsha 님이 작성:

> Jungtaek,
>  Some of the recent slides in hadoop summit san jose
>
>
> http://www.slideshare.net/HadoopSummit/resource-aware-scheduling-in-apache-spark
>
> http://www.slideshare.net/HadoopSummit/the-future-of-apache-storm-63920895?qid=4a6e1165-6dbf-47b9-a8c0-1648308762aa==_search=3
> https://www.youtube.com/watch?v=_Q2uzRIkTd8=youtu.be
>
>
> http://www.slideshare.net/HadoopSummit/managing-hadoop-hbase-and-storm-clusters-at-yahoo-scale
> https://www.youtube.com/watch?v=Ib8ch-xVzbw
>
> -Harsha
>
> On Thu, Aug 11, 2016, at 08:42 PM, Jungtaek Lim wrote:
> > Hi dev,
> >
> > Along with taking care of some documentation issues, I also add some
> > missing committer/PMCs into people page.
> >
> > I feel that it would be better to update 'Talks and slideshows' to topic
> > of
> > recent Hadoop summit or others.
> > Any ideas or recommendations?
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
>

Re: [Vote] Make Java 8 as minimum requirement for Storm 2.0

[P. Taylor Goetz]

+1

JFDI. I've not seen anything but +1s across the board. We have consensus. This 
is no different than any other code change.

-Taylor

> On Aug 18, 2016, at 8:53 PM, Jungtaek Lim  wrote:
> 
> Before stalling, whether this should need VOTE or not, we confirmed our
> consensus regarding this and all positive.
> 
> I'll go on merging the pull request. Please let me know if you feel
> it's inappropriate so that I can revert.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> On Wednesday, August 17, 2016, Bobby Evans 
> wrote:
> 
>> OK I think we should clarify that in the case of code changes.
>> - Bobby
>> 
>>On Wednesday, August 17, 2016 9:40 AM, Jungtaek Lim > > wrote:
>> 
>> 
>> As Harsha said, I also think everyone has different interpretation of
>> BYLAWS. I requested Harsha to go on voting before applying actual change,
>> because BYLAWS [1] clearly states that,
>> 
>> "Decisions regarding the project are made by votes on the primary project
>> development mailing list. ... Votes are clearly indicated by subject line
>> starting with [VOTE]. ... Voting is carried out by replying to the vote
>> mail."
>> 
>> I interpret this as it is, since it clearly states about what the vote is,
>> and how to vote. We could even more strictly like this, "Every decisions
>> regarding the project should pass VOTE for making effect on project", but
>> we implicitly allowed general consensus.
>> 
>> I just thought this is beyond the thing which can be handled with general
>> consensus, but I'm also fine if we all agree this can be handled as a code
>> change.
>> 
>> - Jungtaek Lim (HeartSaVioR)
>> 
>> [1] http://storm.apache.org/contribute/BYLAWS.html
>> 
>> 
>> 2016년 8월 17일 (수) 오후 11:21, Bobby Evans 님이 작성:
>> 
>>> I don't think there is anything wrong with checking with others around a
>>> fairly major change like this, which by the way I am +1 on, but switching
>>> the travis config and changing a 7 to an 8 in the pom.xml is technically
>> a
>>> code change, and only really needs the votes accordingly for that.  If
>> you
>>> want that vote to happen here and not on the JIRA for visibility that
>>> technically is not according to the bylaws but it follows the spirit so I
>>> personally am OK with it.  http://storm.apache.org/
>> contribute/BYLAWS.html
>>> 
>>> Which on a side note, that whole discussion about changing which mailing
>>> lists get updated when, we might want to either change the bylaws or
>> update
>>> some documentation so that votes specifically are forwarded to the dev
>>> list.  Otherwise we start to run into a situation where an important vote
>>> for a code change is happening like this, and some people are not aware
>> of
>>> it, because they are only watching the dev list. - Bobby
>>> 
>>>   On Tuesday, August 16, 2016 11:59 PM, Harsha Chintalapani <
>>> st...@harsha.io > wrote:
>>> 
>>> 
>>> I guess everyone has different interpretation of what Bylaws means .
>> More
>>> context
>>> 
>>> https://github.com/apache/storm/pull/1628
>>> 
>>> anything wrong with Vote thread?
>>> 
>>> 
>>> On Tue, Aug 16, 2016 at 8:04 PM P. Taylor Goetz > > wrote:
>>> 
 Why is this a VOTE?
 
 
> On Aug 16, 2016, at 9:18 PM, Harsha Chintalapani > >
 wrote:
> 
> Hi All,
>   We had a discussion thread for removing Java 7 support for
>> Storm
> 2.0.
> Here is a formal voting thread and the JIRA
> https://issues.apache.org/jira/browse/STORM-2041.
> 
> Thanks,
> Harsha
 
 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 
> -- 
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior

[GitHub] storm issue #1631: [STORM-2042] Nimbus client connections not closed properl...

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/storm/pull/1631
  
@harshach this is the port of https://github.com/apache/storm/pull/1630 
into master. Can you review and merge this well ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1633
  
Need to modify README to state that trident mode of storm-sql can support 
the "group by" with basic aggregation functions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-2048) Refactor code blocks which are ported to for-loop to Java Stream API

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427487#comment-15427487
 ] 

Jungtaek Lim commented on STORM-2048:
-

We might have multiple pull requests for this issue, so please hold on making 
as resolved when pull request is merged to master.

> Refactor code blocks which are ported to for-loop to Java Stream API
> 
>
> Key: STORM-2048
> URL: https://issues.apache.org/jira/browse/STORM-2048
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>
> We just changed minimum requirement for master branch to Java 1.8 from 
> STORM-2041. 
> Thanks for the change we can change ported code block which was functional 
> style to similar style again.
> We could even broaden the boundary of this issue for applying other benefits 
> from Java 8, or file separate issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-2048) Refactor code blocks which are ported to for-loop to Java Stream API

[Jungtaek Lim (JIRA)]

Jungtaek Lim created STORM-2048:
---

 Summary: Refactor code blocks which are ported to for-loop to Java 
Stream API
 Key: STORM-2048
 URL: https://issues.apache.org/jira/browse/STORM-2048
 Project: Apache Storm
  Issue Type: Improvement
  Components: storm-core
Affects Versions: 2.0.0
Reporter: Jungtaek Lim


We just changed minimum requirement for master branch to Java 1.8 from 
STORM-2041. 

Thanks for the change we can change ported code block which was functional 
style to similar style again.

We could even broaden the boundary of this issue for applying other benefits 
from Java 8, or file separate issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-2041.
-
Resolution: Fixed

Thanks [~sriharsha] for taking care of this. I merged into master.

> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1628: STORM-2041. Make Java 8 as minimum requirement for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1628


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1628: STORM-2041. Make Java 8 as minimum requirement for 2.0 re...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1628
  
I will just do +1 and go merging since we already see our consensus from 
multiple places.
I don't want to become a lawyer :) so that's enough.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1627: STORM-2039: backpressure refactoring in worker and execut...

[zhuoliu]

Github user zhuoliu commented on the issue:

https://github.com/apache/storm/pull/1627
  
@abellina looks good to me, just one extra suggestion:
In

https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/daemon/worker.clj#L156
To reduce the traffic to ZK, we checked the previous flag to update to 
Zookeeper only when it is necessary (the flag has changed), but still there may 
some weird case we have inconsistency between local "prev-backpressure-flag" 
and the worker backpressure node at Zookeeper (see STORM-1949).

So to strike a tradeoff and avoid the possible wrong stall of a topology, 
could you add do a random number generation here, e.g., generate a number 
between 0-9, if it is zero, we will call the try block no matter 
"prev-backpressure-flag" and "curr-backpressure-flag" is equal or not.

Thanks a lot!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Vote] Make Java 8 as minimum requirement for Storm 2.0

[Jungtaek Lim]

Before stalling, whether this should need VOTE or not, we confirmed our
consensus regarding this and all positive.

I'll go on merging the pull request. Please let me know if you feel
it's inappropriate so that I can revert.

- Jungtaek Lim (HeartSaVioR)

On Wednesday, August 17, 2016, Bobby Evans 
wrote:

> OK I think we should clarify that in the case of code changes.
>  - Bobby
>
> On Wednesday, August 17, 2016 9:40 AM, Jungtaek Lim  > wrote:
>
>
>  As Harsha said, I also think everyone has different interpretation of
> BYLAWS. I requested Harsha to go on voting before applying actual change,
> because BYLAWS [1] clearly states that,
>
> "Decisions regarding the project are made by votes on the primary project
> development mailing list. ... Votes are clearly indicated by subject line
> starting with [VOTE]. ... Voting is carried out by replying to the vote
> mail."
>
> I interpret this as it is, since it clearly states about what the vote is,
> and how to vote. We could even more strictly like this, "Every decisions
> regarding the project should pass VOTE for making effect on project", but
> we implicitly allowed general consensus.
>
> I just thought this is beyond the thing which can be handled with general
> consensus, but I'm also fine if we all agree this can be handled as a code
> change.
>
> - Jungtaek Lim (HeartSaVioR)
>
> [1] http://storm.apache.org/contribute/BYLAWS.html
>
>
> 2016년 8월 17일 (수) 오후 11:21, Bobby Evans 님이 작성:
>
> > I don't think there is anything wrong with checking with others around a
> > fairly major change like this, which by the way I am +1 on, but switching
> > the travis config and changing a 7 to an 8 in the pom.xml is technically
> a
> > code change, and only really needs the votes accordingly for that.  If
> you
> > want that vote to happen here and not on the JIRA for visibility that
> > technically is not according to the bylaws but it follows the spirit so I
> > personally am OK with it.  http://storm.apache.org/
> contribute/BYLAWS.html
> >
> > Which on a side note, that whole discussion about changing which mailing
> > lists get updated when, we might want to either change the bylaws or
> update
> > some documentation so that votes specifically are forwarded to the dev
> > list.  Otherwise we start to run into a situation where an important vote
> > for a code change is happening like this, and some people are not aware
> of
> > it, because they are only watching the dev list. - Bobby
> >
> >On Tuesday, August 16, 2016 11:59 PM, Harsha Chintalapani <
> > st...@harsha.io > wrote:
> >
> >
> >  I guess everyone has different interpretation of what Bylaws means .
> More
> > context
> >
> > https://github.com/apache/storm/pull/1628
> >
> > anything wrong with Vote thread?
> >
> >
> > On Tue, Aug 16, 2016 at 8:04 PM P. Taylor Goetz  > wrote:
> >
> > > Why is this a VOTE?
> > >
> > >
> > > > On Aug 16, 2016, at 9:18 PM, Harsha Chintalapani  >
> > > wrote:
> > > >
> > > > Hi All,
> > > >We had a discussion thread for removing Java 7 support for
> Storm
> > > > 2.0.
> > > > Here is a formal voting thread and the JIRA
> > > > https://issues.apache.org/jira/browse/STORM-2041.
> > > >
> > > > Thanks,
> > > > Harsha
> > >
> > >
> >
> >
> >
>
>



-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Fwd: Too many machine mails

[Jungtaek Lim]

FYI: Forwarding Talyor's reply.
(I need to have a habit to remove all personal recipients before sending
mail to mailing list so that 'reply' works properly.)

-- Forwarded message --
From: *P. Taylor Goetz* 
Date: Friday, August 19, 2016
Subject: Too many machine mails
To: 임정택 


I’m on it. it might take a while (up to a week or more) depending on
INFRA’s workload.

-Taylor

On Aug 18, 2016, at 12:18 PM, Jungtaek Lim > wrote:

INFRA-12453 is done. Following Github comment should be written to work log
and notification from JIRA shouldn't be sent.
Please let me know if it's not.

2016년 8월 19일 (금) 오전 1:10, Jungtaek Lim >님이 작성:

> I filed a new issue for creating mailing list for issues@ but infra
> redirects me to https://infra.apache.org/officers/mlreq and close issue.
> (https://issues.apache.org/jira/browse/INFRA-12452)
>
> Submit button is disabled and saying 'Only ASF members and Officers may
> submit mailing list requests'.
> I guess Taylor and Bobby are ASF member. Could you help setup this?
>
> When complete I can go filing new issue for moving JIRA and Github
> notifications to that list.
>
> I also just filed another new issue for changing the way to post Github
> events to JIRA issue : comment -> work log.
> https://issues.apache.org/jira/browse/INFRA-12453
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 8월 16일 (화) 오후 3:53, Erik Weathers  >님이 작성:
>
>> The "Work Log" archival sounds good to me too!
>>
>> On Mon, Aug 15, 2016 at 11:27 PM, Jungtaek Lim > > wrote:
>>
>> > We seem to meet general consensus.
>> >
>> > Do we need to just file issues for Apache INFRA? Or do we need to do
>> other
>> > things?
>> > (excluding updating website and announce to dev@ for new arrival of
>> issue@
>> > )
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2016년 8월 13일 (토) 오전 2:11, Kyle Nusbaum > .invalid>님이
>> > 작성:
>> >
>> > > +1 for Taylor's proposed solution.
>> > >  -- Kyle
>> > >
>> > > On Friday, August 12, 2016 4:57 AM, Jungtaek Lim <
>> kabh...@gmail.com >
>> > > wrote:
>> > >
>> > >
>> > >  IMHO, all committers/PMCs need to be noticed for JIRA and Github
>> > > activities.
>> > > (Sometimes pull requests are submitted without having associated JIRA
>> > issue
>> > > as I did it yesterday. Actually I don't strictly think committers/PMCs
>> > need
>> > > to be noticed for all of pull request comments, but at least need to
>> be
>> > > noticed for open/close pull request activities.)
>> > >
>> > > So separating list is only for contributors which would like to hear
>> > about
>> > > community activities but don't want to see the details (issue level),
>> > > and reducing
>> > > duplication should be handled even though we can move JIRA / Github
>> > > activities out of dev@ list.
>> > >
>> > > I assume that adding JIRA work log doesn't send notification. If it
>> is,
>> > > that's enough for me. Separating list is optional.
>> > >
>> > > - Jungtaek Lim (HeartSaVioR)
>> > >
>> > >
>> > >
>> > > 2016년 8월 12일 (금) 오전 6:20, Kyle Nusbaum > .
>> > invalid>님이
>> > > 작성:
>> > >
>> > > > If this other mailing list gets notified of all github activity (all
>> > > > comments, etc.), is that sufficient for being "archived" on ASF
>> > hardware?
>> > > > I'm assuming the ASF is hosting their own mail servers.
>> > > > I'd much rather have all github activity go to a mailing list for
>> > archive
>> > > > than go to jira and end up in the mailing list 4 times anyway.
>> > > > -- Kyle
>> > > >
>> > > >On Thursday, August 11, 2016 3:54 PM, P. Taylor Goetz <
>> > > > ptgo...@gmail.com
>> > wrote:
>> > > >
>> > > >
>> > > >  We don’t need a formal vote if we have a general consensus.
>> > > > This is an issue I’d like to see fixed since it drives me nuts and
>> is
>> > > > amplified by the number of mailing lists I’m subscribed to.
>> > > > The requirement to link github pull request comments, etc. to JIRA
>> > > > originates from the ASF policy that all artifacts of the
>> > decision-making
>> > > > process (email, issues, etc.) be archived on ASF-controlled
>> hardware.
>> > The
>> > > > linking of github activity to JIRA partly addressed that, but it’s
>> not
>> > > > optimal (e.g. What happens when a PR isn’t linked to a JIRA?).
>> > > > Personally, I want to get notified of all JIRA and github activity,
>> but
>> > > > what I don’t want is 4 copies of every notification. I have some
>> 

[jira] [Commented] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427368#comment-15427368
 ] 

Raghav Kumar Gautam commented on STORM-2047:


Setting {{network.negotiate-auth.trusted-uris}} to something like 
{{.apache.org}} works for firefox.

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
> Attachments: screenshot-1.png
>
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghav Kumar Gautam updated STORM-2047:
---
Component/s: documentation

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
> Attachments: screenshot-1.png
>
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427315#comment-15427315
 ] 

Raghav Kumar Gautam commented on STORM-2047:


Thanks [~sriharsha] for helping with debugging. The issue is that browser needs 
to be able to talk to supervisor machines. In safari this works cleanly. In 
firefox, you have to change settings to allow supervisors.

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
> Attachments: screenshot-1.png
>
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghav Kumar Gautam updated STORM-2047:
---
Attachment: screenshot-1.png

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
> Attachments: screenshot-1.png
>
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghav Kumar Gautam updated STORM-2047:
---
Assignee: Arun Mahadevan

> In secure setup the log page can't be viewed
> 
>
> Key: STORM-2047
> URL: https://issues.apache.org/jira/browse/STORM-2047
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Arun Mahadevan
>
> This is about the topology inspector feature. When we click events button on 
> the bolt page, we expect that we will get to a log page which will show 
> tuples. Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-2047) In secure setup the log page can't be viewed

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created STORM-2047:
--

 Summary: In secure setup the log page can't be viewed
 Key: STORM-2047
 URL: https://issues.apache.org/jira/browse/STORM-2047
 Project: Apache Storm
  Issue Type: Bug
Reporter: Raghav Kumar Gautam


This is about the topology inspector feature. When we click events button on 
the bolt page, we expect that we will get to a log page which will show tuples. 
Instead we get  authentication required error, see attached image.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1630: [STORM-2042] Nimbus client connections not closed ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1630


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1591: STORM-1038: Upgrade netty to 4.x in 1.x-branch

[knusbaum]

Github user knusbaum commented on a diff in the pull request:

https://github.com/apache/storm/pull/1591#discussion_r75354518
  
--- Diff: storm-core/src/jvm/org/apache/storm/Config.java ---
@@ -73,6 +73,20 @@
 public static final String STORM_MESSAGING_NETTY_BUFFER_SIZE = 
"storm.messaging.netty.buffer_size";
 
 /**
+ * Netty based messaging: The write buffer high water mark for write 
buffer
--- End diff --

These config descriptions are not descriptive. Without reading the code, I 
have no idea what this does.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-2046) Errors when using TOPOLOGY_TESTING_ALWAYS_TRY_SERIALIZE in local mode.

[Cory Kolbeck (JIRA)]

Cory Kolbeck created STORM-2046:
---

 Summary: Errors when using TOPOLOGY_TESTING_ALWAYS_TRY_SERIALIZE 
in local mode.
 Key: STORM-2046
 URL: https://issues.apache.org/jira/browse/STORM-2046
 Project: Apache Storm
  Issue Type: Bug
Affects Versions: 1.0.2
 Environment: Ubuntu 16.04 linux, 4.4.0-34-generic
Java 1.8.0_92
Reporter: Cory Kolbeck
Priority: Minor


When using a LocalCluster during tests, if 
{{TOPOLOGY_TESTING_ALWAYS_TRY_SERIALIZE}} is specified, 
{{assert-can-serialize}} attempts to destructure a Java model object and 
throws, killing the worker. A minimal-ish case and the full logs are here: 
https://gist.github.com/ckolbeck/557734429e62b097efa9382a714122b0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Too many machine mails

[Jungtaek Lim]

INFRA-12453 is done. Following Github comment should be written to work log
and notification from JIRA shouldn't be sent.
Please let me know if it's not.

2016년 8월 19일 (금) 오전 1:10, Jungtaek Lim 님이 작성:

> I filed a new issue for creating mailing list for issues@ but infra
> redirects me to https://infra.apache.org/officers/mlreq and close issue.
> (https://issues.apache.org/jira/browse/INFRA-12452)
>
> Submit button is disabled and saying 'Only ASF members and Officers may
> submit mailing list requests'.
> I guess Taylor and Bobby are ASF member. Could you help setup this?
>
> When complete I can go filing new issue for moving JIRA and Github
> notifications to that list.
>
> I also just filed another new issue for changing the way to post Github
> events to JIRA issue : comment -> work log.
> https://issues.apache.org/jira/browse/INFRA-12453
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 8월 16일 (화) 오후 3:53, Erik Weathers 님이
> 작성:
>
>> The "Work Log" archival sounds good to me too!
>>
>> On Mon, Aug 15, 2016 at 11:27 PM, Jungtaek Lim  wrote:
>>
>> > We seem to meet general consensus.
>> >
>> > Do we need to just file issues for Apache INFRA? Or do we need to do
>> other
>> > things?
>> > (excluding updating website and announce to dev@ for new arrival of
>> issue@
>> > )
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2016년 8월 13일 (토) 오전 2:11, Kyle Nusbaum > .invalid>님이
>> > 작성:
>> >
>> > > +1 for Taylor's proposed solution.
>> > >  -- Kyle
>> > >
>> > > On Friday, August 12, 2016 4:57 AM, Jungtaek Lim <
>> kabh...@gmail.com>
>> > > wrote:
>> > >
>> > >
>> > >  IMHO, all committers/PMCs need to be noticed for JIRA and Github
>> > > activities.
>> > > (Sometimes pull requests are submitted without having associated JIRA
>> > issue
>> > > as I did it yesterday. Actually I don't strictly think committers/PMCs
>> > need
>> > > to be noticed for all of pull request comments, but at least need to
>> be
>> > > noticed for open/close pull request activities.)
>> > >
>> > > So separating list is only for contributors which would like to hear
>> > about
>> > > community activities but don't want to see the details (issue level),
>> > > and reducing
>> > > duplication should be handled even though we can move JIRA / Github
>> > > activities out of dev@ list.
>> > >
>> > > I assume that adding JIRA work log doesn't send notification. If it
>> is,
>> > > that's enough for me. Separating list is optional.
>> > >
>> > > - Jungtaek Lim (HeartSaVioR)
>> > >
>> > >
>> > >
>> > > 2016년 8월 12일 (금) 오전 6:20, Kyle Nusbaum > > invalid>님이
>> > > 작성:
>> > >
>> > > > If this other mailing list gets notified of all github activity (all
>> > > > comments, etc.), is that sufficient for being "archived" on ASF
>> > hardware?
>> > > > I'm assuming the ASF is hosting their own mail servers.
>> > > > I'd much rather have all github activity go to a mailing list for
>> > archive
>> > > > than go to jira and end up in the mailing list 4 times anyway.
>> > > > -- Kyle
>> > > >
>> > > >On Thursday, August 11, 2016 3:54 PM, P. Taylor Goetz <
>> > > > ptgo...@gmail.com> wrote:
>> > > >
>> > > >
>> > > >  We don’t need a formal vote if we have a general consensus.
>> > > > This is an issue I’d like to see fixed since it drives me nuts and
>> is
>> > > > amplified by the number of mailing lists I’m subscribed to.
>> > > > The requirement to link github pull request comments, etc. to JIRA
>> > > > originates from the ASF policy that all artifacts of the
>> > decision-making
>> > > > process (email, issues, etc.) be archived on ASF-controlled
>> hardware.
>> > The
>> > > > linking of github activity to JIRA partly addressed that, but it’s
>> not
>> > > > optimal (e.g. What happens when a PR isn’t linked to a JIRA?).
>> > > > Personally, I want to get notified of all JIRA and github activity,
>> but
>> > > > what I don’t want is 4 copies of every notification. I have some
>> email
>> > > > filters to help alleviate that, but it’s not a good solution.
>> > > > One short-term solution for those who don’t care about JIRA/GitHub
>> > > traffic
>> > > > would be to setup issues@storm.a.o and funnel all generated mail
>> there
>> > > > (that’s just an INFRA ticket followed by some patience). That won’t
>> > > > alleviate the issue of duplicates for those that subscribe to that
>> > list.
>> > > > Another option that’s available now (it wasn’t in the past) is to
>> > switch
>> > > > over to github issues and stop using JIRA. Then people could set
>> their
>> > > own
>> > > > notification preferences in github. The downside is to close
>> issues, it
>> > > > would require a git commit (the same mechanism we use today to close
>> > > > abandoned pull requests). Another downside is that issues would be
>> > > > read-only, so we wouldn’t be able to add labels, etc.
>> > > > Actually as I was typing this I was also 

Re: Too many machine mails

[Jungtaek Lim]

I filed a new issue for creating mailing list for issues@ but infra
redirects me to https://infra.apache.org/officers/mlreq and close issue.
(https://issues.apache.org/jira/browse/INFRA-12452)

Submit button is disabled and saying 'Only ASF members and Officers may
submit mailing list requests'.
I guess Taylor and Bobby are ASF member. Could you help setup this?

When complete I can go filing new issue for moving JIRA and Github
notifications to that list.

I also just filed another new issue for changing the way to post Github
events to JIRA issue : comment -> work log.
https://issues.apache.org/jira/browse/INFRA-12453

Thanks,
Jungtaek Lim (HeartSaVioR)


2016년 8월 16일 (화) 오후 3:53, Erik Weathers 님이
작성:

> The "Work Log" archival sounds good to me too!
>
> On Mon, Aug 15, 2016 at 11:27 PM, Jungtaek Lim  wrote:
>
> > We seem to meet general consensus.
> >
> > Do we need to just file issues for Apache INFRA? Or do we need to do
> other
> > things?
> > (excluding updating website and announce to dev@ for new arrival of
> issue@
> > )
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2016년 8월 13일 (토) 오전 2:11, Kyle Nusbaum  .invalid>님이
> > 작성:
> >
> > > +1 for Taylor's proposed solution.
> > >  -- Kyle
> > >
> > > On Friday, August 12, 2016 4:57 AM, Jungtaek Lim <
> kabh...@gmail.com>
> > > wrote:
> > >
> > >
> > >  IMHO, all committers/PMCs need to be noticed for JIRA and Github
> > > activities.
> > > (Sometimes pull requests are submitted without having associated JIRA
> > issue
> > > as I did it yesterday. Actually I don't strictly think committers/PMCs
> > need
> > > to be noticed for all of pull request comments, but at least need to be
> > > noticed for open/close pull request activities.)
> > >
> > > So separating list is only for contributors which would like to hear
> > about
> > > community activities but don't want to see the details (issue level),
> > > and reducing
> > > duplication should be handled even though we can move JIRA / Github
> > > activities out of dev@ list.
> > >
> > > I assume that adding JIRA work log doesn't send notification. If it is,
> > > that's enough for me. Separating list is optional.
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > >
> > >
> > > 2016년 8월 12일 (금) 오전 6:20, Kyle Nusbaum  > invalid>님이
> > > 작성:
> > >
> > > > If this other mailing list gets notified of all github activity (all
> > > > comments, etc.), is that sufficient for being "archived" on ASF
> > hardware?
> > > > I'm assuming the ASF is hosting their own mail servers.
> > > > I'd much rather have all github activity go to a mailing list for
> > archive
> > > > than go to jira and end up in the mailing list 4 times anyway.
> > > > -- Kyle
> > > >
> > > >On Thursday, August 11, 2016 3:54 PM, P. Taylor Goetz <
> > > > ptgo...@gmail.com> wrote:
> > > >
> > > >
> > > >  We don’t need a formal vote if we have a general consensus.
> > > > This is an issue I’d like to see fixed since it drives me nuts and is
> > > > amplified by the number of mailing lists I’m subscribed to.
> > > > The requirement to link github pull request comments, etc. to JIRA
> > > > originates from the ASF policy that all artifacts of the
> > decision-making
> > > > process (email, issues, etc.) be archived on ASF-controlled hardware.
> > The
> > > > linking of github activity to JIRA partly addressed that, but it’s
> not
> > > > optimal (e.g. What happens when a PR isn’t linked to a JIRA?).
> > > > Personally, I want to get notified of all JIRA and github activity,
> but
> > > > what I don’t want is 4 copies of every notification. I have some
> email
> > > > filters to help alleviate that, but it’s not a good solution.
> > > > One short-term solution for those who don’t care about JIRA/GitHub
> > > traffic
> > > > would be to setup issues@storm.a.o and funnel all generated mail
> there
> > > > (that’s just an INFRA ticket followed by some patience). That won’t
> > > > alleviate the issue of duplicates for those that subscribe to that
> > list.
> > > > Another option that’s available now (it wasn’t in the past) is to
> > switch
> > > > over to github issues and stop using JIRA. Then people could set
> their
> > > own
> > > > notification preferences in github. The downside is to close issues,
> it
> > > > would require a git commit (the same mechanism we use today to close
> > > > abandoned pull requests). Another downside is that issues would be
> > > > read-only, so we wouldn’t be able to add labels, etc.
> > > > Actually as I was typing this I was also chatting with INFRA about
> it.
> > > And
> > > > it turns out you can funnel github activity into the work log for a
> > JIRA,
> > > > rather than comments. That will eliminate the duplicate emails.
> > > > So it seems the solution might be:
> > > > 1. Funnel github activity to the associated JIRA work log.2. setup
> > > > issues@storm.a.o. and route JIRA notification to 

Re: [Discussion] Stopping feature development for storm-core on 1.x-branch

[Jungtaek Lim]

I agree with you Alessandro. Makes perfect sense for me, and that would be
more realistic way to handle the issues.

- Jungtaek Lim (HeartSaVioR)

2016년 8월 18일 (목) 오전 11:46, Alessandro Bellina
님이 작성:

> Just a contributors point of view..
> I think the idea to reclaim jiras that are assigned but without progress
> is a good one. After that is done, in order to not end up in the same
> place, I think after a task gets assigned it needs to be given a due date
> (something sane), and updated periodically if it is a long task (e.g. port
> nimbus). The policy being that if after the due date not meaningful
> progress is made, then the assignment should be given away to someone else
> or put back in the unassigned pile.
> Port/merge issues which are assigned, but don't have "ASF GitHub" watching
> them (so no PR it looks like):
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20in%20(java-migration%2C%20jstorm-merger)%20AND%20assignee%20!%3D%20EMPTY%20and%20watcher%20not%20in%20(%22ASF%20GitHub%20Bot%22)
> I think freezing feature development in master is too extreme. Instead, I
> think that as part of the code review checks, if we see changes to a file
> and it is currently under a port task which isn't past due or stalled, then
> the PR should be held back and submitter informed that the file they are
> modifying is truly being ported. This, in turn, puts the ball back on the
> court of the person doing the porting, as it should, and hopefully gives
> the submitter a due date after which they can PR a ported change. If the
> porting task is stalled, I see no reason why the PR shouldn't be considered
> immediately. That said, something tells me this is hard to implement.
> Thanks,
> Alessandro
>
>
> On Wednesday, August 17, 2016 8:54 PM, Jungtaek Lim 
> wrote:
>
>
>  I'm OK to stop feature dev. for storm-core on 1.x branch. As Taylor said
> that was the original plan, and we broke it.
> We even might need to stop feature development for storm-core on 'master'
> if it touches un-ported files, but it would tend to break so I couldn't
> claim that.
>
> Moreover, I propose unassigning all issues regarding port if pull request
> is not open, and go on with 'no assignee' or 'competitive assignee' for
> porting related issues. We had been set assignee for the issues, and
> assignee holds it even though he/she can't or don't work on that for a long
> time.
>
> Before stopping, I'd like to include STORM-2016
>  to 1.x version line.
> (target for 1.1.0)
>
> - This will heavily affect to upcoming storm-sql improvements (I'd say that
> without STORM-2016 we even have very hard time to launch storm-sql runner)
> - This will help avoiding to copy libraries (and its transitive
> dependencies) to extlib directory when it's necessary for only some of
> topologies. Clear example is STORM-1881
> .
> - This will give great flexibility to configure user topology jar and
> submit topology.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 17일 (수) 오후 11:58, P. Taylor Goetz 님이 작성:
>
> > I agree, and that was the original plan.
> >
> > However, the clojure to java migration stalled for a number of reasons
> > (peoples’ $dayjob responsibilities can change often and affect the amount
> > of time they have to contribute to the project — this is to be expected
> and
> > not considered a problem.). Prior to “feature freezing” the 1.x line, I
> > think we should get 1.1.0 released. In my opinion the only remaining
> issue
> > for the release is a resolution to STORM-2006, particularly adding an
> > interface for aggregated metrics.
> >
> > I don’t have a good feel for how long it will take to get the master
> > branch (aka “2.0”) in a releasable state (again it comes down to
> > contributor $dayjobs, which are largely out of anyone’s control). So I
> > think we should plan on supporting 1.x for a while.
> >
> > -Taylor
> >
> > > On Aug 16, 2016, at 10:40 PM, Harsha Chintalapani 
> > wrote:
> > >
> > > Hi All,
> > >  Currently we are undertaken JStorm merger and ongoing
> migration
> > > of existing Clojure code. I am proposing that we should stop any
> feature
> > > development for 1.x branch so that we can make progress on java
> migration
> > > and get it done before adding any further features. If any one
> interested
> > > adding features they can do so on master for 2.0.
> > >This will give us time to complete the migration and keeps
> the
> > > core code more or less the same . Adding more code to core makes the
> > > migration that much harder.
> > > We should be ok on adding any connectors or so as that can be
> independent
> > > of core and can be easily added to master.
> > >What you think?
> > >
> > > Thanks,
> > > Harsha
> >
> >
>
>

[GitHub] storm pull request #1591: STORM-1038: Upgrade netty to 4.x in 1.x-branch

[hsun-cnnxty]

GitHub user hsun-cnnxty reopened a pull request:

https://github.com/apache/storm/pull/1591

STORM-1038: Upgrade netty to 4.x in 1.x-branch

This is to add the feature to 1.x-branch.  The original PR for master 
branch is #728.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsun-cnnxty/storm 1.x-branch-netty4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1591


commit 57fbccc7e0d710ccaf04a5a828f0ff4cf29ec855
Author: Hang Sun 
Date:   2016-07-25T22:30:54Z

STORM-1038: Upgraded netty to 4.x

commit 144b6eddf314ce9a0b5beb8c99500c56ff0b8ec0
Author: Hang Sun 
Date:   2016-07-30T23:42:52Z

STORM-1038: removed unused imports




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426639#comment-15426639
 ] 

ASF GitHub Bot commented on STORM-1038:
---

GitHub user hsun-cnnxty reopened a pull request:

https://github.com/apache/storm/pull/1591

STORM-1038: Upgrade netty to 4.x in 1.x-branch

This is to add the feature to 1.x-branch.  The original PR for master 
branch is #728.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsun-cnnxty/storm 1.x-branch-netty4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1591


commit 57fbccc7e0d710ccaf04a5a828f0ff4cf29ec855
Author: Hang Sun 
Date:   2016-07-25T22:30:54Z

STORM-1038: Upgraded netty to 4.x

commit 144b6eddf314ce9a0b5beb8c99500c56ff0b8ec0
Author: Hang Sun 
Date:   2016-07-30T23:42:52Z

STORM-1038: removed unused imports




> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426638#comment-15426638
 ] 

ASF GitHub Bot commented on STORM-1038:
---

Github user hsun-cnnxty closed the pull request at:

https://github.com/apache/storm/pull/1591


> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1591: STORM-1038: Upgrade netty to 4.x in 1.x-branch

[hsun-cnnxty]

Github user hsun-cnnxty closed the pull request at:

https://github.com/apache/storm/pull/1591


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1370: [STORM-1707] Remove two minute timeout after worke...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1370#discussion_r75329245
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/daemon/supervisor/SyncProcessEvent.java ---
@@ -110,34 +117,47 @@ public void run() {
 keeperWorkerIds.add(entry.getKey());
 
keepPorts.add(stateHeartbeat.getHeartbeat().get_port());
 }
+if (stateHeartbeat.getState() == State.NOT_STARTED) {
+keeperWorkerIds.add(entry.getKey());
+
keepPorts.add(supervisorData.getWorkerIdsToPorts().get(entry.getKey()));
+}
 }
 Map reassignExecutors = 
getReassignExecutors(assignedExecutors, keepPorts);
 Map newWorkerIds = new HashMap<>();
 for (Integer port : reassignExecutors.keySet()) {
 newWorkerIds.put(port, Utils.uuid());
 }
+
 LOG.debug("Assigned executors: {}", assignedExecutors);
 LOG.debug("Allocated: {}", localWorkerStats);
+LOG.debug("Keeper worker ids: {}", keeperWorkerIds);
+LOG.debug("Keep ports: {}", keepPorts);
+LOG.debug("LaunchTimes: {}", 
supervisorData.getWorkerIdsToLaunchTimes());
+LOG.debug("Ids Ports: {}", 
supervisorData.getWorkerIdsToPorts());
 
 for (Map.Entry entry : 
localWorkerStats.entrySet()) {
 StateHeartbeat stateHeartbeat = entry.getValue();
-if (stateHeartbeat.getState() != State.VALID) {
+if ((stateHeartbeat.getState() != State.VALID && 
stateHeartbeat.getState() != State.NOT_STARTED) ||
+(stateHeartbeat.getState() == State.NOT_STARTED && 
isWorkerStartTimeoutExpired(entry.getKey( { 
 LOG.info("Shutting down and clearing state for id {}, 
Current supervisor time: {}, State: {}, Heartbeat: {}", entry.getKey(), now,
--- End diff --

The clojure equivalent for this logs also when the worker could not be 
started. So, in addition to the "Shutting down" message, there is a "Worker X 
failed to start" message in the case where the timeout expired (e.g. state is 
NOT_STARTED). That's a nice thing to have I think (should probably be a 
LOG.error).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1701) Add simple JSON mapping to storm-hbase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426633#comment-15426633
 ] 

ASF GitHub Bot commented on STORM-1701:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1427#discussion_r75329259
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -415,23 +427,35 @@
 ;; 6. wait for workers launch
 
 (log-debug "Syncing processes")
+(log-debug "Keepers: " keepers)
+(log-debug "Keep ports: " keep-ports)
+(log-debug "Reassigned executors: " reassign-executors)
 (log-debug "Assigned executors: " assigned-executors)
 (log-debug "Allocated: " allocated)
 (doseq [[id [state heartbeat]] allocated]
-  (when (not= :valid state)
-(log-message
- "Shutting down and clearing state for id " id
- ". Current supervisor time: " now
- ". State: " state
- ", Heartbeat: " (pr-str heartbeat))
-(shutdown-worker supervisor id)))
+  (let
+[worker-launchtime (:launchtime (@(:worker-launchtime-atom 
supervisor) id))]
+(when
+  (or 
+(and (not= :valid state)
+ (not= :not-started state))
+(and (= :not-started state)
+ (or (nil? worker-launchtime)
+   (is-worker-launchtime-timed-out? now worker-launchtime 
conf
+  (if (= :not-started state)
+(log-message "Worker " id " failed to start"))
--- End diff --

log-error?


> Add simple JSON mapping to storm-hbase
> --
>
> Key: STORM-1701
> URL: https://issues.apache.org/jira/browse/STORM-1701
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Kristopher Kane
>Priority: Trivial
> Fix For: 2.0.0
>
>
> storm-hbase includes a way to map Storm fields to HBase CQs.  This adds a 
> similar ability where a single field contains a flat JSON and the keys map to 
> CQs.  The flat JSON must contain the row key.  
> No intention of reverse mapping from HBaseLookUp as the existing HBaseMapper 
> works well for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1427: [STORM-1701] Port PR 1370 to 1.x-branch (Git rid o...

[abellina]

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1427#discussion_r75329259
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -415,23 +427,35 @@
 ;; 6. wait for workers launch
 
 (log-debug "Syncing processes")
+(log-debug "Keepers: " keepers)
+(log-debug "Keep ports: " keep-ports)
+(log-debug "Reassigned executors: " reassign-executors)
 (log-debug "Assigned executors: " assigned-executors)
 (log-debug "Allocated: " allocated)
 (doseq [[id [state heartbeat]] allocated]
-  (when (not= :valid state)
-(log-message
- "Shutting down and clearing state for id " id
- ". Current supervisor time: " now
- ". State: " state
- ", Heartbeat: " (pr-str heartbeat))
-(shutdown-worker supervisor id)))
+  (let
+[worker-launchtime (:launchtime (@(:worker-launchtime-atom 
supervisor) id))]
+(when
+  (or 
+(and (not= :valid state)
+ (not= :not-started state))
+(and (= :not-started state)
+ (or (nil? worker-launchtime)
+   (is-worker-launchtime-timed-out? now worker-launchtime 
conf
+  (if (= :not-started state)
+(log-message "Worker " id " failed to start"))
--- End diff --

log-error?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1707) Improve supervisor latency by removing 2-min wait

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426632#comment-15426632
 ] 

ASF GitHub Bot commented on STORM-1707:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1370#discussion_r75329245
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/daemon/supervisor/SyncProcessEvent.java ---
@@ -110,34 +117,47 @@ public void run() {
 keeperWorkerIds.add(entry.getKey());
 
keepPorts.add(stateHeartbeat.getHeartbeat().get_port());
 }
+if (stateHeartbeat.getState() == State.NOT_STARTED) {
+keeperWorkerIds.add(entry.getKey());
+
keepPorts.add(supervisorData.getWorkerIdsToPorts().get(entry.getKey()));
+}
 }
 Map reassignExecutors = 
getReassignExecutors(assignedExecutors, keepPorts);
 Map newWorkerIds = new HashMap<>();
 for (Integer port : reassignExecutors.keySet()) {
 newWorkerIds.put(port, Utils.uuid());
 }
+
 LOG.debug("Assigned executors: {}", assignedExecutors);
 LOG.debug("Allocated: {}", localWorkerStats);
+LOG.debug("Keeper worker ids: {}", keeperWorkerIds);
+LOG.debug("Keep ports: {}", keepPorts);
+LOG.debug("LaunchTimes: {}", 
supervisorData.getWorkerIdsToLaunchTimes());
+LOG.debug("Ids Ports: {}", 
supervisorData.getWorkerIdsToPorts());
 
 for (Map.Entry entry : 
localWorkerStats.entrySet()) {
 StateHeartbeat stateHeartbeat = entry.getValue();
-if (stateHeartbeat.getState() != State.VALID) {
+if ((stateHeartbeat.getState() != State.VALID && 
stateHeartbeat.getState() != State.NOT_STARTED) ||
+(stateHeartbeat.getState() == State.NOT_STARTED && 
isWorkerStartTimeoutExpired(entry.getKey( { 
 LOG.info("Shutting down and clearing state for id {}, 
Current supervisor time: {}, State: {}, Heartbeat: {}", entry.getKey(), now,
--- End diff --

The clojure equivalent for this logs also when the worker could not be 
started. So, in addition to the "Shutting down" message, there is a "Worker X 
failed to start" message in the case where the timeout expired (e.g. state is 
NOT_STARTED). That's a nice thing to have I think (should probably be a 
LOG.error).


> Improve supervisor latency by removing 2-min wait
> -
>
> Key: STORM-1707
> URL: https://issues.apache.org/jira/browse/STORM-1707
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> After launching workers, the supervisor waits up to 2 minutes synchronously 
> for the workers to be "launched".
> We should remove this, and instead keep track of launch time, making the 
> "killer" function smart enough to determine the difference between a worker 
> that's still launching, one that's timed out, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-2028) Exceptions in JDBCClient are hidden by subsequent SQL-Exception in close()

[Rick Moritz (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426427#comment-15426427
 ] 

Rick Moritz edited comment on STORM-2028 at 8/18/16 1:20 PM:
-

>From my analysis, this issue is restricted to that class, so I think you 
>should be able to solve it within that scope.


was (Author: rpcmoritz):
>From my analysis, this is issue is restricted to that class, so I think you 
>should be able to solve it within that scope.

> Exceptions in JDBCClient are hidden by subsequent SQL-Exception in close()
> --
>
> Key: STORM-2028
> URL: https://issues.apache.org/jira/browse/STORM-2028
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-jdbc
>Affects Versions: 2.0.0
>Reporter: Rick Moritz
>Priority: Minor
>  Labels: newbie
>
> When an Exception is triggered in JdbcClient.executeInsertQuery there is the 
> potential for a follow-up Exception in close() to take precedence over the 
> previously thrown Exception, when triggered in the finally block. This makes 
> debugging the actual Exception impossible.
> As far as I can tell it would be better to catch the Exception form close() 
> in the finally-block, and to combine it with the existing Exception, so that 
> the key information for debugging purposes isn't lost.
> For data consistency purposes we have to make sure that the Exception from 
> closing the connection is thrown (or do we? can we be sure that a successful 
> commit has persisted the data?) but "overlapping" Exceptions have to be dealt 
> with.
> Alternatively it might be a good idea to log the Exceptions before throwing 
> them, so that the stack trace isn't lost. This is probably easier than 
> tracking in the finally block whether a previous Exception has been thrown, 
> and what to do with it.
> If there's a workaround for this, that I might have missed, to get to the 
> root of the Exception, I would also be interested in hearing, I'm currently 
> looking at a situation where jdbc fails, and there being no indication of 
> what's going on.
> I labelled this newbie-level, since the implementation is pretty trivial; but 
> the decision of which way to pursue isn't as clear to me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2028) Exceptions in JDBCClient are hidden by subsequent SQL-Exception in close()

[Rick Moritz (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426427#comment-15426427
 ] 

Rick Moritz commented on STORM-2028:


>From my analysis, this is issue is restricted to that class, so I think you 
>should be able to solve it within that scope.

> Exceptions in JDBCClient are hidden by subsequent SQL-Exception in close()
> --
>
> Key: STORM-2028
> URL: https://issues.apache.org/jira/browse/STORM-2028
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-jdbc
>Affects Versions: 2.0.0
>Reporter: Rick Moritz
>Priority: Minor
>  Labels: newbie
>
> When an Exception is triggered in JdbcClient.executeInsertQuery there is the 
> potential for a follow-up Exception in close() to take precedence over the 
> previously thrown Exception, when triggered in the finally block. This makes 
> debugging the actual Exception impossible.
> As far as I can tell it would be better to catch the Exception form close() 
> in the finally-block, and to combine it with the existing Exception, so that 
> the key information for debugging purposes isn't lost.
> For data consistency purposes we have to make sure that the Exception from 
> closing the connection is thrown (or do we? can we be sure that a successful 
> commit has persisted the data?) but "overlapping" Exceptions have to be dealt 
> with.
> Alternatively it might be a good idea to log the Exceptions before throwing 
> them, so that the stack trace isn't lost. This is probably easier than 
> tracking in the finally block whether a previous Exception has been thrown, 
> and what to do with it.
> If there's a workaround for this, that I might have missed, to get to the 
> root of the Exception, I would also be interested in hearing, I'm currently 
> looking at a situation where jdbc fails, and there being no indication of 
> what's going on.
> I labelled this newbie-level, since the implementation is pretty trivial; but 
> the decision of which way to pursue isn't as clear to me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: error comes when set topology.acker.executors as 0

[杨浩]

We are in a big company, and the infra group have changed the storm source
code , at the same time it's not easy to drive the group to update it.
Our project is a legacy system written by others, and we don't want to
reconstruct it. I let the acker number down, and add the batch size, data
in kafka can be proceessed now. But as the data grows, the same program may
reproduce

2016-08-18 9:18 GMT+08:00 Manu Zhang :

> Hi,
>
> That is pretty old version of Storm.
> I'd recommend to upgrade to Storm 0.9.6 or 0.10.x.
>
> Thanks,
> Manu
>
> On Wed, Aug 17, 2016 at 2:43 PM 杨浩  wrote:
>
> > -  0.9.1
> > - more than 250M B/s some time
> >
> > The flow is growing bigger and bigger and we use offline program to
> update
> > the realtime result. We don't care so much about real-time and stability,
> > we want the program to process the high throughput.
> >
> > 2016-08-17 12:16 GMT+08:00 Manu Zhang :
> >
> > > Hi Hao,
> > >
> > > A few questions
> > >
> > > - Which version of storm are you using ?
> > > - What's the current throughput  number ?
> > > - What's your expected throughput number ?
> > >
> > > You may try larger batch size.
> > >
> > > Thanks,
> > > Manu
> > >
> > > On Wed, Aug 17, 2016 at 11:10 AM 杨浩  wrote:
> > >
> > > > thanks for that, is there any other method to improve the throuput
> when
> > > > using trident
> > > >
> > > > 2016-08-17 11:05 GMT+08:00 Manu Zhang :
> > > >
> > > > > Hi,
> > > > >
> > > > > Trident must have ackers to work. You can't set
> > > topology.acker.executors
> > > > as
> > > > > 0.
> > > > >
> > > > > Thanks,
> > > > > Manu
> > > > >
> > > > > On Wed, Aug 17, 2016 at 10:50 AM 杨浩  wrote:
> > > > >
> > > > > > I want to improve the throughput of Storm, and I
> > > > > > set topology.acker.executors as 0, then program runs into error.
> I
> > > > write
> > > > > > storm using trident, please help to solve the program
> > > > > > [image: 内嵌图片 1]
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Commented] (STORM-2045) NPE in SpoutExecutor in 2.0 branch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426357#comment-15426357
 ] 

ASF GitHub Bot commented on STORM-2045:
---

GitHub user unsleepy22 opened a pull request:

https://github.com/apache/storm/pull/1634

[STORM-2045] fixed SpoutExecutor NPE



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/unsleepy22/storm executor-npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit 7cf7647e546201d3f175860526888cd92b3927b3
Author: 卫乐 
Date:   2016-08-18T12:30:11Z

fixed SpoutExecutor NPE




> NPE in SpoutExecutor in 2.0 branch
> --
>
> Key: STORM-2045
> URL: https://issues.apache.org/jira/browse/STORM-2045
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Cody
>Assignee: Cody
> Fix For: 2.0.0
>
>
> This issue was raised in [STORM-1949], but since the original issue mainly 
> discusses about whether to disable ABP by default, I'd like to pick this NPE 
> as another issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-2045) NPE in SpoutExecutor in 2.0 branch

[Cody (JIRA)]

Cody created STORM-2045:
---

 Summary: NPE in SpoutExecutor in 2.0 branch
 Key: STORM-2045
 URL: https://issues.apache.org/jira/browse/STORM-2045
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 2.0.0
Reporter: Cody
Assignee: Cody
 Fix For: 2.0.0


This issue was raised in [STORM-1949], but since the original issue mainly 
discusses about whether to disable ABP by default, I'd like to pick this NPE as 
another issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1695) Create trident spout that uses the new kafka consumer API

[Hugo Louro (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426213#comment-15426213
 ] 

Hugo Louro commented on STORM-1695:
---

Yes. Ideally, this storm-kafka-client will become the de facto, and the old one 
will eventually be deprecated.

> Create trident spout that uses the new kafka consumer API
> -
>
> Key: STORM-1695
> URL: https://issues.apache.org/jira/browse/STORM-1695
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Thomas Graves
>Assignee: Hugo Louro
>
> In storm-822 we added a new kafka spout 
> (org.apache.storm.kafka.spout.KafkaSpout) that uses the new consumer Api in 
> Kafka 0.9.  We decided in that one to handle the Trident support separately.  
> So this jira is to add Trident support for it similar to what 
> OpaqueTridentKafkaSpout does for the kafka old consumer api.  We need to 
> support the new consumer api to allow access to a secure Kafka cluster which 
> was added in Kafka 0.9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426102#comment-15426102
 ] 

ASF GitHub Bot commented on STORM-1881:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1480
  
STORM-2016 is on reviewing, and it should help this get resolved.


> storm-redis is missing dependant libraries in distribution
> --
>
> Key: STORM-1881
> URL: https://issues.apache.org/jira/browse/STORM-1881
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-redis
>Affects Versions: 1.0.1
>Reporter: Daniel Klessing
>
> Despite the documentation on 
> http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not 
> enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the 
> {{RedisKeyValueStateProvider}} working. Depending jedis and 
> apache-commons-pool2 jars are missing and must be copied by hand to get it 
> working. Else one is greeted with exception stack traces like:
> {code}
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.commons.pool2.impl.GenericObjectPoolConfig
> {code}
> or
> {code}
> Caused by: java.lang.ClassNotFoundException: 
> redis.clients.jedis.JedisPoolConfig
> {code}
> Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to 
> {{extlib}} solves the issue.
> It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide 
> documentation, which libraries have to be made available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm issue #1480: STORM-1881: Create fat-jar of storm-redis which contains ...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1480
  
STORM-2016 is on reviewing, and it should help this get resolved.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1446) Compile the Calcite logical plan to Storm physical plan

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426095#comment-15426095
 ] 

Jungtaek Lim commented on STORM-1446:
-

[~wheat9]
STORM-1433 is introducing the change, so please let me know if you think the 
change covers this issue. 
https://github.com/apache/storm/pull/1633

Thanks in advance!

> Compile the Calcite logical plan to Storm physical plan
> ---
>
> Key: STORM-1446
> URL: https://issues.apache.org/jira/browse/STORM-1446
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-sql
>Reporter: Haohui Mai
>
> As suggested in 
> https://issues.apache.org/jira/browse/STORM-1040?focusedCommentId=15036651=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15036651,
>  compiling the logical plan from Calcite down to Storm physical plan will 
> clarify the implementation of StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426063#comment-15426063
 ] 

ASF GitHub Bot commented on STORM-1434:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1633#discussion_r75264055
  
--- Diff: 
external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/ExprCompiler.java
 ---
@@ -67,8 +68,9 @@ public ExprCompiler(PrintWriter pw, JavaTypeFactory 
typeFactory) {
   public String visitInputRef(RexInputRef rexInputRef) {
 String name = reserveName();
 String typeName = javaTypeName(rexInputRef);
+String boxedTypeName = boxedJavaTypeName(rexInputRef);
--- End diff --

Actually converting Object to primitive type is allowed for Java 7 or 
upper, but Janino throws error while compiling script so I just converted to 
boxed type.


> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1633#discussion_r75264055
  
--- Diff: 
external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/ExprCompiler.java
 ---
@@ -67,8 +68,9 @@ public ExprCompiler(PrintWriter pw, JavaTypeFactory 
typeFactory) {
   public String visitInputRef(RexInputRef rexInputRef) {
 String name = reserveName();
 String typeName = javaTypeName(rexInputRef);
+String boxedTypeName = boxedJavaTypeName(rexInputRef);
--- End diff --

Actually converting Object to primitive type is allowed for Java 7 or 
upper, but Janino throws error while compiling script so I just converted to 
boxed type.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1633
  
When reviewing, please share your idea if you find something to optimize 
further.
I didn't play with Trident hardly so I'm not sure I'm following the best 
practice.

I didn't touch UDF yet since coverage of STORM-1434 would be too broad if 
STORM-1434 contains UDF. Will file an issue to track and plan to address that.

Also STORM-2016 should be addressed soon in order to run the topology.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426058#comment-15426058
 ] 

ASF GitHub Bot commented on STORM-1434:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1633
  
When reviewing, please share your idea if you find something to optimize 
further.
I didn't play with Trident hardly so I'm not sure I'm following the best 
practice.

I didn't touch UDF yet since coverage of STORM-1434 would be too broad if 
STORM-1434 contains UDF. Will file an issue to track and plan to address that.

Also STORM-2016 should be addressed soon in order to run the topology.


> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request #1633: STORM-1434 Support the GROUP BY clause in StormSQL

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1633

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime

This change can handle the below sql statements in runtime:

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM 
ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE SUMMARY_ORDERS (ID INT PRIMARY KEY, UNIT_PRICE_AVG 
DOUBLE, SUM_OF_QUANTITY INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO SUMMARY_ORDERS SELECT ID, AVG(UNIT_PRICE) AS UNIT_PRICE_AVG, 
SUM(QUANTITY) AS SUM_OF_QUANTITY FROM ORDERS GROUP BY ID
```

This can be easily backported to 1.x branch since it only touches storm-sql 
module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1454

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit fc93fdad03c25967e7b00cc9c2289f46d71fb0a9
Author: Jungtaek Lim 
Date:   2016-08-18T07:32:46Z

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426050#comment-15426050
 ] 

ASF GitHub Bot commented on STORM-1434:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1633

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime

This change can handle the below sql statements in runtime:

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM 
ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE SUMMARY_ORDERS (ID INT PRIMARY KEY, UNIT_PRICE_AVG 
DOUBLE, SUM_OF_QUANTITY INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO SUMMARY_ORDERS SELECT ID, AVG(UNIT_PRICE) AS UNIT_PRICE_AVG, 
SUM(QUANTITY) AS SUM_OF_QUANTITY FROM ORDERS GROUP BY ID
```

This can be easily backported to 1.x branch since it only touches storm-sql 
module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1454

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit fc93fdad03c25967e7b00cc9c2289f46d71fb0a9
Author: Jungtaek Lim 
Date:   2016-08-18T07:32:46Z

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime




> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-2044) Nimbus should not make assignments crazily when Pacemaker goes down and up

[Yuzhao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuzhao Chen updated STORM-2044:
---
Summary: Nimbus should not make assignments crazily when Pacemaker goes 
down and up  (was: Nimbus should not make assignments crazily when Pacemaker 
goes down)

> Nimbus should not make assignments crazily when Pacemaker goes down and up
> --
>
> Key: STORM-2044
> URL: https://issues.apache.org/jira/browse/STORM-2044
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 1.0.2
> Environment: CentOS 6.5
>Reporter: Yuzhao Chen
>  Labels: patch
> Fix For: 1.1.0
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> Now pacemaker is a stand-alone service and no HA is supported. When 
> it goes down, all the workers's heartbeats will be lost. It will take a long 
> time to recover even if pacemaker goes up immediately if there are dozens GB 
> of heartbeats. During the time worker heartbeats are not restored completely, 
> Nimbus will think these workers are dead because of heartbeats timeout and 
> reassign these "dead" workers continuously until heartbeats restore to 
> normal. So, during recovery time, many topologies will be reassigned 
> continuously and the throughout will goes very down.  
> This is not acceptable. 
> So i think, pacemaker is not suitable for production if the problem 
> above exists.
>i think several ways to solve this problem:
>   1. pacemaker HA
>   2. when pacemaker does down, notice nimbus not to reassign any 
> more until it recover



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Yuzhao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425987#comment-15425987
 ] 

Yuzhao Chen edited comment on STORM-1949 at 8/18/16 7:01 AM:
-

I have test storm 1.0.1 ABP in our staging cluster[ 10 machine nodes and a 
topology with 32 workers, 120 executors ] on, the topology will stall within 10 
mins very possibly
i see that when all the topology tasks throttle off, the backpressure root 
node on ZK still have child nodes which cause   the "stall", if i delete the 
child node manually, the spouts will continue to consume normally(but will 
again stall within 10 minutes), so  i agree with @Roshan Naik


was (Author: danny0405):
I have test storm 1.0.1 ABP in our staging cluster[ 10 machine nodes and a 
topology with 32 workers, 120 executors ] on, the topology will stall within 10 
mins very possibly, i agree with @Roshan Naik 

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
> Attachments: 1.x-branch-works-perfect.png
>
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology

[Yuzhao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425987#comment-15425987
 ] 

Yuzhao Chen commented on STORM-1949:


I have test storm 1.0.1 ABP in our staging cluster[ 10 machine nodes and a 
topology with 32 workers, 120 executors ] on, the topology will stall within 10 
mins very possibly, i agree with @Roshan Naik 

> Backpressure can cause spout to stop emitting and stall topology
> 
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
> Attachments: 1.x-branch-works-perfect.png
>
>
> Problem can be reproduced by this [Word count 
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt 
> instances.
> The problem is more easily reproduced with WC topology as it causes an 
> explosion of tuples due to splitting a sentence tuple into word tuples. As 
> the bolts have to process more tuples than the  spout is producing, spout 
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but 
> typically under 10 mins. 
> *My theory:*  I suspect there is a race condition in the way ZK is being 
> utilized to enable/disable back pressure. When congested (i.e pressure 
> exceeds high water mark), the bolt's worker records this congested situation 
> in ZK by creating a node. Once the congestion is reduced below the low water 
> mark, it deletes this node. 
> The spout's worker has setup a watch on the parent node, expecting a callback 
> whenever there is change in the child nodes. On receiving the callback the 
> spout's worker lists the parent node to check if there are 0 or more child 
> nodes it is essentially trying to figure out the nature of state change 
> in ZK to determine whether to throttle or not. Subsequently  it setsup 
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these 
> ZK nodes. Between the time the worker receives a callback and sets up the 
> next watch.. many changes may have undergone in ZK which will go unnoticed by 
> the spout. 
> The condition that the bolts are no longer congested may not get noticed as a 
> result of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)