[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426639#comment-15426639
 ] 

ASF GitHub Bot commented on STORM-1038:
---

GitHub user hsun-cnnxty reopened a pull request:

https://github.com/apache/storm/pull/1591

STORM-1038: Upgrade netty to 4.x in 1.x-branch

This is to add the feature to 1.x-branch.  The original PR for master 
branch is #728.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsun-cnnxty/storm 1.x-branch-netty4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1591


commit 57fbccc7e0d710ccaf04a5a828f0ff4cf29ec855
Author: Hang Sun 
Date:   2016-07-25T22:30:54Z

STORM-1038: Upgraded netty to 4.x

commit 144b6eddf314ce9a0b5beb8c99500c56ff0b8ec0
Author: Hang Sun 
Date:   2016-07-30T23:42:52Z

STORM-1038: removed unused imports




> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426638#comment-15426638
 ] 

ASF GitHub Bot commented on STORM-1038:
---

Github user hsun-cnnxty closed the pull request at:

https://github.com/apache/storm/pull/1591


> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1701) Add simple JSON mapping to storm-hbase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426633#comment-15426633
 ] 

ASF GitHub Bot commented on STORM-1701:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1427#discussion_r75329259
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -415,23 +427,35 @@
 ;; 6. wait for workers launch
 
 (log-debug "Syncing processes")
+(log-debug "Keepers: " keepers)
+(log-debug "Keep ports: " keep-ports)
+(log-debug "Reassigned executors: " reassign-executors)
 (log-debug "Assigned executors: " assigned-executors)
 (log-debug "Allocated: " allocated)
 (doseq [[id [state heartbeat]] allocated]
-  (when (not= :valid state)
-(log-message
- "Shutting down and clearing state for id " id
- ". Current supervisor time: " now
- ". State: " state
- ", Heartbeat: " (pr-str heartbeat))
-(shutdown-worker supervisor id)))
+  (let
+[worker-launchtime (:launchtime (@(:worker-launchtime-atom 
supervisor) id))]
+(when
+  (or 
+(and (not= :valid state)
+ (not= :not-started state))
+(and (= :not-started state)
+ (or (nil? worker-launchtime)
+   (is-worker-launchtime-timed-out? now worker-launchtime 
conf
+  (if (= :not-started state)
+(log-message "Worker " id " failed to start"))
--- End diff --

log-error?


> Add simple JSON mapping to storm-hbase
> --
>
> Key: STORM-1701
> URL: https://issues.apache.org/jira/browse/STORM-1701
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Kristopher Kane
>Priority: Trivial
> Fix For: 2.0.0
>
>
> storm-hbase includes a way to map Storm fields to HBase CQs.  This adds a 
> similar ability where a single field contains a flat JSON and the keys map to 
> CQs.  The flat JSON must contain the row key.  
> No intention of reverse mapping from HBaseLookUp as the existing HBaseMapper 
> works well for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1707) Improve supervisor latency by removing 2-min wait

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426632#comment-15426632
 ] 

ASF GitHub Bot commented on STORM-1707:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1370#discussion_r75329245
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/daemon/supervisor/SyncProcessEvent.java ---
@@ -110,34 +117,47 @@ public void run() {
 keeperWorkerIds.add(entry.getKey());
 
keepPorts.add(stateHeartbeat.getHeartbeat().get_port());
 }
+if (stateHeartbeat.getState() == State.NOT_STARTED) {
+keeperWorkerIds.add(entry.getKey());
+
keepPorts.add(supervisorData.getWorkerIdsToPorts().get(entry.getKey()));
+}
 }
 Map reassignExecutors = 
getReassignExecutors(assignedExecutors, keepPorts);
 Map newWorkerIds = new HashMap<>();
 for (Integer port : reassignExecutors.keySet()) {
 newWorkerIds.put(port, Utils.uuid());
 }
+
 LOG.debug("Assigned executors: {}", assignedExecutors);
 LOG.debug("Allocated: {}", localWorkerStats);
+LOG.debug("Keeper worker ids: {}", keeperWorkerIds);
+LOG.debug("Keep ports: {}", keepPorts);
+LOG.debug("LaunchTimes: {}", 
supervisorData.getWorkerIdsToLaunchTimes());
+LOG.debug("Ids Ports: {}", 
supervisorData.getWorkerIdsToPorts());
 
 for (Map.Entry entry : 
localWorkerStats.entrySet()) {
 StateHeartbeat stateHeartbeat = entry.getValue();
-if (stateHeartbeat.getState() != State.VALID) {
+if ((stateHeartbeat.getState() != State.VALID && 
stateHeartbeat.getState() != State.NOT_STARTED) ||
+(stateHeartbeat.getState() == State.NOT_STARTED && 
isWorkerStartTimeoutExpired(entry.getKey( { 
 LOG.info("Shutting down and clearing state for id {}, 
Current supervisor time: {}, State: {}, Heartbeat: {}", entry.getKey(), now,
--- End diff --

The clojure equivalent for this logs also when the worker could not be 
started. So, in addition to the "Shutting down" message, there is a "Worker X 
failed to start" message in the case where the timeout expired (e.g. state is 
NOT_STARTED). That's a nice thing to have I think (should probably be a 
LOG.error).


> Improve supervisor latency by removing 2-min wait
> -
>
> Key: STORM-1707
> URL: https://issues.apache.org/jira/browse/STORM-1707
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> After launching workers, the supervisor waits up to 2 minutes synchronously 
> for the workers to be "launched".
> We should remove this, and instead keep track of launch time, making the 
> "killer" function smart enough to determine the difference between a worker 
> that's still launching, one that's timed out, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2045) NPE in SpoutExecutor in 2.0 branch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426357#comment-15426357
 ] 

ASF GitHub Bot commented on STORM-2045:
---

GitHub user unsleepy22 opened a pull request:

https://github.com/apache/storm/pull/1634

[STORM-2045] fixed SpoutExecutor NPE



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/unsleepy22/storm executor-npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit 7cf7647e546201d3f175860526888cd92b3927b3
Author: 卫乐 
Date:   2016-08-18T12:30:11Z

fixed SpoutExecutor NPE




> NPE in SpoutExecutor in 2.0 branch
> --
>
> Key: STORM-2045
> URL: https://issues.apache.org/jira/browse/STORM-2045
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Cody
>Assignee: Cody
> Fix For: 2.0.0
>
>
> This issue was raised in [STORM-1949], but since the original issue mainly 
> discusses about whether to disable ABP by default, I'd like to pick this NPE 
> as another issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426102#comment-15426102
 ] 

ASF GitHub Bot commented on STORM-1881:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1480
  
STORM-2016 is on reviewing, and it should help this get resolved.


> storm-redis is missing dependant libraries in distribution
> --
>
> Key: STORM-1881
> URL: https://issues.apache.org/jira/browse/STORM-1881
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-redis
>Affects Versions: 1.0.1
>Reporter: Daniel Klessing
>
> Despite the documentation on 
> http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not 
> enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the 
> {{RedisKeyValueStateProvider}} working. Depending jedis and 
> apache-commons-pool2 jars are missing and must be copied by hand to get it 
> working. Else one is greeted with exception stack traces like:
> {code}
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.commons.pool2.impl.GenericObjectPoolConfig
> {code}
> or
> {code}
> Caused by: java.lang.ClassNotFoundException: 
> redis.clients.jedis.JedisPoolConfig
> {code}
> Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to 
> {{extlib}} solves the issue.
> It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide 
> documentation, which libraries have to be made available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426063#comment-15426063
 ] 

ASF GitHub Bot commented on STORM-1434:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1633#discussion_r75264055
  
--- Diff: 
external/sql/storm-sql-core/src/jvm/org/apache/storm/sql/compiler/ExprCompiler.java
 ---
@@ -67,8 +68,9 @@ public ExprCompiler(PrintWriter pw, JavaTypeFactory 
typeFactory) {
   public String visitInputRef(RexInputRef rexInputRef) {
 String name = reserveName();
 String typeName = javaTypeName(rexInputRef);
+String boxedTypeName = boxedJavaTypeName(rexInputRef);
--- End diff --

Actually converting Object to primitive type is allowed for Java 7 or 
upper, but Janino throws error while compiling script so I just converted to 
boxed type.


> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426058#comment-15426058
 ] 

ASF GitHub Bot commented on STORM-1434:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1633
  
When reviewing, please share your idea if you find something to optimize 
further.
I didn't play with Trident hardly so I'm not sure I'm following the best 
practice.

I didn't touch UDF yet since coverage of STORM-1434 would be too broad if 
STORM-1434 contains UDF. Will file an issue to track and plan to address that.

Also STORM-2016 should be addressed soon in order to run the topology.


> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426050#comment-15426050
 ] 

ASF GitHub Bot commented on STORM-1434:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1633

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime

This change can handle the below sql statements in runtime:

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO LARGE_ORDERS SELECT ID, UNIT_PRICE * QUANTITY AS TOTAL FROM 
ORDERS WHERE UNIT_PRICE * QUANTITY > 50
```

```
CREATE EXTERNAL TABLE ORDERS (ID INT PRIMARY KEY, UNIT_PRICE INT, QUANTITY 
INT) LOCATION 'kafka://localhost:2181/brokers?topic=orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
CREATE EXTERNAL TABLE SUMMARY_ORDERS (ID INT PRIMARY KEY, UNIT_PRICE_AVG 
DOUBLE, SUM_OF_QUANTITY INT) LOCATION 
'kafka://localhost:2181/brokers?topic=large_orders' TBLPROPERTIES 
'{"producer":{"bootstrap.servers":"localhost:9092","acks":"1","key.serializer":"org.apache.storm.kafka.IntSerializer","value.serializer":"org.apache.storm.kafka.ByteBufferSerializer"}}'
INSERT INTO SUMMARY_ORDERS SELECT ID, AVG(UNIT_PRICE) AS UNIT_PRICE_AVG, 
SUM(QUANTITY) AS SUM_OF_QUANTITY FROM ORDERS GROUP BY ID
```

This can be easily backported to 1.x branch since it only touches storm-sql 
module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1454

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit fc93fdad03c25967e7b00cc9c2289f46d71fb0a9
Author: Jungtaek Lim 
Date:   2016-08-18T07:32:46Z

STORM-1434 Support the GROUP BY clause in StormSQL

* Support GROUP BY for Trident
* Implement basic functions for aggregation
* Change the way of converting Calcite logical plan to Trident logical plan
** before: creating codes and compile them
** after: use Trident features, only creating code block if evaluation is 
needed
*** Janino comes in to help evaluating code block in runtime




> Support the GROUP BY clause in StormSQL
> ---
>
> Key: STORM-1434
> URL: https://issues.apache.org/jira/browse/STORM-1434
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-sql
>Reporter: Haohui Mai
>
> This jira tracks the effort of implement the support `GROUP BY` clause in 
> StormSQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-822) Kafka Spout New Consumer API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425899#comment-15425899
 ] 

ASF GitHub Bot commented on STORM-822:
--

Github user jianbzhou commented on the issue:

https://github.com/apache/storm/pull/1131
  
@hmcl and all, the new spout is fit for the at least once semantics and 
works fine for us, thanks a lot! Very recently one of our key customers asked 
to use a at most once implementation. Do we have any plan to have a 
at-most-once implementation? They set the topology.acker.executors=0 and found 
the spout is not working. Could you please help to evaluate - 1, will we 
implement this? 2. roughly how long time needed? Thanks

**Requirement from customer** - “topology.acker.executors” is a Storm 
parameter, which refers to at-least-once when it’s not 0, and at-most-once if 
it’s 0. We want to know do we have a at-most-once implementation?


> Kafka Spout New Consumer API
> 
>
> Key: STORM-822
> URL: https://issues.apache.org/jira/browse/STORM-822
> Project: Apache Storm
>  Issue Type: Story
>  Components: storm-kafka
>Reporter: Thomas Becker
>Assignee: Hugo Louro
> Fix For: 1.0.0, 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1701) Add simple JSON mapping to storm-hbase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425764#comment-15425764
 ] 

ASF GitHub Bot commented on STORM-1701:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1427#discussion_r75236698
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/supervisor.clj ---
@@ -415,23 +427,35 @@
 ;; 6. wait for workers launch
 
 (log-debug "Syncing processes")
+(log-debug "Keepers: " keepers)
+(log-debug "Keep ports: " keep-ports)
+(log-debug "Reassigned executors: " reassign-executors)
 (log-debug "Assigned executors: " assigned-executors)
 (log-debug "Allocated: " allocated)
 (doseq [[id [state heartbeat]] allocated]
-  (when (not= :valid state)
-(log-message
- "Shutting down and clearing state for id " id
- ". Current supervisor time: " now
- ". State: " state
- ", Heartbeat: " (pr-str heartbeat))
-(shutdown-worker supervisor id)))
+  (let
+[worker-launchtime (:launchtime (@(:worker-launchtime-atom 
supervisor) id))]
+(when
+  (or 
+(and (not= :valid state)
+ (not= :not-started state))
+(and (= :not-started state)
+ (or (nil? worker-launchtime)
+   (is-worker-launchtime-timed-out? now worker-launchtime 
conf
+  (if (= :not-started state)
+(log-message "Worker " id " failed to start"))
+  (log-message
+"Shutting down and clearing state for id " id
+". Current supervisor time: " now
+". State: " state
+", Heartbeat: " (pr-str heartbeat))
+  (shutdown-worker supervisor id
 (let [valid-new-worker-ids (get-valid-new-worker-ids conf supervisor 
reassign-executors new-worker-ids)]
   (ls-approved-workers! local-state
 (merge
   (select-keys (ls-approved-workers local-state)
 (keys keepers))
-  valid-new-worker-ids))
-  (wait-for-workers-launch conf (keys valid-new-worker-ids)
--- End diff --

since we removed this, we should also remove the wait-for-workers-launch 
private functions.


> Add simple JSON mapping to storm-hbase
> --
>
> Key: STORM-1701
> URL: https://issues.apache.org/jira/browse/STORM-1701
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Kristopher Kane
>Priority: Trivial
> Fix For: 2.0.0
>
>
> storm-hbase includes a way to map Storm fields to HBase CQs.  This adds a 
> similar ability where a single field contains a flat JSON and the keys map to 
> CQs.  The flat JSON must contain the row key.  
> No intention of reverse mapping from HBaseLookUp as the existing HBaseMapper 
> works well for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424606#comment-15424606
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/1628
  
@HeartSaVioR Technically I don't think the bylaws cover changing minimum 
requirements beyond a code change.  If you want to modify the bylaws in that 
way I would be happy to support you in this. 

+1


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2042) Nimbus client connections not closed properly causing connection leaks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424573#comment-15424573
 ] 

ASF GitHub Bot commented on STORM-2042:
---

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/1630
  
+1


> Nimbus client connections not closed properly causing connection leaks
> --
>
> Key: STORM-2042
> URL: https://issues.apache.org/jira/browse/STORM-2042
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.x
>
>
> The nimbus client connections are not closed properly causing connection 
> leaks. After the number of connections exceed nimbus.thrift.threads, a 
> RejectedExecutionException is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2042) Nimbus client connections not closed properly causing connection leaks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424556#comment-15424556
 ] 

ASF GitHub Bot commented on STORM-2042:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1630
  
+1


> Nimbus client connections not closed properly causing connection leaks
> --
>
> Key: STORM-2042
> URL: https://issues.apache.org/jira/browse/STORM-2042
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.x
>
>
> The nimbus client connections are not closed properly causing connection 
> leaks. After the number of connections exceed nimbus.thrift.threads, a 
> RejectedExecutionException is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424472#comment-15424472
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1627
  
@knusbaum I made the changes suggested. Also removed an extra fn wrapper on 
that function triggered by the timer. 


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2042) Nimbus client connections not closed properly causing connection leaks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424133#comment-15424133
 ] 

ASF GitHub Bot commented on STORM-2042:
---

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/1631

[STORM-2042] Nimbus client connections not closed properly causing 
connection leaks

https://github.com/apache/storm/pull/1630 ported to master branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2042-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1631.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1631


commit 6c3cd2a3c3c39dc79e2cae576546876bf2efab0c
Author: Arun Mahadevan 
Date:   2016-08-17T08:32:44Z

[STORM-2042] Nimbus client connections not closed properly causing 
connection leaks

Implement AutoCloseable and close the nimbus client connections properly.




> Nimbus client connections not closed properly causing connection leaks
> --
>
> Key: STORM-2042
> URL: https://issues.apache.org/jira/browse/STORM-2042
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.x
>
>
> The nimbus client connections are not closed properly causing connection 
> leaks. After the number of connections exceed nimbus.thrift.threads, a 
> RejectedExecutionException is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2042) Nimbus client connections not closed properly causing connection leaks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424115#comment-15424115
 ] 

ASF GitHub Bot commented on STORM-2042:
---

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/1630

[STORM-2042] Nimbus client connections not closed properly causing 
connection leaks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2042

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1630.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1630


commit 495c70563cb72241bdebf7e70429c38e60484443
Author: Arun Mahadevan 
Date:   2016-08-17T08:32:44Z

[STORM-2042] Nimbus client connections not closed properly causing 
connection leaks

Implement AutoCloseable and close the nimbus client connections properly.




> Nimbus client connections not closed properly causing connection leaks
> --
>
> Key: STORM-2042
> URL: https://issues.apache.org/jira/browse/STORM-2042
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.x
>
>
> The nimbus client connections are not closed properly causing connection 
> leaks. After the number of connections exceed nimbus.thrift.threads, a 
> RejectedExecutionException is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423941#comment-15423941
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user arunmahadevan commented on the issue:

https://github.com/apache/storm/pull/1628
  
+1


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423734#comment-15423734
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1628
  
@harshach Thanks for the quick follow up. I just voted to VOTE thread. 
Since we've seen some PMCs' positive feedback I guess it should be reflected to 
vote result.


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423718#comment-15423718
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1628
  
@HeartSaVioR started a thread or we can use the PR votes for consensus 
among PMC.



> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423716#comment-15423716
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1628
  
+1


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423616#comment-15423616
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1628
  
@HeartSaVioR whats the formal process? did we ever define that. I already 
started discussion thread.  There are 3 pmc's who said +1 and so far there is 
no -1. Its past 72 hrs as well.


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423605#comment-15423605
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1628
  
+1 for the change but would like to hold this until we finish a formal 
process.

For me changing minimum requirement is an important decision for the 
project so handling this with only general consensus seems not enough.
I don't remember how we change minimum requirement as Java 7, but ideally 
we need to VOTE to confirm/record the decision of the PMC. Some PMC members 
already gave positive feedbacks so it tends to be passed, but it would be 
better to hear more voices from other PMC members, especially representing OK 
(+1) or NO (-1).


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423572#comment-15423572
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1628
  
@HeartSaVioR done.


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423522#comment-15423522
 ] 

ASF GitHub Bot commented on STORM-2041:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1628
  
Please modify .travis.yml as well so that build doesn't occur on Oracle JDK 
7.


> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423483#comment-15423483
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user abellina commented on a diff in the pull request:

https://github.com/apache/storm/pull/1627#discussion_r75030675
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -743,7 +736,7 @@
 (fn [& args]
   (check-credentials-changed)
   (if ((:storm-conf worker) 
TOPOLOGY-BACKPRESSURE-ENABLE)
-(check-throttle-changed
+(topology-backpressure-callback
--- End diff --

yeah, agreed. I'll add another timer for this.


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423209#comment-15423209
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1627
  
+1 for the changes. My comments are suggestions for a couple more.
The travis failures look unrelated. Storm Core passes.


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2041) Make Java 8 as minimum requirement for 2.0 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423199#comment-15423199
 ] 

ASF GitHub Bot commented on STORM-2041:
---

GitHub user harshach opened a pull request:

https://github.com/apache/storm/pull/1628

STORM-2041. Make Java 8 as minimum requirement for 2.0 release.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harshach/incubator-storm STORM-2041

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1628.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1628


commit e2eb8022ef9da6d6506a669168bd178e06e0aeb0
Author: Sriharsha Chintalapani 
Date:   2016-08-16T18:28:40Z

STORM-2041. Make Java 8 as minimum requirement for 2.0 release.




> Make Java 8 as minimum requirement for 2.0 release
> --
>
> Key: STORM-2041
> URL: https://issues.apache.org/jira/browse/STORM-2041
> Project: Apache Storm
>  Issue Type: Task
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423183#comment-15423183
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user knusbaum commented on a diff in the pull request:

https://github.com/apache/storm/pull/1627#discussion_r74990425
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -316,7 +316,6 @@
   :load-mapping (LoadMapping.)
   :assignment-versions assignment-versions
   :backpressure (atom false) ;; whether this worker is going slow
-  :transfer-backpressure (atom false) ;; if the transfer queue is 
backed-up
   :backpressure-trigger (atom false) ;; a trigger for synchronization 
with executors
--- End diff --

Let's just make this an Object. The fact that it's an atom is misleading.


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423166#comment-15423166
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user knusbaum commented on a diff in the pull request:

https://github.com/apache/storm/pull/1627#discussion_r74987767
  
--- Diff: storm-core/src/clj/org/apache/storm/daemon/worker.clj ---
@@ -743,7 +736,7 @@
 (fn [& args]
   (check-credentials-changed)
   (if ((:storm-conf worker) 
TOPOLOGY-BACKPRESSURE-ENABLE)
-(check-throttle-changed
+(topology-backpressure-callback
--- End diff --

This feels like the wrong place for this. 
For some reason the backpressure is tacked onto the credentials timer?

I know this was already the way it worked, but maybe there's a better place 
for it now? I don't think it's good that the value of the 
`task.credentials.poll.secs` config affects the frequency of backpressure 
polling.


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1234) port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422763#comment-15422763
 ] 

ASF GitHub Bot commented on STORM-1234:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1625


> port  backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java
> 
>
> Key: STORM-1234
> URL: https://issues.apache.org/jira/browse/STORM-1234
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> to junit test conversion



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1234) port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422760#comment-15422760
 ] 

ASF GitHub Bot commented on STORM-1234:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1625
  
+1 Build error is from Travis CI.
I pulled this and build from my local and succeed.
Nice work @abhishekagarwal87 


> port  backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java
> 
>
> Key: STORM-1234
> URL: https://issues.apache.org/jira/browse/STORM-1234
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> to junit test conversion



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422526#comment-15422526
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1626
  
Addressed all fixes from 1.x. It's in sync.


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422512#comment-15422512
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
Done. Fixed things after receiving +1 are here:

1. nimbus.clj: just log exception instead of crashing nimbus when 
blob-rm-dependency-jars-in-topology
2. DependencyPropertiesParser.java: check --jars parameter string is empty, 
and treat it as empty list (also add new unit test testing this change)


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422501#comment-15422501
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
I found a bug while testing other stuff with applied cluster. 
StormSubmitter doesn't handle empty --jars option properly. Will address 
this shortly both this and PR for master branch.


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421575#comment-15421575
 ] 

ASF GitHub Bot commented on STORM-2039:
---

GitHub user abellina reopened a pull request:

https://github.com/apache/storm/pull/1627

STORM-2039: backpressure refactoring in worker and executor

For 1.x branch, will submit a PR for 2.x branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abellina/storm 
STORM-2039_backpressure_refactoring_in_worker_and_executor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 75b82f3449a2bd4ae8cf3c0999ee3f5f1defad0c
Author: Alessandro Bellina 
Date:   2016-08-15T19:28:40Z

STORM-2039: backpressure refactoring in worker and executor




> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421574#comment-15421574
 ] 

ASF GitHub Bot commented on STORM-2039:
---

Github user abellina closed the pull request at:

https://github.com/apache/storm/pull/1627


> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2039) Backpressure refactoring in worker and executor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421532#comment-15421532
 ] 

ASF GitHub Bot commented on STORM-2039:
---

GitHub user abellina opened a pull request:

https://github.com/apache/storm/pull/1627

STORM-2039: backpressure refactoring in worker and executor

For 1.x branch, will submit a PR for 2.x branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abellina/storm 
STORM-2039_backpressure_refactoring_in_worker_and_executor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 75b82f3449a2bd4ae8cf3c0999ee3f5f1defad0c
Author: Alessandro Bellina 
Date:   2016-08-15T19:28:40Z

STORM-2039: backpressure refactoring in worker and executor




> Backpressure refactoring in worker and executor
> ---
>
> Key: STORM-2039
> URL: https://issues.apache.org/jira/browse/STORM-2039
> Project: Apache Storm
>  Issue Type: Story
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
>
> * Use backpressure flags directly from disruptor queue instead of in the 
> disruptor-backpressure-handlers in worker and executor
> * Other minor refactoring (eliminate redundant function)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1766) A better algorithm server rack selection for RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420355#comment-15420355
 ] 

ASF GitHub Bot commented on STORM-1766:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1621


> A better algorithm server rack selection for RAS
> 
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
> Fix For: 2.0.0
>
>
> Currently the getBestClustering algorithm for RAS finds the "Best" 
> cluster/rack based on which rack has the most available resources this may be 
> insufficient and may cause topologies not to be able to be scheduled 
> successfully even though there are enough resources to schedule it in the 
> cluster. We attempt to find the rack with the most resources by find the rack 
> with the biggest sum of available memory + available cpu. This method is not 
> effective since it does not consider the number of slots available. This 
> method also fails in identifying racks that are not schedulable due to the 
> exhaustion of one of the resources either memory, cpu, or slots. The current 
> implementation also tries the initial scheduling on one rack and not try to 
> schedule on all the racks before giving up which may cause topologies to be 
> failed to be scheduled due to the above mentioned shortcomings in the current 
> method. Also the current method does not consider failures of workers. When 
> executors of a topology gets unassigned and needs to be scheduled again, the 
> current logic in getBestClustering may be inadequate if not complete wrong. 
> When executors needs to rescheduled due to a fault, getBestClustering will 
> likely return a cluster that is different from where the majority of 
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I 
> have come up with a ordering strategy I dub subordinate resource availability 
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the 
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM 
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU 
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and 
> can potentially schedule the most executors, while rack-2 is the worst rack 
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even 
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for 
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values 
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] 
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective 
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] 
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] 
> effective resources: 0.1951219512195122
> {code}
> The effective resource of a rack, which is also the subordinate resource, is 
> computed by: 
> {code}
> MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
> {code}
> Then we order the racks by the effective resource.
> Thus for our example:
> {code}
> Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
> {code}
> Also to deal with the presence of failures, if a topology is partially 
> scheduled, we find the rack with the most scheduled executors for 

[jira] [Commented] (STORM-1766) A better algorithm server rack selection for RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420354#comment-15420354
 ] 

ASF GitHub Bot commented on STORM-1766:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1621
  
+1


> A better algorithm server rack selection for RAS
> 
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
> Fix For: 2.0.0
>
>
> Currently the getBestClustering algorithm for RAS finds the "Best" 
> cluster/rack based on which rack has the most available resources this may be 
> insufficient and may cause topologies not to be able to be scheduled 
> successfully even though there are enough resources to schedule it in the 
> cluster. We attempt to find the rack with the most resources by find the rack 
> with the biggest sum of available memory + available cpu. This method is not 
> effective since it does not consider the number of slots available. This 
> method also fails in identifying racks that are not schedulable due to the 
> exhaustion of one of the resources either memory, cpu, or slots. The current 
> implementation also tries the initial scheduling on one rack and not try to 
> schedule on all the racks before giving up which may cause topologies to be 
> failed to be scheduled due to the above mentioned shortcomings in the current 
> method. Also the current method does not consider failures of workers. When 
> executors of a topology gets unassigned and needs to be scheduled again, the 
> current logic in getBestClustering may be inadequate if not complete wrong. 
> When executors needs to rescheduled due to a fault, getBestClustering will 
> likely return a cluster that is different from where the majority of 
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I 
> have come up with a ordering strategy I dub subordinate resource availability 
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the 
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM 
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU 
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and 
> can potentially schedule the most executors, while rack-2 is the worst rack 
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even 
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for 
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values 
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] 
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective 
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] 
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] 
> effective resources: 0.1951219512195122
> {code}
> The effective resource of a rack, which is also the subordinate resource, is 
> computed by: 
> {code}
> MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
> {code}
> Then we order the racks by the effective resource.
> Thus for our example:
> {code}
> Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
> {code}
> Also to deal with the presence of failures, if a topology is partially 
> scheduled, we find the rack with the most scheduled 

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420345#comment-15420345
 ] 

ASF GitHub Bot commented on STORM-1913:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1620


> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420343#comment-15420343
 ] 

ASF GitHub Bot commented on STORM-1913:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1620
  
+1


> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2037) debug operation should be whitelisted in SimpleAclAuthorizer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420201#comment-15420201
 ] 

ASF GitHub Bot commented on STORM-2037:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1624
  
merged into master and 1.x-branch


> debug operation should be whitelisted in SimpleAclAuthorizer
> 
>
> Key: STORM-2037
> URL: https://issues.apache.org/jira/browse/STORM-2037
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.x
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> For topology event logging to work in secure mode, the "debug" operation 
> should be whitelisted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2037) debug operation should be whitelisted in SimpleAclAuthorizer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420200#comment-15420200
 ] 

ASF GitHub Bot commented on STORM-2037:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1624


> debug operation should be whitelisted in SimpleAclAuthorizer
> 
>
> Key: STORM-2037
> URL: https://issues.apache.org/jira/browse/STORM-2037
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.x
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> For topology event logging to work in secure mode, the "debug" operation 
> should be whitelisted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418969#comment-15418969
 ] 

ASF GitHub Bot commented on STORM-1038:
---

GitHub user hsun-cnnxty reopened a pull request:

https://github.com/apache/storm/pull/1591

STORM-1038: Upgrade netty to 4.x in 1.x-branch

This is to add the feature to 1.x-branch.  The original PR for master 
branch is #728.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsun-cnnxty/storm 1.x-branch-netty4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1591


commit 57fbccc7e0d710ccaf04a5a828f0ff4cf29ec855
Author: Hang Sun 
Date:   2016-07-25T22:30:54Z

STORM-1038: Upgraded netty to 4.x

commit 144b6eddf314ce9a0b5beb8c99500c56ff0b8ec0
Author: Hang Sun 
Date:   2016-07-30T23:42:52Z

STORM-1038: removed unused imports




> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1038) Upgrade netty transport from 3.x to 4.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418968#comment-15418968
 ] 

ASF GitHub Bot commented on STORM-1038:
---

Github user hsun-cnnxty closed the pull request at:

https://github.com/apache/storm/pull/1591


> Upgrade netty transport from 3.x to 4.x
> ---
>
> Key: STORM-1038
> URL: https://issues.apache.org/jira/browse/STORM-1038
> Project: Apache Storm
>  Issue Type: Dependency upgrade
>  Components: storm-core
>Reporter: Hang Sun
>Priority: Minor
>  Labels: performance
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> It will be nice to upgrade netty to 4.x to take advantage of its more 
> efficient memory usage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418898#comment-15418898
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
@harshach @satishd @manuzhang 
I just fixed one thing from nimbus.clj: just log exception instead of 
crashing nimbus when `blob-rm-dependency-jars-in-topology` throws Exception. 
Before that nimbus test is failing intermittent.

You can just review nimbus.clj once again. Thanks in advance!


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418869#comment-15418869
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
OK, I created pull request against master: #1626


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418867#comment-15418867
 ] 

ASF GitHub Bot commented on STORM-2016:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1626

STORM-2016 Topology submission improvement: support adding local jars and 
maven artifacts on submission

PR for 1.x is here: #1608 

storm-submit is copied version of 1.x, but I have to reimplement (port to 
Java) some part of storm-core. So please have a look thoughtfully when 
reviewing files on storm-core.

I also fixed a bug from outside of pull request regarding symbolic link.

@harshach @satishd 
Please review this when you have some time. Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2016

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1626


commit 97615a4b5e9c3567ffd5b0642b60fc8e55e713ef
Author: Jungtaek Lim 
Date:   2016-08-12T13:58:08Z

STORM-2016 Topology submission improvement: support adding local jars and 
maven artifacts on submission

*NOTE* This contains a bugfix at creating symbolic link

* bin/storm now supports "--jars" and "--artifacts" options
** it's only effective with "storm jar" and "storm sql"
* introduce new module: storm-submit to help resolving dependencies with 
handling transitive dependencies
* StormSubmitter will upload dependencies to BlobStore when submitting 
topology
** StormSubmitter will remove jars blobs when topology submission fails
*** remove jar blobs when only one of AlreadyAliveException, 
InvalidTopologyException, AuthorizationException occurs
* Supervisor will download dependencies from BlobStore when such topology 
is assigned
* Supervisor will launch workers with adding downloaded dependencies to 
worker classpath
* Nimbus will remove jars from BlobStore when topology is killed
** don't remove artifacts from blobstore since it's shared across topologies
* add options to document (Command-line-client.md) and also storm.py
** it will be printed out when 'help jar' and 'help sql' is called




> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1234) port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418655#comment-15418655
 ] 

ASF GitHub Bot commented on STORM-1234:
---

GitHub user abhishekagarwal87 opened a pull request:

https://github.com/apache/storm/pull/1625

Port Test cases - STORM-1234, STORM-1240, STORM-1251, STORM-1256



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abhishekagarwal87/storm clj-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1625.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1625


commit 45bf3ef9f461770eaf342cdfb2d7b9c9a8963f1f
Author: Abhishek Agarwal 
Date:   2016-08-12T10:33:10Z

Port Test cases - STORM-1234, STORM-1240, STORM-1251, STORM-1256




> port  backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java
> 
>
> Key: STORM-1234
> URL: https://issues.apache.org/jira/browse/STORM-1234
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>  Labels: java-migration, jstorm-merger
>
> to junit test conversion



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1890) Employ cache-busting method to ensure newly deployed UI forces browsers to refetch scripts, templates, and CSS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418346#comment-15418346
 ] 

ASF GitHub Bot commented on STORM-1890:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1614
  
+1


> Employ cache-busting method to ensure newly deployed UI forces browsers to 
> refetch scripts, templates, and CSS
> --
>
> Key: STORM-1890
> URL: https://issues.apache.org/jira/browse/STORM-1890
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-ui
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Fix For: 2.0.0, 1.0.2, 1.1.0
>
>
> Currently we don't employ cache busting techniques in the Storm UI while 
> fetching script.js, CSS and templates. Ring is providing the Last-Modified 
> header, but browsers implement a heuristic to when they deem a resource stale 
> (https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2.4). This 
> means that as the Last-Modified for a resource is further away in the past, 
> the longer the browsers are going to wait until they refetch. It looks like 
> 10% padding is common, so if script.js was last modified 100 days ago, the 
> browser will not fetch it until 10 days after the time it was cached.
> An easy approach is to add a url parameter to allow for cache busting 
> whenever storm is packaged (mvn package). A more complicated method is 
> versioning the files (we'd need to specify them in the pom.xml individually 
> using the assembly plugin, unless we use some other plugin). The first method 
> is (was?) considered less effective, since some CDNs/browsers can decide not 
> to cache the query parameter.
> I'd like to go with the simpler method, unless there are strong opinions to 
> changing file names (this means we need to specify files in the assembly 
> pom.xml). Also, going this route we don't need any new plugins, and the 
> assembly build can just be changed to export a variable. We would modify 
> calls to include a value that changes on mvn package:
> {code}
>  
> {code}
> instead of:
> {code}
> 
> {code}
> Where $\{timestamp\} will be replaced at assembly time by maven. This would 
> be the time when the assembly build started.
> The templates will also have the extra parameter. I think providing this to 
> ajaxSetup will do the trick. For example:
> {code}
> $.ajaxSetup({ data: {"_ts" : "${timestamp}"}});
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418393#comment-15418393
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
Please hold off merging. I'll create pull request against master. We can 
merge both of them together.


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2006) Storm metrics feature improvement: support per-worker level metrics aggregation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418341#comment-15418341
 ] 

ASF GitHub Bot commented on STORM-2006:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1595
  
@HeartSaVioR I would like to see this merged into 1.x-branch. Given 
suggestions here can you make the changes , introducing IAggregatedMetrics 
interface and leave the existing the interface as it is.


> Storm metrics feature improvement: support per-worker level metrics 
> aggregation
> ---
>
> Key: STORM-2006
> URL: https://issues.apache.org/jira/browse/STORM-2006
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Affects Versions: 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> Storm provides per-task level metrics which could be huge when topology has a 
> number of tasks. 
> Task level metric is useful for determining load balance between tasks, but 
> it doesn't need to be time-series fashion.
> Before introducing topology level component like TopologyMaster for JStorm, 
> we can utilize SystemBolt to aggregate task level metrics to per-worker level 
> metrics.
> We should provide options and this feature should be turned off by default to 
> keep backward compatibility. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418334#comment-15418334
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1608
  
+1


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418330#comment-15418330
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1608#discussion_r74539478
  
--- Diff: bin/storm.py ---
@@ -232,44 +271,91 @@ def jar(jarfile, klass, *args):
 The process is configured so that StormSubmitter
 
(http://storm.apache.org/releases/current/javadocs/org/apache/storm/StormSubmitter.html)
 will upload the jar at topology-jar-path when the topology is 
submitted.
+
+When you want to ship other jars which is not included to application 
jar, you can pass them to --jars option with comma-separated string.
+For example, --jars "your-local-jar.jar,your-local-jar2.jar" will load 
your-local-jar.jar and your-local-jar2.jar.
+And when you want to ship maven artifacts and its transitive 
dependencies, you can pass them to --artifacts with comma-separated string.
+You can also exclude some dependencies like what you're doing in maven 
pom.
+Please add exclusion artifacts with '^' separated string after the 
artifact.
+For example, --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"
 will load jedis and kafka artifact and all of transitive dependencies but 
exclude slf4j-log4j12 from kafka.
+
+Complete example of both options is here: `./bin/storm jar 
example/storm-starter/storm-starter-topologies-*.jar 
org.apache.storm.starter.RollingTopWords blobstore-remote2 remote --jars 
"./external/storm-redis/storm-redis-1.1.0.jar,./external/storm-kafka/storm-kafka-1.1.0.jar"
 --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"`
+
+When you pass jars and/or artifacts options, StormSubmitter will 
upload them when the topology is submitted, and they will be included to 
classpath of both the process which runs the class, and also workers for that 
topology.
 """
+global DEP_JARS_OPTS, DEP_ARTIFACTS_OPTS
+
+local_jars = DEP_JARS_OPTS
+artifact_to_file_jars = resolve_dependencies(DEP_ARTIFACTS_OPTS)
+
 transform_class = confvalue("client.jartransformer.class", 
[CLUSTER_CONF_DIR])
 if (transform_class != None and transform_class != "nil"):
 tmpjar = os.path.join(tempfile.gettempdir(), 
uuid.uuid1().hex+".jar")
 
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner", 
args=[transform_class, jarfile, tmpjar], fork=True, daemon=False)
+extra_jars = [tmpjar, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 topology_runner_exit_code = exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[tmpjar, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
 fork=True,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar] +
+["-Dstorm.dependency.jars=" + 
",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 os.remove(tmpjar)
 sys.exit(topology_runner_exit_code)
 else:
+extra_jars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile] +
+["-Dstorm.dependency.jars=" + ",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 
 def sql(sql_file, topology_name):
 """Syntax: [storm sql sql-file topology-name]
 
 Compiles the SQL statements into a Trident topology and submits it to 
Storm.
+
+--jars and --artifacts options available for jar are also applied to 
sql command.
+Please refer "help jar" to see how to use --jars and --artifacts 
options.
+You normally want to pass these options since you need to set data 
source to your sql which is an external storage in many cases.
 """
+global DEP_JARS_OPTS, 

[jira] [Commented] (STORM-2023) Add calcite-core to dependency of storm-sql-runtime

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418329#comment-15418329
 ] 

ASF GitHub Bot commented on STORM-2023:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1610


> Add calcite-core to dependency of storm-sql-runtime
> ---
>
> Key: STORM-2023
> URL: https://issues.apache.org/jira/browse/STORM-2023
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-sql
>Affects Versions: 1.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> storm-sql provides both "storm-sql-core" and "storm-sql-runtime". Former is 
> for compiling sql to trident topology, and latter is for running compiled 
> trident topology.
> While testing storm-sql feature I found compiled class codes refers calcite 
> so calcite-core is needed at runtime. 
> I'm not sure we can make calcite-core get out of storm-sql-runtime, so for 
> now we can add calcite-core to storm-sql-runtime so that it can be available 
> on transitive dependencies for storm-sql-runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2037) debug operation should be whitelisted in SimpleAclAuthorizer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418313#comment-15418313
 ] 

ASF GitHub Bot commented on STORM-2037:
---

GitHub user arunmahadevan opened a pull request:

https://github.com/apache/storm/pull/1624

[STORM-2037] debug operation should be whitelisted in SimpleAclAuthorizer

While enabling/disabling topology event logging, theres a call to 
assert-authorized-user with "debug" as the operation in ui/core.clj.
This needs to be whitelisted in SimpleAclAuthorizer to get it working in 
secure mode.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arunmahadevan/storm STORM-2037

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 89a6985aef30dd3de75f2ca8de3d638ff22a88cc
Author: Arun Mahadevan 
Date:   2016-08-12T03:37:08Z

[STORM-2037] debug operation should be whitelisted in SimpleAclAuthorizer

While enabling/disabling topology event logging, theres a call to 
assert-authorized-user with "debug" as the operation in ui/core.clj.
This needs to be whitelisted in SimpleAclAuthorizer to get it working in 
secure mode.




> debug operation should be whitelisted in SimpleAclAuthorizer
> 
>
> Key: STORM-2037
> URL: https://issues.apache.org/jira/browse/STORM-2037
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.x
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>
> For topology event logging to work in secure mode, the "debug" operation 
> should be whitelisted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-841) Thread-safeness of OutputCollector has documented contrary to two official doc.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418266#comment-15418266
 ] 

ASF GitHub Bot commented on STORM-841:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1623


> Thread-safeness of OutputCollector has documented contrary to two official 
> doc.
> ---
>
> Key: STORM-841
> URL: https://issues.apache.org/jira/browse/STORM-841
> Project: Apache Storm
>  Issue Type: Documentation
>  Components: documentation
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> There're some issues with documentation.
> http://storm.apache.org/documentation/Concepts.html says
> {quote}
> Its perfectly fine to launch new threads in bolts that do processing 
> asynchronously. OutputCollector is thread-safe and can be called at any time.
> {quote}
> and http://storm.apache.org/documentation/Troubleshooting.html says
> {quote}
> This is caused by having multiple threads issue methods on the 
> OutputCollector. All emits, acks, and fails must happen on the same thread. 
> One subtle way this can happen is if you make a IBasicBolt that emits on a 
> separate thread. IBasicBolt's automatically ack after execute is called, so 
> this would cause multiple threads to use the OutputCollector leading to this 
> exception. When using a basic bolt, all emits must happen in the same thread 
> that runs execute.
> {quote}
> It is a contradiction, and at least for now OutputCollector is not 
> thread-safe.
> https://www.mail-archive.com/dev@storm.incubator.apache.org/msg00939.html
> Since newbie of Storm users may think Concepts page as "should read and keep 
> it mind", it is some kind of critical that that such important documentation 
> page has wrong content.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1999) Update docs with OutputCollector's threadsafety

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418242#comment-15418242
 ] 

ASF GitHub Bot commented on STORM-1999:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1622


> Update docs with OutputCollector's threadsafety
> ---
>
> Key: STORM-1999
> URL: https://issues.apache.org/jira/browse/STORM-1999
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Satish Duggana
>Assignee: Jungtaek Lim
>
> From 1.x branch, Updated docs about OutputCollector;s threadsafety.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418239#comment-15418239
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1618


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1999) Update docs with OutputCollector's threadsafety

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418237#comment-15418237
 ] 

ASF GitHub Bot commented on STORM-1999:
---

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1622
  
+1


> Update docs with OutputCollector's threadsafety
> ---
>
> Key: STORM-1999
> URL: https://issues.apache.org/jira/browse/STORM-1999
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Satish Duggana
>Assignee: Jungtaek Lim
>
> From 1.x branch, Updated docs about OutputCollector;s threadsafety.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-841) Thread-safeness of OutputCollector has documented contrary to two official doc.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418236#comment-15418236
 ] 

ASF GitHub Bot commented on STORM-841:
--

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1623
  
+1


> Thread-safeness of OutputCollector has documented contrary to two official 
> doc.
> ---
>
> Key: STORM-841
> URL: https://issues.apache.org/jira/browse/STORM-841
> Project: Apache Storm
>  Issue Type: Documentation
>  Components: documentation
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> There're some issues with documentation.
> http://storm.apache.org/documentation/Concepts.html says
> {quote}
> Its perfectly fine to launch new threads in bolts that do processing 
> asynchronously. OutputCollector is thread-safe and can be called at any time.
> {quote}
> and http://storm.apache.org/documentation/Troubleshooting.html says
> {quote}
> This is caused by having multiple threads issue methods on the 
> OutputCollector. All emits, acks, and fails must happen on the same thread. 
> One subtle way this can happen is if you make a IBasicBolt that emits on a 
> separate thread. IBasicBolt's automatically ack after execute is called, so 
> this would cause multiple threads to use the OutputCollector leading to this 
> exception. When using a basic bolt, all emits must happen in the same thread 
> that runs execute.
> {quote}
> It is a contradiction, and at least for now OutputCollector is not 
> thread-safe.
> https://www.mail-archive.com/dev@storm.incubator.apache.org/msg00939.html
> Since newbie of Storm users may think Concepts page as "should read and keep 
> it mind", it is some kind of critical that that such important documentation 
> page has wrong content.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418230#comment-15418230
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1618
  
Note: I'll cherry pick 4f0f79b since it's only effective commit.


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1766) A better algorithm server rack selection for RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418155#comment-15418155
 ] 

ASF GitHub Bot commented on STORM-1766:
---

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1621#discussion_r74524961
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -103,6 +103,9 @@ public Cluster(Cluster src) {
 this.status.putAll(src.status);
 this.topologyResources.putAll(src.topologyResources);
 this.blackListedHosts.addAll(src.blackListedHosts);
+if (src.networkTopography != null) {
--- End diff --

is this supposed to be == null. why are we creating new Map if there is one 
already.


> A better algorithm server rack selection for RAS
> 
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
> Fix For: 2.0.0
>
>
> Currently the getBestClustering algorithm for RAS finds the "Best" 
> cluster/rack based on which rack has the most available resources this may be 
> insufficient and may cause topologies not to be able to be scheduled 
> successfully even though there are enough resources to schedule it in the 
> cluster. We attempt to find the rack with the most resources by find the rack 
> with the biggest sum of available memory + available cpu. This method is not 
> effective since it does not consider the number of slots available. This 
> method also fails in identifying racks that are not schedulable due to the 
> exhaustion of one of the resources either memory, cpu, or slots. The current 
> implementation also tries the initial scheduling on one rack and not try to 
> schedule on all the racks before giving up which may cause topologies to be 
> failed to be scheduled due to the above mentioned shortcomings in the current 
> method. Also the current method does not consider failures of workers. When 
> executors of a topology gets unassigned and needs to be scheduled again, the 
> current logic in getBestClustering may be inadequate if not complete wrong. 
> When executors needs to rescheduled due to a fault, getBestClustering will 
> likely return a cluster that is different from where the majority of 
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I 
> have come up with a ordering strategy I dub subordinate resource availability 
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the 
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM 
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU 
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and 
> can potentially schedule the most executors, while rack-2 is the worst rack 
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even 
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for 
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values 
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] 
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective 
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] 
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] 
> effective 

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418148#comment-15418148
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1618
  
+1 Great finding.


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-841) Thread-safeness of OutputCollector has documented contrary to two official doc.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418135#comment-15418135
 ] 

ASF GitHub Bot commented on STORM-841:
--

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1623

STORM-841 Thread-safeness of OutputCollector has documented contrary to two 
official doc (0.10.x)

* Fix for 0.10.x

This should be also downmerged to 0.9.x branch, and applied to docs of 
0.10.0, 0.10.1, 0.9.6.

NOTE: This is non-code change so lazy consensus is applied. (don't need to 
wait for +1 and also 24 hrs)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-841-0.10.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1623.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1623


commit baacc0c032f4086c902005c5684bb9093b11d1ff
Author: Jungtaek Lim 
Date:   2016-08-11T23:41:22Z

STORM-841 Thread-safeness of OutputCollector has documented contrary to two 
official doc.

* Fix for 0.10.x




> Thread-safeness of OutputCollector has documented contrary to two official 
> doc.
> ---
>
> Key: STORM-841
> URL: https://issues.apache.org/jira/browse/STORM-841
> Project: Apache Storm
>  Issue Type: Documentation
>  Components: documentation
>Affects Versions: 0.10.0, 0.9.6
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> There're some issues with documentation.
> http://storm.apache.org/documentation/Concepts.html says
> {quote}
> Its perfectly fine to launch new threads in bolts that do processing 
> asynchronously. OutputCollector is thread-safe and can be called at any time.
> {quote}
> and http://storm.apache.org/documentation/Troubleshooting.html says
> {quote}
> This is caused by having multiple threads issue methods on the 
> OutputCollector. All emits, acks, and fails must happen on the same thread. 
> One subtle way this can happen is if you make a IBasicBolt that emits on a 
> separate thread. IBasicBolt's automatically ack after execute is called, so 
> this would cause multiple threads to use the OutputCollector leading to this 
> exception. When using a basic bolt, all emits must happen in the same thread 
> that runs execute.
> {quote}
> It is a contradiction, and at least for now OutputCollector is not 
> thread-safe.
> https://www.mail-archive.com/dev@storm.incubator.apache.org/msg00939.html
> Since newbie of Storm users may think Concepts page as "should read and keep 
> it mind", it is some kind of critical that that such important documentation 
> page has wrong content.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1999) Update docs with OutputCollector's threadsafety

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418119#comment-15418119
 ] 

ASF GitHub Bot commented on STORM-1999:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1622

STORM-1999 Update docs with OutputCollector's threadsafety

* From Storm 1.0.0, OutputCollector is thread safe

This should be also downmerged to 1.x and 1.0.x branches, and applied to 
docs of 1.0.0, 1.0.1, 1.0.2, 2.0.0-SNAPSHOT.

NOTE: This is non-code change so lazy consensus is applied. (don't need to 
wait for +1 and also 24 hrs)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1999

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1622


commit 3e4d9444c4a637538bf66d36f99c8f36cd50edf5
Author: Jungtaek Lim 
Date:   2016-08-11T23:27:39Z

STORM-1999 Update docs with OutputCollector's threadsafety

* From Storm 1.0.0, OutputCollector is thread safe




> Update docs with OutputCollector's threadsafety
> ---
>
> Key: STORM-1999
> URL: https://issues.apache.org/jira/browse/STORM-1999
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Satish Duggana
>
> From 1.x branch, Updated docs about OutputCollector;s threadsafety.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417766#comment-15417766
 ] 

ASF GitHub Bot commented on STORM-1913:
---

GitHub user knusbaum reopened a pull request:

https://github.com/apache/storm/pull/1620

STORM-1913: Additions and Improvements for Trident RAS API

This is just a backport of STORM-1913 (#1500) to the 1.x branch.
I'm not sure this actually needs a PR, but since it's been a while since 
#1500 was merged, I'll put one up anyway.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm 1.x-STORM-1913

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1620


commit c5cf73b9078176482d273907c79ca8e9a1e85498
Author: Kyle Nusbaum 
Date:   2016-06-17T17:59:06Z

Improvements for Trident RAS API.




> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417765#comment-15417765
 ] 

ASF GitHub Bot commented on STORM-1913:
---

Github user knusbaum closed the pull request at:

https://github.com/apache/storm/pull/1620


> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1766) A better algorithm server rack selection for RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417728#comment-15417728
 ] 

ASF GitHub Bot commented on STORM-1766:
---

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1621
  
+1



> A better algorithm server rack selection for RAS
> 
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
> Fix For: 2.0.0
>
>
> Currently the getBestClustering algorithm for RAS finds the "Best" 
> cluster/rack based on which rack has the most available resources this may be 
> insufficient and may cause topologies not to be able to be scheduled 
> successfully even though there are enough resources to schedule it in the 
> cluster. We attempt to find the rack with the most resources by find the rack 
> with the biggest sum of available memory + available cpu. This method is not 
> effective since it does not consider the number of slots available. This 
> method also fails in identifying racks that are not schedulable due to the 
> exhaustion of one of the resources either memory, cpu, or slots. The current 
> implementation also tries the initial scheduling on one rack and not try to 
> schedule on all the racks before giving up which may cause topologies to be 
> failed to be scheduled due to the above mentioned shortcomings in the current 
> method. Also the current method does not consider failures of workers. When 
> executors of a topology gets unassigned and needs to be scheduled again, the 
> current logic in getBestClustering may be inadequate if not complete wrong. 
> When executors needs to rescheduled due to a fault, getBestClustering will 
> likely return a cluster that is different from where the majority of 
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I 
> have come up with a ordering strategy I dub subordinate resource availability 
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the 
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM 
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU 
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and 
> can potentially schedule the most executors, while rack-2 is the worst rack 
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even 
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for 
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values 
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] 
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% Slots 20.0% ] effective resources: 
> 0.0
> rack-4 Avail [ CPU 50.0% MEM 2.4390243902439024% Slots 20.0% ] effective 
> resources: 0.024390243902439025
> rack-1 Avail [ CPU 16.39344262295082% MEM 9.75609756097561% Slots 20.0% ] 
> effective resources: 0.0975609756097561
> rack-0 Avail [ CPU 32.78688524590164% MEM 19.51219512195122% Slots 20.0% ] 
> effective resources: 0.1951219512195122
> {code}
> The effective resource of a rack, which is also the subordinate resource, is 
> computed by: 
> {code}
> MIN(resource availability percentage of {CPU, Memory, # of free Slots}).
> {code}
> Then we order the racks by the effective resource.
> Thus for our example:
> {code}
> Sorted rack: [rack-0, rack-1, rack-4, rack-3, rack-2]
> {code}
> Also to deal with the presence of failures, if a topology is partially 
> scheduled, we find the rack with the most scheduled 

[jira] [Commented] (STORM-2036) Fix minor bug in RAS Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417529#comment-15417529
 ] 

ASF GitHub Bot commented on STORM-2036:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1207


> Fix minor bug in RAS Tests
> --
>
> Key: STORM-2036
> URL: https://issues.apache.org/jira/browse/STORM-2036
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Trivial
> Fix For: 1.0.0, 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1766) A better algorithm server rack selection for RAS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417494#comment-15417494
 ] 

ASF GitHub Bot commented on STORM-1766:
---

GitHub user jerrypeng opened a pull request:

https://github.com/apache/storm/pull/1621

[STORM-1766] - A better algorithm server rack selection for RAS

Backport of #1398 to 1.x branch.  I'm not sure this actually needs a PR, 
but since it's been a while since #1500 was merged, I'll put one anyways since 
the code went into 2.x a while ago

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerrypeng/storm 1.x-STORM-1766

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1621.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1621


commit d50a2437923f2919fbee130b8e9e86a62a2d9f48
Author: Boyang Jerry Peng 
Date:   2016-05-04T22:08:57Z

[STORM-1766] - A better algorithm server rack selection for RAS




> A better algorithm server rack selection for RAS
> 
>
> Key: STORM-1766
> URL: https://issues.apache.org/jira/browse/STORM-1766
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
> Fix For: 2.0.0
>
>
> Currently the getBestClustering algorithm for RAS finds the "Best" 
> cluster/rack based on which rack has the most available resources this may be 
> insufficient and may cause topologies not to be able to be scheduled 
> successfully even though there are enough resources to schedule it in the 
> cluster. We attempt to find the rack with the most resources by find the rack 
> with the biggest sum of available memory + available cpu. This method is not 
> effective since it does not consider the number of slots available. This 
> method also fails in identifying racks that are not schedulable due to the 
> exhaustion of one of the resources either memory, cpu, or slots. The current 
> implementation also tries the initial scheduling on one rack and not try to 
> schedule on all the racks before giving up which may cause topologies to be 
> failed to be scheduled due to the above mentioned shortcomings in the current 
> method. Also the current method does not consider failures of workers. When 
> executors of a topology gets unassigned and needs to be scheduled again, the 
> current logic in getBestClustering may be inadequate if not complete wrong. 
> When executors needs to rescheduled due to a fault, getBestClustering will 
> likely return a cluster that is different from where the majority of 
> executors from the topology is originally scheduling in.
> Thus, I propose a different strategy/algorithm to find the "best" cluster. I 
> have come up with a ordering strategy I dub subordinate resource availability 
> ordering (inspired by Dominant Resource Fairness) that sorts racks by the 
> subordinate (not dominant) resource availability.
> For example given 4 racks with the following resource availabilities
> {code}
> //generate some that has alot of memory but little of cpu
> rack-3 Avail [ CPU 100.0 MEM 20.0 Slots 40 ] Total [ CPU 100.0 MEM 
> 20.0 Slots 40 ]
> //generate some supervisors that are depleted of one resource
> rack-2 Avail [ CPU 0.0 MEM 8.0 Slots 40 ] Total [ CPU 0.0 MEM 8.0 
> Slots 40 ]
> //generate some that has a lot of cpu but little of memory
> rack-4 Avail [ CPU 6100.0 MEM 1.0 Slots 40 ] Total [ CPU 6100.0 MEM 
> 1.0 Slots 40 ]
> //generate another rack of supervisors with less resources than rack-0
> rack-1 Avail [ CPU 2000.0 MEM 4.0 Slots 40 ] Total [ CPU 2000.0 MEM 
> 4.0 Slots 40 ]
> rack-0 Avail [ CPU 4000.0 MEM 8.0 Slots 40( ] Total [ CPU 4000.0 MEM 
> 8.0 Slots 40 ]
> Cluster Overall Avail [ CPU 12200.0 MEM 41.0 Slots 200 ] Total [ CPU 
> 12200.0 MEM 41.0 Slots 200 ]
> {code}
> It is clear that rack-0 is the best cluster since its the most balanced and 
> can potentially schedule the most executors, while rack-2 is the worst rack 
> since rack-2 is depleted of cpu resource thus rendering it unschedulable even 
> though there are other resources available.
> We first calculate the resource availability percentage of all the racks for 
> each resource by computing:
> {code}
> (resource available on rack) / (resource available in cluster)
> {code}
> We do this calculation to normalize the values otherwise the resource values 
> would not be comparable.
> So for our example:
> {code}
> rack-3 Avail [ CPU 0.819672131147541% MEM 48.78048780487805% Slots 20.0% ] 
> effective resources: 0.00819672131147541
> rack-2 Avail [ 0.0% MEM 19.51219512195122% 

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417416#comment-15417416
 ] 

ASF GitHub Bot commented on STORM-1913:
---

Github user jerrypeng commented on the issue:

https://github.com/apache/storm/pull/1620
  
+1


> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1913) Additions and Improvements for Trident RAS API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417413#comment-15417413
 ] 

ASF GitHub Bot commented on STORM-1913:
---

GitHub user knusbaum opened a pull request:

https://github.com/apache/storm/pull/1620

STORM-1913: Additions and Improvements for Trident RAS API

This is just a backport of STORM-1913 (#1500) to the 1.x branch.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/incubator-storm 1.x-STORM-1913

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1620.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1620


commit c5cf73b9078176482d273907c79ca8e9a1e85498
Author: Kyle Nusbaum 
Date:   2016-06-17T17:59:06Z

Improvements for Trident RAS API.




> Additions and Improvements for Trident RAS API
> --
>
> Key: STORM-1913
> URL: https://issues.apache.org/jira/browse/STORM-1913
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
> Fix For: 2.0.0
>
>
> Trident's RAS API does not honor the following config values:
> {code}
> topology.component.resources.onheap.memory.mb
> topology.component.resources.offheap.memory.mb
> topology.component.cpu.pcore.percent
> {code}
> Trident does not receive the user's config as part of its builder API, so it 
> does not know the value of these. Instead of altering the existing API (we 
> want to remain backwards-compatible), add some new methods for dealing with 
> this.
> There is also currently no way to set the master coord spouts' resources. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417396#comment-15417396
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR, thanks for taking a look and running locally. As far as the 
supervisor config, I added: https://issues.apache.org/jira/browse/STORM-2035. 
Let me know if that's what you had in mind.

As far as a patch for 1.x, I'll add a new PR. The stats code is in Java in 
2.x, but clojure in 1.x. 

Thanks @knusbaum, and @jerrypeng for jumping in as well. 


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417326#comment-15417326
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user d2r commented on the issue:

https://github.com/apache/storm/pull/1618
  
OK, no problem. Changed it to debug.


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416534#comment-15416534
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1618#discussion_r74367640
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/messaging/netty/StormClientHandler.java ---
@@ -60,7 +60,6 @@ public void messageReceived(ChannelHandlerContext ctx, 
MessageEvent event) {
 //This should be the metrics, and there should only be one 
of them
 List list = (List)message;
 if (list.size() < 1) throw new RuntimeException("Didn't 
see enough load metrics ("+client.getDstAddress()+") "+list);
-if (list.size() != 1) LOG.warn("Messages are not being 
delivered fast enough, got "+list.size()+" metrics messages at 
once("+client.getDstAddress()+")");
--- End diff --

I'm also OK for this approach. If we pick this, comment doesn't need to be 
modified.


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416527#comment-15416527
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1592
  
@jerrypeng Please let me know the issue numbers for these things. I'd also 
try to get some time to backport them.
@knusbaum Agreed. I recently initiated discussion to reduce the scope for 
porting (let clojure tests out of phase 1) and it might be not enough.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416523#comment-15416523
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1608#discussion_r74367408
  
--- Diff: bin/storm.py ---
@@ -232,44 +271,91 @@ def jar(jarfile, klass, *args):
 The process is configured so that StormSubmitter
 
(http://storm.apache.org/releases/current/javadocs/org/apache/storm/StormSubmitter.html)
 will upload the jar at topology-jar-path when the topology is 
submitted.
+
+When you want to ship other jars which is not included to application 
jar, you can pass them to --jars option with comma-separated string.
+For example, --jars "your-local-jar.jar,your-local-jar2.jar" will load 
your-local-jar.jar and your-local-jar2.jar.
+And when you want to ship maven artifacts and its transitive 
dependencies, you can pass them to --artifacts with comma-separated string.
+You can also exclude some dependencies like what you're doing in maven 
pom.
+Please add exclusion artifacts with '^' separated string after the 
artifact.
+For example, --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"
 will load jedis and kafka artifact and all of transitive dependencies but 
exclude slf4j-log4j12 from kafka.
+
+Complete example of both options is here: `./bin/storm jar 
example/storm-starter/storm-starter-topologies-*.jar 
org.apache.storm.starter.RollingTopWords blobstore-remote2 remote --jars 
"./external/storm-redis/storm-redis-1.1.0.jar,./external/storm-kafka/storm-kafka-1.1.0.jar"
 --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"`
+
+When you pass jars and/or artifacts options, StormSubmitter will 
upload them when the topology is submitted, and they will be included to 
classpath of both the process which runs the class, and also workers for that 
topology.
 """
+global DEP_JARS_OPTS, DEP_ARTIFACTS_OPTS
+
+local_jars = DEP_JARS_OPTS
+artifact_to_file_jars = resolve_dependencies(DEP_ARTIFACTS_OPTS)
+
 transform_class = confvalue("client.jartransformer.class", 
[CLUSTER_CONF_DIR])
 if (transform_class != None and transform_class != "nil"):
 tmpjar = os.path.join(tempfile.gettempdir(), 
uuid.uuid1().hex+".jar")
 
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner", 
args=[transform_class, jarfile, tmpjar], fork=True, daemon=False)
+extra_jars = [tmpjar, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 topology_runner_exit_code = exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[tmpjar, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
 fork=True,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar] +
+["-Dstorm.dependency.jars=" + 
",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 os.remove(tmpjar)
 sys.exit(topology_runner_exit_code)
 else:
+extra_jars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile] +
+["-Dstorm.dependency.jars=" + ",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 
 def sql(sql_file, topology_name):
 """Syntax: [storm sql sql-file topology-name]
 
 Compiles the SQL statements into a Trident topology and submits it to 
Storm.
+
+--jars and --artifacts options available for jar are also applied to 
sql command.
+Please refer "help jar" to see how to use --jars and --artifacts 
options.
+You normally want to pass these options since you need to set data 
source to your sql which is an external storage in many cases.
 """
+global DEP_JARS_OPTS, 

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416508#comment-15416508
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR 
Good to hear it. I don't know if we want to stop feature development on 1.x 
since that's our release branch, and 2.x seems to have stalled. The policy 
behind all of this is probably better discussed on the mailing lists.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416500#comment-15416500
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1592
  
@knusbaum @jerrypeng 
OK. I was just not sure that we are seeing same target version for the 
improvements: 1.x vs 2.x.
If we all think we would want to ship our effort to 1.x, we must not block 
this due to port especially given that we all don't know when the port will be 
finished.
(We must deal with this ASAP since it makes the contribution getting 
harder.)

As I said I didn't review this, but I'm +1 for this feature.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416493#comment-15416493
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user jerrypeng commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR 

I have pushed critical bug fixes for RAS to the 1.x branch.  There has been 
a few algorithmic improvements on RAS that I had only put into 2.x but I desire 
to get them into 1.x as well when I have time.  

I have found this feature to be extremely useful regardless of whether RAS 
is being used.  It tells you what components are scheduled in each worker and 
tells you the uptime of each worker so you can monitor the stability of your 
topology.

We have no clear timeline for when the translation of nimbus will be done, 
and I am not sure its good to wait for an uncertain amount of time.

Thus I am +1 for this feature


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416490#comment-15416490
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR, the patch isn't directed for RAS specifically. 

The primary benefit for either non-RAS or RAS is summarizing all workers 
for a topology in one table, and all workers for a supervisor in one table 
(with the new supervisor page). In each of these tables, each worker shows the 
components that are running (with the # of executors per component), uptime per 
worker, and resources assigned.

The supervisor page is useful because you can go and see what other 
topologies are running on machine X, and what components they are running. I 
think that view is unique.



> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416489#comment-15416489
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1608#discussion_r74366031
  
--- Diff: bin/storm.py ---
@@ -754,18 +827,24 @@ def parse_config_opts(args):
 elif token == "--config":
   global CONFFILE
   CONFFILE = curr.pop()
+elif token == "--jars":
+  jars_list.extend(curr.pop().split(','))
+elif token == "--artifacts":
--- End diff --

whats the intention behind artifacts?


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416487#comment-15416487
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1608#discussion_r74365965
  
--- Diff: bin/storm.py ---
@@ -232,44 +271,91 @@ def jar(jarfile, klass, *args):
 The process is configured so that StormSubmitter
 
(http://storm.apache.org/releases/current/javadocs/org/apache/storm/StormSubmitter.html)
 will upload the jar at topology-jar-path when the topology is 
submitted.
+
+When you want to ship other jars which is not included to application 
jar, you can pass them to --jars option with comma-separated string.
+For example, --jars "your-local-jar.jar,your-local-jar2.jar" will load 
your-local-jar.jar and your-local-jar2.jar.
+And when you want to ship maven artifacts and its transitive 
dependencies, you can pass them to --artifacts with comma-separated string.
+You can also exclude some dependencies like what you're doing in maven 
pom.
+Please add exclusion artifacts with '^' separated string after the 
artifact.
+For example, --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"
 will load jedis and kafka artifact and all of transitive dependencies but 
exclude slf4j-log4j12 from kafka.
+
+Complete example of both options is here: `./bin/storm jar 
example/storm-starter/storm-starter-topologies-*.jar 
org.apache.storm.starter.RollingTopWords blobstore-remote2 remote --jars 
"./external/storm-redis/storm-redis-1.1.0.jar,./external/storm-kafka/storm-kafka-1.1.0.jar"
 --artifacts 
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"`
+
+When you pass jars and/or artifacts options, StormSubmitter will 
upload them when the topology is submitted, and they will be included to 
classpath of both the process which runs the class, and also workers for that 
topology.
 """
+global DEP_JARS_OPTS, DEP_ARTIFACTS_OPTS
+
+local_jars = DEP_JARS_OPTS
+artifact_to_file_jars = resolve_dependencies(DEP_ARTIFACTS_OPTS)
+
 transform_class = confvalue("client.jartransformer.class", 
[CLUSTER_CONF_DIR])
 if (transform_class != None and transform_class != "nil"):
 tmpjar = os.path.join(tempfile.gettempdir(), 
uuid.uuid1().hex+".jar")
 
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner", 
args=[transform_class, jarfile, tmpjar], fork=True, daemon=False)
+extra_jars = [tmpjar, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 topology_runner_exit_code = exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[tmpjar, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
 fork=True,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar] +
+["-Dstorm.dependency.jars=" + 
",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 os.remove(tmpjar)
 sys.exit(topology_runner_exit_code)
 else:
+extra_jars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR]
+extra_jars.extend(local_jars)
+extra_jars.extend(artifact_to_file_jars.values())
 exec_storm_class(
 klass,
 jvmtype="-client",
-extrajars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR],
+extrajars=extra_jars,
 args=args,
 daemon=False,
-jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile])
+jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile] +
+["-Dstorm.dependency.jars=" + ",".join(local_jars)] +
+["-Dstorm.dependency.artifacts=" + 
json.dumps(artifact_to_file_jars)])
 
 def sql(sql_file, topology_name):
 """Syntax: [storm sql sql-file topology-name]
 
 Compiles the SQL statements into a Trident topology and submits it to 
Storm.
+
+--jars and --artifacts options available for jar are also applied to 
sql command.
+Please refer "help jar" to see how to use --jars and --artifacts 
options.
+You normally want to pass these options since you need to set data 
source to your sql which is an external storage in many cases.
 """
+global DEP_JARS_OPTS, 

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416471#comment-15416471
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1618
  
+1, would like minor nit to be addressed. 


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416470#comment-15416470
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR 
I'm +1 for the patch.

Nimbus has been in progress for porting for a long time. We can't 
(apparently) halt all progress until it's ported. You recently submitted a 
patch which only modified 11 lines. If many people submit patches that only 
modify 11 lines, it's no different. Things seem to have been pretty 
consistently merged regardless of the fact that they touch un-ported clojure 
code.

This is also a pretty desirable code change. It provides something that we 
really need to provide if we want the platform to be successful. It's already 
helped me personally solve three or four issues that would have been extremely 
irritating to solve otherwise.

If we had a time-frame for Nimbus being ported, I might say "let's wait on 
it," but we don't. This is a really good feature. I say, let's get it in.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416468#comment-15416468
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/1618#discussion_r74365279
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/messaging/netty/StormClientHandler.java ---
@@ -60,7 +60,6 @@ public void messageReceived(ChannelHandlerContext ctx, 
MessageEvent event) {
 //This should be the metrics, and there should only be one 
of them
 List list = (List)message;
 if (list.size() < 1) throw new RuntimeException("Didn't 
see enough load metrics ("+client.getDstAddress()+") "+list);
-if (list.size() != 1) LOG.warn("Messages are not being 
delivered fast enough, got "+list.size()+" metrics messages at 
once("+client.getDstAddress()+")");
--- End diff --

nit: Can we have a debug statement saying more than one task message 
received instead of the removed warning message? 


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416465#comment-15416465
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1592
  
@abellina 
First of all, freezing development was what we planned though it was broken 
now.

Let me clear on this,

- Recent patches for RAS are only applied to master (2.0.0)
  - I don't know this is for freezing development on 1.x, or just forget 
backporting to 1.x.
- At a glimpse, this patch seems to help for RAS users.
  - Could you explain what benefits this patch give to non RAS users?
- So for me this patch is more likely for 2.0.0, not 1.x. 
  - If this is also for 1.x, personally I'm OK to allow modification for 
nimbus.clj.
  - But if this is just for 2.0.0, porting nimbus is same milestone so why 
not porting it first before adding the workload for porting?

I'm also the one who works two times for same pull request because of 
porting, so I might be a bit keen on this.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416448#comment-15416448
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1592
  
@HeartSaVioR, Are we freezing development on nimbus until it is ported? I 
can wait and submit when that is ported if nimbus is close, else I am not sure 
if it is a show stopper. Maybe I misunderstand your point.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416389#comment-15416389
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1592
  
Hi @abellina,
this is a bit huge patch (even without thrift generated code) including 
front-end change, so seems not easy for me to have time to review.

Before reviewing, two considerations from me:

1. Since recent RAS patches only applied to master branch, I'm not sure we 
may want to apply RAS related improvement to 1.x as well. Would like to hear 
opinions on @jerrypeng.
2. This changes nimbus.clj which is not ported yet, and it seems not small 
- more than 300 lines changed only on nimbus.clj. I recently submitted a patch 
which modifies nimbus.clj but it only changes 11 lines.

Btw, you said @kishorvpatil and @knusbaum, and @d2r already reviewed the 
patch. Then why not review this again (and leave +1)? Reviewing internally is 
no effect for Apache side.


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416361#comment-15416361
 ] 

ASF GitHub Bot commented on STORM-2032:
---

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1618#discussion_r74361223
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/messaging/netty/StormClientHandler.java ---
@@ -60,7 +60,6 @@ public void messageReceived(ChannelHandlerContext ctx, 
MessageEvent event) {
 //This should be the metrics, and there should only be one 
of them
--- End diff --

If we want to remove warning message because this is not a part of design, 
how about modifying this comment as well?
Maybe like this : `This should be the metrics, and it always picks the last 
one when more than one metrics messages are available.`


> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1994) Add table with per-topology & worker resource usage and components in (new) supervisor and topology pages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415515#comment-15415515
 ] 

ASF GitHub Bot commented on STORM-1994:
---

Github user abellina commented on the issue:

https://github.com/apache/storm/pull/1592
  
Hi, I haven't gotten reviews on this ticket yet. There are a lot of 
generated files, so it is not as bad as it looks!


> Add table with per-topology & worker resource usage and components in (new) 
> supervisor and topology pages
> -
>
> Key: STORM-1994
> URL: https://issues.apache.org/jira/browse/STORM-1994
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core, storm-ui
>Reporter: Alessandro Bellina
>Assignee: Alessandro Bellina
>Priority: Minor
> Attachments: supervisor_page_worker_table.png, 
> topology_page_worker_table.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2032) "not fast enough" metrics WARN message in netty client can be misinterpreted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415443#comment-15415443
 ] 

ASF GitHub Bot commented on STORM-2032:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1618

[STORM-2032] removes warning in case more than one metrics tuple is received



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm 
storm-2032-remove-not-fast-enough-log

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1618


commit 45499c4c1259c668a7b03cebf7010f4766610cf3
Author: Derek Dagit 
Date:   2016-08-10T15:24:04Z

removes warning in case more than one metrics tuple is received




> "not fast enough" metrics WARN message in netty client can be misinterpreted
> 
>
> Key: STORM-2032
> URL: https://issues.apache.org/jira/browse/STORM-2032
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Minor
>
> Example:
> bq. WARN Messages are not being delivered fast enough, got 3 metrics messages 
> at once
> It appears to some users to be a good signal for monitoring/alerting, but 
> really this is not part of the design.
> We should remove it, change it, or lower the log level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2026) Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, bolt-transfer-fn)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415284#comment-15415284
 ] 

ASF GitHub Bot commented on STORM-2026:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1616
  
Thanks @satishd for reviewing and merging!


> Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, 
> bolt-transfer-fn)
> -
>
> Key: STORM-2026
> URL: https://issues.apache.org/jira/browse/STORM-2026
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
> Fix For: 2.0.0
>
>
> As I left the comment from 
> https://github.com/apache/storm/pull/1445#discussion_r73255197 for pull 
> request, there's some difference between SpoutExecutor / BoltExecutor and 
> spout-transfer-fn / bolt-transfer-fn.
> While it's not that big to fix, I just want to not block port work and just 
> address from here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415221#comment-15415221
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user manuzhang commented on the issue:

https://github.com/apache/storm/pull/1608
  
The exception is caused by the escaping "\" character previously. Things 
are working now after I updated to the latest version. 


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2026) Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, bolt-transfer-fn)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415149#comment-15415149
 ] 

ASF GitHub Bot commented on STORM-2026:
---

Github user satishd commented on the issue:

https://github.com/apache/storm/pull/1616
  
Thanks @HeartSaVioR 


> Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, 
> bolt-transfer-fn)
> -
>
> Key: STORM-2026
> URL: https://issues.apache.org/jira/browse/STORM-2026
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> As I left the comment from 
> https://github.com/apache/storm/pull/1445#discussion_r73255197 for pull 
> request, there's some difference between SpoutExecutor / BoltExecutor and 
> spout-transfer-fn / bolt-transfer-fn.
> While it's not that big to fix, I just want to not block port work and just 
> address from here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2026) Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, bolt-transfer-fn)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415148#comment-15415148
 ] 

ASF GitHub Bot commented on STORM-2026:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1616


> Inconsistency between (SpoutExecutor, BoltExecutor) and (spout-transfer-fn, 
> bolt-transfer-fn)
> -
>
> Key: STORM-2026
> URL: https://issues.apache.org/jira/browse/STORM-2026
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> As I left the comment from 
> https://github.com/apache/storm/pull/1445#discussion_r73255197 for pull 
> request, there's some difference between SpoutExecutor / BoltExecutor and 
> spout-transfer-fn / bolt-transfer-fn.
> While it's not that big to fix, I just want to not block port work and just 
> address from here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2016) Topology submission improvement: support adding local jars and maven artifacts on submission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414985#comment-15414985
 ] 

ASF GitHub Bot commented on STORM-2016:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1608
  
@manuzhang A bit confusing so would like to double check on this. Do you 
succeed to resolve 1.1.1-SNAPSHOT after `mvn clean install`? Or you're still 
failing?


> Topology submission improvement: support adding local jars and maven 
> artifacts on submission
> 
>
> Key: STORM-2016
> URL: https://issues.apache.org/jira/browse/STORM-2016
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-core
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> This JIRA tracks actual work on below proposal / design document.
> https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission
> Proposal discussion thread is here: 
> http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3ccaf5108i9+tjanz0lgrktmkvqel7f+53k9uyzxct6zhsu6oh...@mail.gmail.com%3E
> Let's post on discussion thread if we have any opinions / ideas on this 
> instead of leaving comments on this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-2023) Add calcite-core to dependency of storm-sql-runtime

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414643#comment-15414643
 ] 

ASF GitHub Bot commented on STORM-2023:
---

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1610
  
Build failure is not related to this.

```
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.2.1:process (default) 
on project storm-hive: Error resolving project artifact: Could not transfer 
artifact net.hydromatic:linq4j:pom:0.4 from/to sonatype-apache 
(https://repository.apache.org/releases/): Connect to repository.apache.org:443 
[repository.apache.org/207.244.88.143] failed: Connection timed out for project 
net.hydromatic:linq4j:jar:0.4 -> [Help 1]
```

I requested Travis support team to take a look at recent often time out for 
repository.apache.org. I'll update the news once I get one.


> Add calcite-core to dependency of storm-sql-runtime
> ---
>
> Key: STORM-2023
> URL: https://issues.apache.org/jira/browse/STORM-2023
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-sql
>Affects Versions: 1.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> storm-sql provides both "storm-sql-core" and "storm-sql-runtime". Former is 
> for compiling sql to trident topology, and latter is for running compiled 
> trident topology.
> While testing storm-sql feature I found compiled class codes refers calcite 
> so calcite-core is needed at runtime. 
> I'm not sure we can make calcite-core get out of storm-sql-runtime, so for 
> now we can add calcite-core to storm-sql-runtime so that it can be available 
> on transitive dependencies for storm-sql-runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)