share singleton between topology and multiple bolts
Hi All, My topology initializes a singleton containing a s set of static objects. These objects can be complex but are static. I want all the bolts to have access to this singleton. It seems that the bolts can access the primitives such as strings in this singleton, but cannot access more complex objects such as hashmaps built in the singleton initialized by the topology. Is anyone familiar with this issue? Am I doing something wrong? The topology runs in cluster mode. thanks Clay
[GitHub] storm pull request: STORM-581. Add rebalance params to Storm REST ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/370#issuecomment-71254799 +1 the code looks good. It needs to be upmerged, but other then that it looks fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-635. logviewer returns 404 if storm_home...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/391#issuecomment-71253695 +1 looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-635) logviewer returns 404 if storm_home/logs is a symlinked dir
[ https://issues.apache.org/jira/browse/STORM-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289826#comment-14289826 ] ASF GitHub Bot commented on STORM-635: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/391#issuecomment-71253695 +1 looks good to me. logviewer returns 404 if storm_home/logs is a symlinked dir --- Key: STORM-635 URL: https://issues.apache.org/jira/browse/STORM-635 Project: Apache Storm Issue Type: Bug Reporter: Sriharsha Chintalapani Assignee: Sriharsha Chintalapani -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-630) Support for Clojure 1.6.0
[ https://issues.apache.org/jira/browse/STORM-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289864#comment-14289864 ] ASF GitHub Bot commented on STORM-630: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/285#issuecomment-71256412 The change looks good to me and matches with STORM-630, so I am +1 and think I will pull both in together. Support for Clojure 1.6.0 - Key: STORM-630 URL: https://issues.apache.org/jira/browse/STORM-630 Project: Apache Storm Issue Type: Bug Reporter: Dave Kincaid Assignee: Kai Sasaki Trying to use Storm under Clojure 1.6.0 causes errors with namespace conflicts (specifically the function some?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-629] Place Link to Source Code Reposito...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/386#issuecomment-71263897 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-629) Place Link to Source Code Repository on Webpage
[ https://issues.apache.org/jira/browse/STORM-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289955#comment-14289955 ] ASF GitHub Bot commented on STORM-629: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/386#issuecomment-71263897 +1 Place Link to Source Code Repository on Webpage --- Key: STORM-629 URL: https://issues.apache.org/jira/browse/STORM-629 Project: Apache Storm Issue Type: Task Reporter: Henning Kropp Assignee: Kai Sasaki Priority: Minor Maybe it just me, but I was unable to find a link to the source code repository on the webpage. Good places for that could be: * http://storm.apache.org/downloads.html * http://storm.apache.org/documentation/Contributing-to-Storm.html * http://storm.apache.org/documentation/Setting-up-development-environment.html * http://storm.apache.org/about/free-and-open-source.html I assume it is this one git://git.apache.org/incubator-storm.git ?? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-629) Place Link to Source Code Repository on Webpage
[ https://issues.apache.org/jira/browse/STORM-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289981#comment-14289981 ] ASF GitHub Bot commented on STORM-629: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/386 Place Link to Source Code Repository on Webpage --- Key: STORM-629 URL: https://issues.apache.org/jira/browse/STORM-629 Project: Apache Storm Issue Type: Task Reporter: Henning Kropp Assignee: Kai Sasaki Priority: Minor Maybe it just me, but I was unable to find a link to the source code repository on the webpage. Good places for that could be: * http://storm.apache.org/downloads.html * http://storm.apache.org/documentation/Contributing-to-Storm.html * http://storm.apache.org/documentation/Setting-up-development-environment.html * http://storm.apache.org/about/free-and-open-source.html I assume it is this one git://git.apache.org/incubator-storm.git ?? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-609] Add storm-redis to storm external
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/365#issuecomment-71273044 +1 the changes still look good Are there any other committers that want to take a look at the code? I would feel more comfortable with more people looking at the code, just because it is so large. If no one wants to take a look I guess I'll just pull it in sometime next week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-598:Newly submitted topologies do not sh...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/355#issuecomment-71263582 The code compiles now, but I still have not been able to reproduce the original cause of this problem. am I doing something wrong? ``` ./bin/storm dev-zookeeper ./bin/storm nimbus ./bin/storm supervisor ./bin/storm ui ./bin/storm logviewer ``` wait for the UI to come up ``` ./bin/storm jar ./examples/storm-starter/storm-starter-topologies-0.10.0-SNAPSHOT.jar storm.starter.WordCountTopology wc -c topology.workers=4 ./bin/storm jar ./examples/storm-starter/storm-starter-topologies-0.10.0-SNAPSHOT.jar storm.starter.WordCountTopology wc-2 -c topology.workers=4 ``` Both show up in the TopologySummary section of the index.html page. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-629] Place Link to Source Code Reposito...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/386 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-609) add storm-redis to storm external
[ https://issues.apache.org/jira/browse/STORM-609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290058#comment-14290058 ] ASF GitHub Bot commented on STORM-609: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/365#issuecomment-71273044 +1 the changes still look good Are there any other committers that want to take a look at the code? I would feel more comfortable with more people looking at the code, just because it is so large. If no one wants to take a look I guess I'll just pull it in sometime next week. add storm-redis to storm external - Key: STORM-609 URL: https://issues.apache.org/jira/browse/STORM-609 Project: Apache Storm Issue Type: New Feature Affects Versions: 0.10.0 Reporter: DashengJu Assignee: DashengJu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-534) Store Nimbus Server Information in zookeeper path {storm.zookeeper.root}/nimbus
[
https://issues.apache.org/jira/browse/STORM-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288992#comment-14288992
]
ASF GitHub Bot commented on STORM-534:
--
GitHub user caofangkun opened a pull request:
https://github.com/apache/storm/pull/394
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
like {nimbus_host_name}:{nimbus_thrift_port}
May add more information like nimbus server version Information ?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/caofangkun/apache-storm storm-534
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/394.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #394
commit 6c04e1bea802d28ae92a234ddbaad2c1ed0a6835
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T01:28:46Z
Merge pull request #4 from apache/master
Merger from apache/storm to caofangkun/apache-storm
commit 52cbe5f6459f0923c7eda573df1ad5b792c5ba98
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T08:55:42Z
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
commit 55f95b52f5f49c5572d81d6b653babafa67750c4
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T09:02:21Z
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
---
Key: STORM-534
URL: https://issues.apache.org/jira/browse/STORM-534
Project: Apache Storm
Issue Type: Improvement
Affects Versions: 0.10.0
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
1) {nimbus.host} {nimbus.thrift.port} {storm.version} will be stored in
{storm.zookeeper.root}/nimbus like localhost:8826:0.9.3-r1234
2) Storm Clients only need to configure {storm.zookeeper.root} to get Nimbus
Server Information, Configuration like {nimbus.host} {nimbus.thrift.port}
{storm.version} will be ignored
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290432#comment-14290432 ] ASF GitHub Bot commented on STORM-583: -- Github user sasincj commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71300125 @revans2 Without storm-eventhubs, Developers who are using storm with eventhub has to write their own Spout to receive the events. So I thought it would save developer's time when we have it as storm's add-on. For now I'll use it from @shanyu 's branch and let you know my experience... Add spout and bolt implementation for Azure Eventhubs - Key: STORM-583 URL: https://issues.apache.org/jira/browse/STORM-583 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: shanyu zhao Assignee: shanyu zhao Fix For: 0.10.0 Attachments: Storm-EventHubsDesign.docx Add spout and bolt implementations for Azure Eventhubs - a messaging service that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-583: Initiali check-in for storm-eventhu...
Github user sasincj commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71300125 @revans2 Without storm-eventhubs, Developers who are using storm with eventhub has to write their own Spout to receive the events. So I thought it would save developer's time when we have it as storm's add-on. For now I'll use it from @shanyu 's branch and let you know my experience... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-583: Initiali check-in for storm-eventhu...
Github user sasincj commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71170550 @ptgoetz Is storm-eventhubs available to use? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-534:Store Nimbus Server Information in z...
GitHub user caofangkun opened a pull request:
https://github.com/apache/storm/pull/394
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
like {nimbus_host_name}:{nimbus_thrift_port}
May add more information like nimbus server version Information ?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/caofangkun/apache-storm storm-534
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/394.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #394
commit 6c04e1bea802d28ae92a234ddbaad2c1ed0a6835
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T01:28:46Z
Merge pull request #4 from apache/master
Merger from apache/storm to caofangkun/apache-storm
commit 52cbe5f6459f0923c7eda573df1ad5b792c5ba98
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T08:55:42Z
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
commit 55f95b52f5f49c5572d81d6b653babafa67750c4
Author: caofangkun caofang...@gmail.com
Date: 2015-01-23T09:02:21Z
STORM-534:Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-632: New grouping for better load balanc...
GitHub user gdfm opened a pull request: https://github.com/apache/storm/pull/395 STORM-632: New grouping for better load balancing You can merge this pull request into a Git repository by running: $ git pull https://github.com/gdfm/storm STORM-632 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/395.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #395 commit 2c181e9b57ad4f56f8ccca79ca2ceac574492bc1 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2013-12-12T11:35:47Z Update README.markdown commit ab8a77614f26737427f2f8c69bf1e74e169c78a6 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2013-12-12T12:03:40Z add eclipse files to .gitingore commit 1d9bfb38f9f49672df05657bd65935fbb346b588 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2014-12-08T15:15:52Z Merge branch 'master' of github.com:apache/incubator-storm Conflicts: .gitignore README.markdown commit 42398f6ffb7d4b0df14a127edb54cde62a81 Author: Gianmarco De Francisci Morales g...@apache.org Date: 2015-01-23T11:48:46Z Merge branch 'master' of github.com:apache/incubator-storm commit 259c8c25ae7187b3a5fc735a111d14b77d6233c0 Author: Gianmarco De Francisci Morales g...@apache.org Date: 2015-01-23T14:36:08Z Java implementation of partial key grouping + test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-632) New grouping for better load balancing
[ https://issues.apache.org/jira/browse/STORM-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289305#comment-14289305 ] ASF GitHub Bot commented on STORM-632: -- GitHub user gdfm opened a pull request: https://github.com/apache/storm/pull/395 STORM-632: New grouping for better load balancing You can merge this pull request into a Git repository by running: $ git pull https://github.com/gdfm/storm STORM-632 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/395.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #395 commit 2c181e9b57ad4f56f8ccca79ca2ceac574492bc1 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2013-12-12T11:35:47Z Update README.markdown commit ab8a77614f26737427f2f8c69bf1e74e169c78a6 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2013-12-12T12:03:40Z add eclipse files to .gitingore commit 1d9bfb38f9f49672df05657bd65935fbb346b588 Author: Gianmarco De Francisci Morales gdfm+git...@gdfm.me Date: 2014-12-08T15:15:52Z Merge branch 'master' of github.com:apache/incubator-storm Conflicts: .gitignore README.markdown commit 42398f6ffb7d4b0df14a127edb54cde62a81 Author: Gianmarco De Francisci Morales g...@apache.org Date: 2015-01-23T11:48:46Z Merge branch 'master' of github.com:apache/incubator-storm commit 259c8c25ae7187b3a5fc735a111d14b77d6233c0 Author: Gianmarco De Francisci Morales g...@apache.org Date: 2015-01-23T14:36:08Z Java implementation of partial key grouping + test New grouping for better load balancing -- Key: STORM-632 URL: https://issues.apache.org/jira/browse/STORM-632 Project: Apache Storm Issue Type: New Feature Reporter: Gianmarco De Francisci Morales Hi, We have recently studied the problem of load balancing in Storm [1]. In particular, we focused on what happens when the key distribution of the stream is skewed when using key grouping. We developed a new stream partitioning scheme (which we call Partial Key Grouping). It achieves better load balancing than key grouping while being more scalable than shuffle grouping in terms of memory. In the paper we show a number of mining algorithms that are easy to implement with partial key grouping, and whose performance can benefit from it. We think that it might also be useful for a larger class of algorithms. We don't have experience in Clojure, however partial key grouping is very easy to implement: it requires just a few lines of code in Java when implemented as a custom grouping in Storm [2]. We believe it should be very easy to port from Java. For all these reasons, we believe it will be a nice addition to the standard groupings available in Storm. If the community thinks it's a good idea, we will be happy to offer support in the porting. References: [1] https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf [2] https://github.com/gdfm/partial-key-grouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289409#comment-14289409 ] ASF GitHub Bot commented on STORM-583: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71211421 @ptgoetz @revans2 I am +1 on merging this in Add spout and bolt implementation for Azure Eventhubs - Key: STORM-583 URL: https://issues.apache.org/jira/browse/STORM-583 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: shanyu zhao Assignee: shanyu zhao Fix For: 0.10.0 Attachments: Storm-EventHubsDesign.docx Add spout and bolt implementations for Azure Eventhubs - a messaging service that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-583: Initiali check-in for storm-eventhu...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71211421 @ptgoetz @revans2 I am +1 on merging this in --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-583) Add spout and bolt implementation for Azure Eventhubs
[ https://issues.apache.org/jira/browse/STORM-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289443#comment-14289443 ] ASF GitHub Bot commented on STORM-583: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/336#issuecomment-71215853 @sasincj I don't understand the EventHub system well enough to really be able to judge if the code is functioning properly in corner cases, etc. I get at a high level what it does (http://azure.microsoft.com/en-us/services/event-hubs/) but I have never used it myself. The code itself looks OK, but I am not comfortable voting on it either way with my current knowledge. I am a bit nervous about pulling this in as we don't have a very good way to test it beyond the unit tests. EventHubs is a Microsoft Azure specific system. I cannot spin up my own instance of it to validate that the code is doing what I expect prior to a release. Although it may be a bit painful, I can do that for Kafka, Redis, HDFS, and HBase integration that we have accepted in so far. That doesn't mean we cannot/should not pull it in. To me it mostly means that we need more assurances and support prior to doing so. Add spout and bolt implementation for Azure Eventhubs - Key: STORM-583 URL: https://issues.apache.org/jira/browse/STORM-583 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3 Reporter: shanyu zhao Assignee: shanyu zhao Fix For: 0.10.0 Attachments: Storm-EventHubsDesign.docx Add spout and bolt implementations for Azure Eventhubs - a messaging service that supports AMQP protocol. Just like storm-kafka/storm-hbase, we need to add the project to the /external folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-632: New grouping for better load balanc...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/395#issuecomment-71210952 +1 I also filed STORM-637 as a follow on JIRA to finish the integration. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-534) Store Nimbus Server Information in zookeeper path {storm.zookeeper.root}/nimbus
[
https://issues.apache.org/jira/browse/STORM-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289417#comment-14289417
]
Sriharsha Chintalapani commented on STORM-534:
--
[~caofangkun] can you explain benefit of storing this info on zookeeper ,
instead of in config files like today.
Store Nimbus Server Information in zookeeper path
{storm.zookeeper.root}/nimbus
---
Key: STORM-534
URL: https://issues.apache.org/jira/browse/STORM-534
Project: Apache Storm
Issue Type: Improvement
Affects Versions: 0.10.0
Reporter: caofangkun
Assignee: caofangkun
Priority: Minor
1) {nimbus.host} {nimbus.thrift.port} {storm.version} will be stored in
{storm.zookeeper.root}/nimbus like localhost:8826:0.9.3-r1234
2) Storm Clients only need to configure {storm.zookeeper.root} to get Nimbus
Server Information, Configuration like {nimbus.host} {nimbus.thrift.port}
{storm.version} will be ignored
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-632: New grouping for better load balanc...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/395#issuecomment-7126 Oops I missed one thing. None of the new files have the Apache header in them. I am +1 once the header in all of the other java files is also in these. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-632: New grouping for better load balanc...
Github user gdfm commented on the pull request: https://github.com/apache/storm/pull/395#issuecomment-71212234 I always forget them :) Fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-637) Integrate PartialKeyGrouping into storm API
Robert Joseph Evans created STORM-637: - Summary: Integrate PartialKeyGrouping into storm API Key: STORM-637 URL: https://issues.apache.org/jira/browse/STORM-637 Project: Apache Storm Issue Type: Improvement Reporter: Robert Joseph Evans STORM-632 adds support for PartialKey Grouping. It would be really nice to integrate this fully with the storm APIs. So this would mean Update PartialKeysGrouping to optionally take a Fields parameter BoltDeclarer to have a partialKeysGrouping API and TridentTopologyBuilder to have a partialKeysGrouping API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-637) Integrate PartialKeyGrouping into storm API
[ https://issues.apache.org/jira/browse/STORM-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-637: -- Description: STORM-632 adds support for PartialKey Grouping. It would be really nice to integrate this fully with the storm APIs. So this would mean Update PartialKeysGrouping to optionally take a Fields parameter BoltDeclarer to have a partialKeysGrouping API TridentTopologyBuilder to have a partialKeysGrouping API and documentation on how to use a partialKeysGrouping, including some examples. was: STORM-632 adds support for PartialKey Grouping. It would be really nice to integrate this fully with the storm APIs. So this would mean Update PartialKeysGrouping to optionally take a Fields parameter BoltDeclarer to have a partialKeysGrouping API and TridentTopologyBuilder to have a partialKeysGrouping API Integrate PartialKeyGrouping into storm API --- Key: STORM-637 URL: https://issues.apache.org/jira/browse/STORM-637 Project: Apache Storm Issue Type: Improvement Reporter: Robert Joseph Evans STORM-632 adds support for PartialKey Grouping. It would be really nice to integrate this fully with the storm APIs. So this would mean Update PartialKeysGrouping to optionally take a Fields parameter BoltDeclarer to have a partialKeysGrouping API TridentTopologyBuilder to have a partialKeysGrouping API and documentation on how to use a partialKeysGrouping, including some examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-632) New grouping for better load balancing
[ https://issues.apache.org/jira/browse/STORM-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289399#comment-14289399 ] ASF GitHub Bot commented on STORM-632: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/395#issuecomment-71210952 +1 I also filed STORM-637 as a follow on JIRA to finish the integration. New grouping for better load balancing -- Key: STORM-632 URL: https://issues.apache.org/jira/browse/STORM-632 Project: Apache Storm Issue Type: New Feature Reporter: Gianmarco De Francisci Morales Hi, We have recently studied the problem of load balancing in Storm [1]. In particular, we focused on what happens when the key distribution of the stream is skewed when using key grouping. We developed a new stream partitioning scheme (which we call Partial Key Grouping). It achieves better load balancing than key grouping while being more scalable than shuffle grouping in terms of memory. In the paper we show a number of mining algorithms that are easy to implement with partial key grouping, and whose performance can benefit from it. We think that it might also be useful for a larger class of algorithms. We don't have experience in Clojure, however partial key grouping is very easy to implement: it requires just a few lines of code in Java when implemented as a custom grouping in Storm [2]. We believe it should be very easy to port from Java. For all these reasons, we believe it will be a nice addition to the standard groupings available in Storm. If the community thinks it's a good idea, we will be happy to offer support in the porting. References: [1] https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf [2] https://github.com/gdfm/partial-key-grouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-632) New grouping for better load balancing
[ https://issues.apache.org/jira/browse/STORM-632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289422#comment-14289422 ] ASF GitHub Bot commented on STORM-632: -- Github user gdfm commented on the pull request: https://github.com/apache/storm/pull/395#issuecomment-71212234 I always forget them :) Fixed. New grouping for better load balancing -- Key: STORM-632 URL: https://issues.apache.org/jira/browse/STORM-632 Project: Apache Storm Issue Type: New Feature Reporter: Gianmarco De Francisci Morales Hi, We have recently studied the problem of load balancing in Storm [1]. In particular, we focused on what happens when the key distribution of the stream is skewed when using key grouping. We developed a new stream partitioning scheme (which we call Partial Key Grouping). It achieves better load balancing than key grouping while being more scalable than shuffle grouping in terms of memory. In the paper we show a number of mining algorithms that are easy to implement with partial key grouping, and whose performance can benefit from it. We think that it might also be useful for a larger class of algorithms. We don't have experience in Clojure, however partial key grouping is very easy to implement: it requires just a few lines of code in Java when implemented as a custom grouping in Storm [2]. We believe it should be very easy to port from Java. For all these reasons, we believe it will be a nice addition to the standard groupings available in Storm. If the community thinks it's a good idea, we will be happy to offer support in the porting. References: [1] https://melmeric.files.wordpress.com/2014/11/the-power-of-both-choices-practical-load-balancing-for-distributed-stream-processing-engines.pdf [2] https://github.com/gdfm/partial-key-grouping -- This message was sent by Atlassian JIRA (v6.3.4#6332)
