[GitHub] storm issue #1794: STORM-2193: Fix FilterConfiguration parameter order
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1794 Great finding. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1795: STORM-2215: validate blobs are present before subm...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1795#discussion_r89416093
--- Diff: storm-core/src/jvm/org/apache/storm/StormSubmitter.java ---
@@ -587,4 +592,10 @@ private static double
getMaxExecutorMemoryUsageForTopo(StormTopology topology, M
}
return largestMemoryOperator;
}
+
+private static Set getListOfKeysFromBlobStore(Map stormConf) {
+NimbusBlobStore client = new NimbusBlobStore();
--- End diff --
Need to guarantee calling client.shutdown().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1795: STORM-2215: validate blobs are present before subm...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1795#discussion_r89416315
--- Diff: storm-core/src/jvm/org/apache/storm/utils/Utils.java ---
@@ -1338,6 +1339,26 @@ public static void handleUncaughtException(Throwable
t) {
}
}
+public static void validateTopologyBlobStoreMap(Map stormConf,
Set blobStoreKeys) throws InvalidTopologyException {
+boolean containsAllBlobs = true;
--- End diff --
Minor: We're adding missing keys to Set so no need to have additional flag.
`missingKeys.size() > 0` is same to `containsAllBlobs == false`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1796: STORM-2216: prefer JSONValue.parseWithException
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1796 +1 Looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1783: STORM-2204 Adding caching capabilities in HBaseLookupBolt
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1783 @revans2 What do you think about this? I'm in favor of adopting Caffeine, and I'm even OK to use Caffeine to master and Guava to 1.x. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1744: STORM-1276: line for line translation of nimbus to java
Github user abellina commented on the issue: https://github.com/apache/storm/pull/1744 nimbus.clj, I made it to mkAssignments. Will continue from there later: - nit: makeBlobCachMap -> makeBlobCacheMap - change: `numDesiredWorkers < numAssignedWorkers` to `numDesiredWorkers > numAssignedWorkers` in: https://github.com/revans2/incubator-storm/blob/39148edf07f2028c2edcfecc53ea9bfac525988f/storm-core/src/jvm/org/apache/storm/daemon/nimbus/Nimbus.java#L1605 - in getResourcesForTopology: do we care if the values are set? (is_set_cpu for example). Those values are doubles. - nit: update comment //TODO remove both of swaps below at first opportunity. But yeah "uglyResources" is right :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
FOSDEM 2017 HPC, Bigdata and Data Science DevRoom CFP is closing soon
Hi! apologies for the extra wide distribution (this exhausts my once a year ASF mail-to-all-bigdata-projects quota ;-)) but I wanted to suggest that all of you should consider submitting talks to FOSDEM 2017 HPC, Bigdata and Data Science DevRoom: https://hpc-bigdata-fosdem17.github.io/ It was a great success this year and we hope to make it an even bigger success in 2017. Besides -- FOSDEM is the biggest gathering of open source developers on the face of the earth -- don't miss it! Thanks, Roman. P.S. If you have any questions -- please email me directly and see you all in Brussels!
[GitHub] storm pull request #1789: STORM-2209: Update documents adding new integratio...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1789 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1789: STORM-2209: Update documents adding new integration for s...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1789 Still +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1791: STORM-2212: Remove Redundant Declarations in Maven...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1791 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1791: STORM-2212: Remove Redundant Declarations in Maven POM Fi...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1791 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1792: STORM-2213 ShellSpout has race condition when ShellSpout ...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1792 +1 looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1790: STORM-2210 - remove array shuffle from ShuffleGrou...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1790 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1797: STORM-2210 remove array shuffle from ShuffleGroupi...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1797 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1797: STORM-2210 remove array shuffle from ShuffleGrouping when...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1797 +1, because this is from #1790 that was already approved I will not wait the full 24 hours. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Feature Branch for Apache Beam Runner
Hugo, This appears to be the branch: - https://github.com/apache/storm/tree/beam-runner - Erik On Wed, Nov 23, 2016 at 9:34 AM, Hugo Da Cruz Louro wrote: > I somehow missed this email … I would like to contribute to this effort as > well. Please keep me posted. > Thanks. > > > On Oct 19, 2016, at 8:51 AM, Satish Duggana > wrote: > > > > +1, waiting for that. :) > > Currently,there are API changes going on in Beam. It seem they plan to > get > > that done by the end of 2016. > > > > ~Satish. > > > > On Wed, Oct 19, 2016 at 9:19 PM, Bobby Evans > > > wrote: > > > >> +1 - Bobby > >> > >>On Wednesday, October 19, 2016 10:30 AM, Arun Mahadevan < > >> ar...@apache.org> wrote: > >> > >> > >> +1 > >> > >> On 10/19/16, 8:58 PM, "P. Taylor Goetz" wrote: > >> > >>> If there are no objections, I’d like to create the feature branch and > >> push what I have so far. I’ve not had too much time lately to work on > it, > >> but other’s have expressed interest in contributing so I’d like to make > it > >> available. > >>> > >>> -Taylor > >>> > >>> > On Sep 19, 2016, at 11:15 AM, Bobby Evans > > >> wrote: > > +1 on the idea. I would love to contribute, but I doubt I will find > >> time to do it any time soon. - Bobby > > On Friday, September 16, 2016 12:05 AM, Satish Duggana < > >> satish.dugg...@gmail.com> wrote: > > > Taylor, > I am interested in contributing to this effort. Gone through Beam APIs > earlier and had some initial thoughts on Storm runner. We can start > with > existing core storm constructs but it is better to design in such a > way > that these can be replaced with new APIs. > > Thanks, > Satish. > > On Fri, Sep 16, 2016 at 3:35 AM, P. Taylor Goetz > >> wrote: > > > I'm open to change, but yes, I started with core storm since it > offers > >> the > > most flexibility wrt how Beam constructs are translated. > > > > -Taylor > > > >> On Sep 15, 2016, at 5:51 PM, Roshan Naik > >> wrote: > >> > >> Good idea. Will the Beam API be implemented to run on top Storm Core > >> primitives ? > >> -roshan > >> > >> > >>> On 9/15/16, 2:00 PM, "P. Taylor Goetz" wrote: > >>> > >>> I¹ve been tinkering with implementing an Apache Beam runner on top > of > >>> Storm and would like to open it up so others in the community can > >>> contribute. To that end I¹d like to propose creating a feature > branch > > for > >>> that work if there are others who are interested in getting > >> involved. We > >>> did that a while back when storm-sql was originally developed. > >>> > >>> Basically, review requirements for that branch would be relaxed > >> during > >>> development, with a final, strict review before merging back to one > >> of > >>> our main branches. > >>> > >>> I¹d like to document what I have and future improvements in a > >> proposal > >>> document, and follow that with pushing the code to the feature > branch > > for > >>> group collaboration. > >>> > >>> Any thoughts? Anyone interested in contributing to such an effort? > >>> > >>> -Taylor > >> > > > > >>> > >> > >> > >> > >> > >> > >
Re: [DISCUSS] Feature Branch for Apache Beam Runner
I somehow missed this email … I would like to contribute to this effort as well. Please keep me posted. Thanks. > On Oct 19, 2016, at 8:51 AM, Satish Duggana wrote: > > +1, waiting for that. :) > Currently,there are API changes going on in Beam. It seem they plan to get > that done by the end of 2016. > > ~Satish. > > On Wed, Oct 19, 2016 at 9:19 PM, Bobby Evans > wrote: > >> +1 - Bobby >> >>On Wednesday, October 19, 2016 10:30 AM, Arun Mahadevan < >> ar...@apache.org> wrote: >> >> >> +1 >> >> On 10/19/16, 8:58 PM, "P. Taylor Goetz" wrote: >> >>> If there are no objections, I’d like to create the feature branch and >> push what I have so far. I’ve not had too much time lately to work on it, >> but other’s have expressed interest in contributing so I’d like to make it >> available. >>> >>> -Taylor >>> >>> On Sep 19, 2016, at 11:15 AM, Bobby Evans >> wrote: +1 on the idea. I would love to contribute, but I doubt I will find >> time to do it any time soon. - Bobby On Friday, September 16, 2016 12:05 AM, Satish Duggana < >> satish.dugg...@gmail.com> wrote: Taylor, I am interested in contributing to this effort. Gone through Beam APIs earlier and had some initial thoughts on Storm runner. We can start with existing core storm constructs but it is better to design in such a way that these can be replaced with new APIs. Thanks, Satish. On Fri, Sep 16, 2016 at 3:35 AM, P. Taylor Goetz >> wrote: > I'm open to change, but yes, I started with core storm since it offers >> the > most flexibility wrt how Beam constructs are translated. > > -Taylor > >> On Sep 15, 2016, at 5:51 PM, Roshan Naik >> wrote: >> >> Good idea. Will the Beam API be implemented to run on top Storm Core >> primitives ? >> -roshan >> >> >>> On 9/15/16, 2:00 PM, "P. Taylor Goetz" wrote: >>> >>> I¹ve been tinkering with implementing an Apache Beam runner on top of >>> Storm and would like to open it up so others in the community can >>> contribute. To that end I¹d like to propose creating a feature branch > for >>> that work if there are others who are interested in getting >> involved. We >>> did that a while back when storm-sql was originally developed. >>> >>> Basically, review requirements for that branch would be relaxed >> during >>> development, with a final, strict review before merging back to one >> of >>> our main branches. >>> >>> I¹d like to document what I have and future improvements in a >> proposal >>> document, and follow that with pushing the code to the feature branch > for >>> group collaboration. >>> >>> Any thoughts? Anyone interested in contributing to such an effort? >>> >>> -Taylor >> > >>> >> >> >> >> >>
Should zookeeper be run on the worker machines?
Hi, Can someone please respond to this zookeeper-for-storm Stack-Overflow question: http://stackoverflow.com/questions/40755137/should-zookeeper-be-run-on-the-worker-machines Thanks TI
[GitHub] storm issue #1790: STORM-2210 - remove array shuffle from ShuffleGrouping wh...
Github user kevpeek commented on the issue: https://github.com/apache/storm/pull/1790 @HeartSaVioR I created a PR on master with the squashed version of these commits. I hope that's flexible enough to do what you need. https://github.com/apache/storm/pull/1797 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1797: STORM-2210 remove array shuffle from ShuffleGroupi...
GitHub user kevpeek opened a pull request: https://github.com/apache/storm/pull/1797 STORM-2210 remove array shuffle from ShuffleGrouping when counter res⦠This is a squashed version of PR1790 made against master. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevpeek/storm STORM-2210-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1797.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1797 commit 69de52aa06ebdcb47a994f945c15f6a8aa429adc Author: Kevin Peek Date: 2016-11-23T13:18:26Z STORM-2210 remove array shuffle from ShuffleGrouping when counter resets to zero --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Storm sending rate
Hi everyone, Could anyone has an experience to explain the factors affecting sending rate in Storm? Thank you--RegardsWA
[GitHub] storm issue #1781: STORM-1369: Add MapState implementation to storm-cassandr...
Github user mkoch1 commented on the issue: https://github.com/apache/storm/pull/1781 That should do it - the batching option has been removed from MapState, in favor of parallel processing with existing opaque and transactional logic to handle consistency. Cassandra batch statements are more trouble than they're worth. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1789: STORM-2209: Update documents adding new integration for s...
Github user vesense commented on the issue: https://github.com/apache/storm/pull/1789 Added kubernetes integration. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
