Re: [VOTE] Release Apache Storm 1.1.1 (rc2)

2017-07-27 Thread Jungtaek Lim
+1 (binding) > source - verify file (signature, MD5, SHA) -- source, tar.gz : OK -- source, zip : OK - extract file -- source, tar.gz : OK -- source, zip : OK - diff-ing extracted files between tar.gz and zip : OK - build source with JDK 7 -- source, tar.gz : - build source dist -

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r130010442 --- Diff: storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.java --- @@ -39,7 +38,7 @@ public KryoTupleDeserializer(final Map c

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r130010224 --- Diff: storm-client/src/jvm/org/apache/storm/topology/SimpleWindowPartitionCache.java --- @@ -0,0 +1,191 @@ +/** + * Licensed to the Apache Softw

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r130009889 --- Diff: storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.java --- @@ -39,7 +38,7 @@ public KryoTupleDeserializer(final Map conf,

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r130009804 --- Diff: storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.java --- @@ -39,7 +38,7 @@ public KryoTupleDeserializer(final Map c

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2241 Agree with @HeartSaVioR. If possible lets break this down into multiple patches like (1) JCQ replacing disruptor (2) changing the threading model (3) micro optimizations and so on which makes i

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread arunmahadevan
Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r130009128 --- Diff: storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.java --- @@ -39,7 +38,7 @@ public KryoTupleDeserializer(final Map

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 Just leaving a note to make my requirements clear (it is quite simple): - new system doesn't break anything it worked - if they're unavoidable it should be discussed from Storm comm

[GitHub] storm issue #2200: STORM-2616: Documentation for built in metrics

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2200 +1 Nice documentation! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishe

[GitHub] storm issue #2245: Restore issues missing from changelog since fb2446075de20...

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2245 As you may know, you can even just made a change if it was wrong. Now you also think it's hard to maintain CHANGELOG, let's see how the discussion goes. --- If your project is set up for it, you

[GitHub] storm issue #2245: Restore issues missing from changelog since fb2446075de20...

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2245 +1 You can merge this without waiting 24hr. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

Re: [DISCUSS] Remove CHANGELOG file

2017-07-27 Thread Jungtaek Lim
correction: other projects -> *some* other projects, though they're popular projects (including in competition) 2017년 7월 28일 (금) 오전 10:51, Jungtaek Lim 님이 작성: > I'm happy that there're other guys having same difficult and sharing same > feeling. > > This discussion has been initiating several tim

Re: [DISCUSS] Remove CHANGELOG file

2017-07-27 Thread Jungtaek Lim
I'm happy that there're other guys having same difficult and sharing same feeling. This discussion has been initiating several times (from me) and getting some +1s for each thread but didn't reach to actual work. We already utilize JIRA, and I'm subscribing issues@ and taking care of issues forgo

Re: Request for holding off commits to storm-client module

2017-07-27 Thread Jungtaek Lim
Now pull request for STORM-2306 is up, and it looks like requiring several weeks (months?) to be verified. IMHO the effort of rebasing is unavoidable (though we could put effort to minimize), especially pull requests were up already. Unfortunately, that's about who takes the load, and I don't want

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 Btw, I think talking about current state is less meaningful. This patch has lots of TODO and some critical identified issues, so it should be addressed, and after that the number is going to be r

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 I don't think utilizing metrics consumer in TVL is the issue: it might matter if results are close so that contributions of other system component does matter, but it is just not acceptable laten

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on the issue: https://github.com/apache/storm/pull/2241 @revans2 About that "better than sliced bread" : how could i not be offended.. at least briefly ;-) but you could buy me lunch if this PR turns out better than you were initially a

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2241 @revans2 I am trying to reproduce the worst-case in your last chart. Running TVL topology with 4 spout, 10 splitters, 4 counters, 2 ackers. Here is the code https://gist.github.com/harshach/73d

Re: [DISCUSS] Remove CHANGELOG file

2017-07-27 Thread Stig Rohde Døssing
We already have to keep JIRA updated, and keeping JIRA consistent is easier since there isn't one view of the resolved issues for each git branch like we have with CHANGELOG, so there's no worry about e.g. master having a different opinion on solved issues in 1.2.0 than 1.x-branch has. I think we

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2241 @revans2 I am trying to reproduce the worst-case in your last chart. Running TVL topology with 4 spout, 10 splitters, 4 counters, 2 ackers. Here is the code https://gist.github.com/harshach/73d

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on the issue: https://github.com/apache/storm/pull/2241 Some points covering prev comments by @HeartSaVioR and @revans2 **Throughput limiting:** That only makes sense if you are measuring Throughput vs CPU/other resource usage. Latency measur

[GitHub] storm pull request #476: STORM-720: Storm.cmd should return ERRORLEVEL befor...

2017-07-27 Thread rtandonmsft
Github user rtandonmsft closed the pull request at: https://github.com/apache/storm/pull/476 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] Remove CHANGELOG file

2017-07-27 Thread Hugo Da Cruz Louro
I am +1 for discontinuing CHANGELOG. However, using JIRA to compile this info will only work if contributors are very disciplined and consistent updating JIRA. That leads to the question, is it any easier to maintain JIRA consistent then it is to keep CHANGELOG consistent? A clean and consistent

Re: [DISCUSS] Remove CHANGELOG file

2017-07-27 Thread Stig Rohde Døssing
Sorry to necro this thread, but I think it's worth bringing this issue up again. As Jungtaek mentioned a manual changelog is easy to break, and it looks like some issues are listed wrong on master and missing from 1.x ( https://github.com/apache/storm/pull/2245) I think dropping CHANGELOG is a gre

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on the issue: https://github.com/apache/storm/pull/2241 For the new messaging system.. the scaling rule of thumb I have found so far is quite simple. For fast topos (and CPU intensive topos) ... 1 executor thread per *physical core*. It appl

[GitHub] storm pull request #2245: Restore issues missing from changelog since fb2446...

2017-07-27 Thread srdo
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/2245 Restore issues missing from changelog since fb2446075de202387658b0cc8… …434ee53ded5d35a See https://github.com/apache/storm/commit/fb2446075de202387658b0cc8434ee53ded5d35a I've gr

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129946825 --- Diff: storm-client/src/jvm/org/apache/storm/Config.java --- @@ -548,22 +518,14 @@ public static final String TOPOLOGY_AUTO_TASK_HOOKS="topolo

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread revans2
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2241 I have another chart now showing a comparison between master and this branch, just varying the number of splitter bolts in the topology. There are 2 ackers, 4 spouts, and 4 count bolts all within a

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129945279 --- Diff: storm-client/src/jvm/org/apache/storm/utils/TransferDrainer.java --- @@ -21,101 +21,100 @@ import java.util.HashMap; import java.util.

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129938970 --- Diff: storm-client/src/jvm/org/apache/storm/daemon/Task.java --- @@ -122,28 +130,33 @@ public Task(Executor executor, Integer taskId) throws IOExcepti

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129937665 --- Diff: storm-client/src/jvm/org/apache/storm/daemon/GrouperFactory.java --- @@ -137,7 +137,7 @@ public void prepare(WorkerTopologyContext context, Glo

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129936657 --- Diff: storm-client/src/jvm/org/apache/storm/task/IOutputCollector.java --- @@ -30,4 +31,5 @@ void ack(Tuple input); void fail(Tuple

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129934123 --- Diff: conf/defaults.yaml --- @@ -304,6 +303,7 @@ storm.cgroup.resources: storm.cgroup.hierarchy.name: "storm" storm.supervisor.cgroup.rootdir

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129930495 --- Diff: conf/defaults.yaml --- @@ -231,16 +228,13 @@ topology.multilang.serializer: "org.apache.storm.multilang.JsonSerializer" topology.shellbolt.

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129931296 --- Diff: conf/defaults.yaml --- @@ -231,16 +228,13 @@ topology.multilang.serializer: "org.apache.storm.multilang.JsonSerializer" topology.shellbolt.

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129934167 --- Diff: examples/storm-elasticsearch-examples/src/main/java/org/apache/storm/elasticsearch/common/EsTestUtil.java --- @@ -43,7 +43,7 @@ public Fields g

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread roshannaik
Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/2241#discussion_r129934940 --- Diff: examples/storm-perf/src/main/java/org/apache/storm/perf/ConstSpoutIdBoltNullBoltTopo.java --- @@ -76,6 +76,8 @@ public static StormTopology get

[RESULT] [VOTE] Release Apache Storm 1.0.4 (rc1)

2017-07-27 Thread P. Taylor Goetz
This vote is now closed and passes with X binding +1 votes and no 0 or -1 votes. Vote tally (* indicates binding votes): +1: Stig Rhode Døssing* Bobby Evans* Kishor Patil* Jungtaek Lim* P. Taylor Goetz* I will release the staged artifacts and announce the release after the 24 hour waiting perio

Re: [VOTE] Release Apache Storm 1.0.4 (rc1)

2017-07-27 Thread P. Taylor Goetz
+1 (binding) -Taylor > On Jul 24, 2017, at 2:24 PM, P. Taylor Goetz wrote: > > This is a call to vote on releasing Apache Storm 1.0.4 (rc1) > > Full list of changes in this release: > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=50878fab679973a1230466920

[VOTE] Release Apache Storm 1.1.1 (rc2)

2017-07-27 Thread P. Taylor Goetz
This is a call to vote on releasing Apache Storm 1.1.1 (rc2) Full list of changes in this release: https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=41bfea87b1a002565333bd18a06d766af1ca3816 The tag/commit to be voted upon is v1.1.1: https://git-wip-us.apache.org

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread revans2
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2241 I have run some more tests looking at modifying the parallelism of different components. First I kept the parallelism of everything else at 4 and modified the acker count. ![chart_acker

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread revans2
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2241 Sorry, but we also need to think about low throughput use cases. I have several that I care about and I am seeing very long latency for low throughput. min in some cases is 5 seconds, max

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread revans2
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2241 @harshach Reiterating what @HeartSaVioR said about benchmarking. Most benchmarking is done where you push a system to its limits and see what maximum throughput it can do. This is far fro

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 I have updated all the results for TVL with second parameter set to 1. Also added rate 5. The CPU usage from current master doesn't fluctuate from all of rates, even 5, whereas with t

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 Let me share a quick test result with passing `1 1` to TVL parameter: > STORM-2306 ``` uptime: 30 acked: 144,000 acked/sec: 4,800.00 failed:0 99%: 3,070,2

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129809356 --- Diff: docs/Windowing.md --- @@ -266,3 +266,108 @@ tuples can be received within the timeout period. An example toplogy `SlidingWindowTopology` shows

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 @harshach The second argument is effectively representing worker count: you can see that topology set worker count as parallelism. I agree that the name is really misleading even I ran tests with

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2241 @HeartSaVioR Its not 12 executors per worker. If you don't pass a command-line argument, it sets parallelism variable here to 4 https://github.com/apache/storm/blob/master/examples/storm-starter/sr

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r128993342 --- Diff: storm-client/src/jvm/org/apache/storm/topology/PersistentWindowedBoltExecutor.java --- @@ -0,0 +1,563 @@ +/** + * Licensed to the Apache S

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129003027 --- Diff: storm-client/src/jvm/org/apache/storm/topology/PersistentWindowedBoltExecutor.java --- @@ -0,0 +1,563 @@ +/** + * Licensed to the Apache S

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129791075 --- Diff: storm-client/src/jvm/org/apache/storm/topology/PersistentWindowedBoltExecutor.java --- @@ -0,0 +1,596 @@ +/** + * Licensed to the Apache S

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r128991661 --- Diff: storm-client/src/jvm/org/apache/storm/topology/base/BaseStatefulWindowedBolt.java --- @@ -151,6 +158,38 @@ return this; }

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r128992641 --- Diff: storm-client/src/jvm/org/apache/storm/topology/IStatefulWindowedBolt.java --- @@ -15,12 +15,33 @@ * See the License for the specific language

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129760482 --- Diff: storm-client/src/jvm/org/apache/storm/topology/PersistentWindowedBoltExecutor.java --- @@ -0,0 +1,563 @@ +/** + * Licensed to the Apache S

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129803794 --- Diff: storm-client/src/jvm/org/apache/storm/topology/WindowPartitionCache.java --- @@ -0,0 +1,142 @@ +/** + * Licensed to the Apache Software Fou

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-27 Thread satishd
Github user satishd commented on a diff in the pull request: https://github.com/apache/storm/pull/2218#discussion_r129804787 --- Diff: storm-client/src/jvm/org/apache/storm/topology/SimpleWindowPartitionCache.java --- @@ -0,0 +1,191 @@ +/** + * Licensed to the Apache Softw

Re: [GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread Alexandre Vermeerbergen
Hello Roshan, Thank you for your detailed answer. Details are important, because in my organization, I am often asked to re-assess the reasons why we chose Storm over its competitors. Best regards, Alexandre Vermeerbergen 2017-07-25 23:36 GMT+02:00 roshannaik : > Github user roshannaik comment

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2241 @harshach For ThroughputvsLatency, throttling spout is intended. We set desired throughput and see histogram of latency and other metrics. (CPU, GC, etc.) There're 3 components in topology w

[GitHub] storm issue #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-27 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2241 @revans2 @HeartSaVioR Here are my findings https://docs.google.com/spreadsheets/d/1wPpC3YXp-vTIelRTUVoLxuxIYxUekiIHUpZ2ZEysC4Y/edit#gid=1644511... 1. Looking at ThroughputvsLatency

[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...

2017-07-27 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2218 @srdo, addressed your comments, let me know if I missed something. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your projec