[GitHub] storm pull request #1913: STORM-2244 Some shaded jars doesn't exclude depend...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1913

STORM-2244 Some shaded jars doesn't exclude dependency signature files

* Add filter option to all of shade plugin configurations

PR for master branch (#1834) is already merged.

@harshach Could you review this as well? Thanks!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2244-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1913


commit 30b80c70eb00edcff13f4b789b90951b3f68d004
Author: Jungtaek Lim 
Date:   2016-12-18T22:24:08Z

STORM-2244 Some shaded jars doesn't exclude dependency signature files

* Add filter option to all of shade plugin configurations




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1905: STORM-2321 Handle blobstore zk key deletion in KeySequenc...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1905
  
@harshach Could you review this as well? Thanks in advance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1907: STORM-2327 ConfigurableTopology

[jnioche]

Github user jnioche commented on the issue:

https://github.com/apache/storm/pull/1907
  
Thanks @HeartSaVioR 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1912: Storm 2334 - Bolt for joining multiple streams

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1912
  
@roshannaik You seemed to pick wrong target branch. Could you fix it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1912: Storm 2334 - Bolt for joining multiple streams

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/1912
  
@HeartSaVioR  ... oops sorry.. will redo the PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1912: Storm 2334 - Bolt for joining multiple streams

[roshannaik]

Github user roshannaik closed the pull request at:

https://github.com/apache/storm/pull/1912


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[roshannaik]

GitHub user roshannaik opened a pull request:

https://github.com/apache/storm/pull/1914

STORM-2334 - Join Bolt implementation with unit tests

For details see : 
https://issues.apache.org/jira/browse/STORM-2334

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/roshannaik/storm STORM-2334

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1914


commit f193e266a5f2701c2a91fa8e73778d6e4c2fa3d7
Author: Roshan Naik 
Date:   2017-02-01T01:21:46Z

STORM-2334 Join Bolt implementation with unit tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1910: STORM-2335 Fix broken Topology visualization with ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1910


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[raghavgautam]

Github user raghavgautam commented on a diff in the pull request:

https://github.com/apache/storm/pull/1914#discussion_r99135627
  
--- Diff: storm-core/src/jvm/org/apache/storm/bolt/JoinBolt.java ---
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.bolt;
+
+
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseWindowedBolt;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TupleWindow;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+
+public class JoinBolt extends BaseWindowedBolt {
+
+private OutputCollector collector;
+
+// Map[StreamName -> Map[Key -> List]  ]
+HashMap >> hashedInputs = new 
HashMap<>(); // holds remaining streams
+
+// Map[StreamName -> JoinInfo]
+protected LinkedHashMap joinCriteria = new 
LinkedHashMap<>();
+protected String[][] outputKeys;  // specified via bolt.select() ... 
used in declaring Output fields
+protected String[] dotSeparatedOutputKeyNames; // flattened (de 
nested) keyNames, used for naming output fields
+protected String outputStreamName;
+
+// Use streamId, source component name OR field in tuple to 
distinguish incoming tuple streams
+public enum Selector { STREAM, SOURCE }
+protected final Selector selectorType;
+
+
+/**
+ * StreamId to start the join with. Equivalent SQL ...
+ *   select  from streamId ...
+ * @param type Specifies whether 'streamId' refers to stream 
name/source component
+ * @param streamId name of stream/source component
+ * @param key the fieldName to use as key for the stream (used for 
performing joins)
+ */
+public JoinBolt(Selector type, String streamId, String key) {
+selectorType = type;
+joinCriteria.put(streamId, new JoinInfo(key) );
+}
+
+/**
+ * Defines the name of the output stream
+ */
+public JoinBolt withOutputStream(String streamName) {
+this.outputStreamName = streamName;
+return this;
+}
+
+/**
+ * Performs inner Join.
+ *  SQL:   from priorStream inner join newStream on newStream.key 
= priorStream.key1
+ *  same as:   new WindowedQueryBolt(priorStream,key1). 
join(newStream, key, priorStream);
+ *
+ *  Note: priorStream must be previously joined.
+ *Valid ex:new WindowedQueryBolt(s1,k1). join(s2,k2, s1). 
join(s3,k3, s2);
--- End diff --

It would be probably better if the user could specify joins by saying:
new WindowedQueryBolt(s1,k1). join(s2,k2). join(s3,k3);
(note missing s1, s2 parameters in the join method)
This will be less error prone and more in line with SQL expectation where 
user specifies:
table1.key1 = table2.key2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1911: STORM-2336 Close Localizer and AsyncLocalizer when...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1911


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log...

[srdo]

Github user srdo closed the pull request at:

https://github.com/apache/storm/pull/1381


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log...

[srdo]

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1381

STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to warn from error

This error is transient most of the time and will occur during normal 
operation. Downgrading it to warning makes the error log much less noisy.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1746

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1381.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1381


commit 7e5b9c50c54dba43e2cac1b5f0ad3c7eb001d060
Author: Stig Rohde Døssing 
Date:   2016-04-29T09:20:32Z

STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to warn from error

commit 8307a860629f6cbbfe10c0401143180d7a9a06d0
Author: Stig Rohde Døssing 
Date:   2016-04-29T10:34:18Z

STORM-1746: Remove duplicate log from KafkaUtils, add FailedFetchException 
to TridentKafkaEmitter log




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log...

[srdo]

Github user srdo closed the pull request at:

https://github.com/apache/storm/pull/1381


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1381: STORM-1746: Downgrade NOT_LEADER_FOR_PARTITION log to war...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/1381
  
Closing since it seems like this is a fairly irrelevant change to what is 
now legacy code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[harshach]

Github user harshach commented on a diff in the pull request:

https://github.com/apache/storm/pull/1914#discussion_r99169659
  
--- Diff: storm-core/src/jvm/org/apache/storm/bolt/JoinBolt.java ---
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.bolt;
+
+
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseWindowedBolt;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TupleWindow;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+
+public class JoinBolt extends BaseWindowedBolt {
+
+private OutputCollector collector;
+
+// Map[StreamName -> Map[Key -> List]  ]
+HashMap >> hashedInputs = new 
HashMap<>(); // holds remaining streams
+
+// Map[StreamName -> JoinInfo]
+protected LinkedHashMap joinCriteria = new 
LinkedHashMap<>();
+protected String[][] outputKeys;  // specified via bolt.select() ... 
used in declaring Output fields
+protected String[] dotSeparatedOutputKeyNames; // flattened (de 
nested) keyNames, used for naming output fields
+protected String outputStreamName;
+
+// Use streamId, source component name OR field in tuple to 
distinguish incoming tuple streams
+public enum Selector { STREAM, SOURCE }
+protected final Selector selectorType;
+
+
+/**
+ * StreamId to start the join with. Equivalent SQL ...
+ *   select  from streamId ...
+ * @param type Specifies whether 'streamId' refers to stream 
name/source component
+ * @param streamId name of stream/source component
+ * @param key the fieldName to use as key for the stream (used for 
performing joins)
+ */
+public JoinBolt(Selector type, String streamId, String key) {
+selectorType = type;
+joinCriteria.put(streamId, new JoinInfo(key) );
+}
+
+/**
+ * Defines the name of the output stream
+ */
+public JoinBolt withOutputStream(String streamName) {
+this.outputStreamName = streamName;
+return this;
+}
+
+/**
+ * Performs inner Join.
+ *  SQL:   from priorStream inner join newStream on newStream.key 
= priorStream.key1
+ *  same as:   new WindowedQueryBolt(priorStream,key1). 
join(newStream, key, priorStream);
+ *
+ *  Note: priorStream must be previously joined.
+ *Valid ex:new WindowedQueryBolt(s1,k1). join(s2,k2, s1). 
join(s3,k3, s2);
+ *Invalid ex:  new WindowedQueryBolt(s1,k1). join(s3,k3, s2). 
join(s2,k2, s1);
+ */
+public JoinBolt join(String newStream, String key, String priorStream) 
{
+return join_common(newStream, key, priorStream, JoinType.INNER);
+}
+
+/**
+ * Performs left Join.
+ *  SQL:   from stream1  left join stream2  on stream2.key = 
stream1.key1
+ *  same as:   new  WindowedQueryBolt(stream1, key1). 
leftJoin(stream2, key, stream1);
+ *
+ *  Note: priorStream must be previously joined
+ *Valid ex:new WindowedQueryBolt(s1,k1). leftJoin(s2,k2, s1). 
leftJoin(s3,k3, s2);
+ *Invalid ex:  new WindowedQueryBolt(s1,k1). leftJoin(s3,k3, s2). 
leftJoin(s2,k2, s1);
+ */
+public JoinBolt leftJoin(String newStream, String key, String 
priorStream) {
+return join_common(newStream, key, priorStream, JoinType.LEFT);
+}
+
+private JoinBolt join_common(String newStream, String key, String 
priorStream, JoinType joinType) {
--- End diff --

can we keep the method names to camelcase like join

[GitHub] storm issue #1913: STORM-2244 Some shaded jars doesn't exclude dependency si...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1913
  
+1. looks good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1905: STORM-2321 Handle blobstore zk key deletion in Key...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1915: STORM-2337: Broken documentation generation for st...

[tibkiss]

GitHub user tibkiss opened a pull request:

https://github.com/apache/storm/pull/1915

STORM-2337: Broken documentation generation for 
storm-metrics-profiling-internal-actions.md and windows-users-guide.md



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tibkiss/storm 
fix/missing-generated-documentation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1915


commit 3806ab369be8fea6529bb7acc8ebd593be3fb977
Author: Tibor Kiss 
Date:   2017-02-02T18:39:19Z

STORM-2337: Broken documentation generation for 
storm-metrics-profiling-internal-actions.md and windows-users-guide.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1739: STORM-1443 [Storm SQL] Support customizing parallelism in...

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1739
  
+1 Being able to control parallelism will make Storm SQL a lot more usable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1743: STORM-2148 [Storm SQL] Trident mode: back to code generat...

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1743
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache Storm 1.0.3 (rc1)

[Alexandre Vermeerbergen]

Hello all,

Using storm 1.0.3rc1 binaries artifacts, I installed our supervision system
(currently made of 11 topologies), with java 8 update 92 on CentOS 6.5 ;
with a cluster type of install (2 big VMs with Supervisors, another VM with
Nimbus, and other VMs for Zookeeper nodes).

So far so good => +1 (non binding)

It's been running for only 1 hour, I may update if I see issues in the
coming days. I'll pay attention to stability & performance (too early to
tell).

Thanks,

Alexandre Vermeerbergen



2017-01-31 21:55 GMT+01:00 P. Taylor Goetz :

> This is a call to vote on releasing Apache Storm 1.0.3 (rc1)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> 6cb735d18ec11eccd3e80bab8f879c989a9b3967;hb=a81ec2580fce1f2ee6349a9028dcb7
> 5763798bec
>
> The tag/commit to be voted upon is v1.0.3:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;h=b8adddc6aa20107288e59cba6a2976c0951742fb;hb=
> a81ec2580fce1f2ee6349a9028dcb75763798bec
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 0.3-rc1/apache-storm-1.0.3-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1043
>
> Please vote on releasing this package as Apache Storm 1.0.3.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.0.3
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/1914#discussion_r99227905
  
--- Diff: storm-core/src/jvm/org/apache/storm/bolt/JoinBolt.java ---
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.bolt;
+
+
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseWindowedBolt;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TupleWindow;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+
+public class JoinBolt extends BaseWindowedBolt {
+
+private OutputCollector collector;
+
+// Map[StreamName -> Map[Key -> List]  ]
+HashMap >> hashedInputs = new 
HashMap<>(); // holds remaining streams
+
+// Map[StreamName -> JoinInfo]
+protected LinkedHashMap joinCriteria = new 
LinkedHashMap<>();
+protected String[][] outputKeys;  // specified via bolt.select() ... 
used in declaring Output fields
+protected String[] dotSeparatedOutputKeyNames; // flattened (de 
nested) keyNames, used for naming output fields
+protected String outputStreamName;
+
+// Use streamId, source component name OR field in tuple to 
distinguish incoming tuple streams
+public enum Selector { STREAM, SOURCE }
+protected final Selector selectorType;
+
+
+/**
+ * StreamId to start the join with. Equivalent SQL ...
+ *   select  from streamId ...
+ * @param type Specifies whether 'streamId' refers to stream 
name/source component
+ * @param streamId name of stream/source component
+ * @param key the fieldName to use as key for the stream (used for 
performing joins)
+ */
+public JoinBolt(Selector type, String streamId, String key) {
+selectorType = type;
+joinCriteria.put(streamId, new JoinInfo(key) );
+}
+
+/**
+ * Defines the name of the output stream
+ */
+public JoinBolt withOutputStream(String streamName) {
+this.outputStreamName = streamName;
+return this;
+}
+
+/**
+ * Performs inner Join.
+ *  SQL:   from priorStream inner join newStream on newStream.key 
= priorStream.key1
+ *  same as:   new WindowedQueryBolt(priorStream,key1). 
join(newStream, key, priorStream);
+ *
+ *  Note: priorStream must be previously joined.
+ *Valid ex:new WindowedQueryBolt(s1,k1). join(s2,k2, s1). 
join(s3,k3, s2);
+ *Invalid ex:  new WindowedQueryBolt(s1,k1). join(s3,k3, s2). 
join(s2,k2, s1);
+ */
+public JoinBolt join(String newStream, String key, String priorStream) 
{
+return join_common(newStream, key, priorStream, JoinType.INNER);
+}
+
+/**
+ * Performs left Join.
+ *  SQL:   from stream1  left join stream2  on stream2.key = 
stream1.key1
+ *  same as:   new  WindowedQueryBolt(stream1, key1). 
leftJoin(stream2, key, stream1);
+ *
+ *  Note: priorStream must be previously joined
+ *Valid ex:new WindowedQueryBolt(s1,k1). leftJoin(s2,k2, s1). 
leftJoin(s3,k3, s2);
+ *Invalid ex:  new WindowedQueryBolt(s1,k1). leftJoin(s3,k3, s2). 
leftJoin(s2,k2, s1);
+ */
+public JoinBolt leftJoin(String newStream, String key, String 
priorStream) {
+return join_common(newStream, key, priorStream, JoinType.LEFT);
+}
+
+private JoinBolt join_common(String newStream, String key, String 
priorStream, JoinType joinType) {
--- End diff --

indeed.


---
If your project is set up for it, y

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/1914#discussion_r99227816
  
--- Diff: storm-core/src/jvm/org/apache/storm/bolt/JoinBolt.java ---
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.bolt;
+
+
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseWindowedBolt;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TupleWindow;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+
+public class JoinBolt extends BaseWindowedBolt {
+
+private OutputCollector collector;
+
+// Map[StreamName -> Map[Key -> List]  ]
+HashMap >> hashedInputs = new 
HashMap<>(); // holds remaining streams
+
+// Map[StreamName -> JoinInfo]
+protected LinkedHashMap joinCriteria = new 
LinkedHashMap<>();
+protected String[][] outputKeys;  // specified via bolt.select() ... 
used in declaring Output fields
+protected String[] dotSeparatedOutputKeyNames; // flattened (de 
nested) keyNames, used for naming output fields
+protected String outputStreamName;
+
+// Use streamId, source component name OR field in tuple to 
distinguish incoming tuple streams
+public enum Selector { STREAM, SOURCE }
+protected final Selector selectorType;
+
+
+/**
+ * StreamId to start the join with. Equivalent SQL ...
+ *   select  from streamId ...
+ * @param type Specifies whether 'streamId' refers to stream 
name/source component
+ * @param streamId name of stream/source component
+ * @param key the fieldName to use as key for the stream (used for 
performing joins)
+ */
+public JoinBolt(Selector type, String streamId, String key) {
+selectorType = type;
+joinCriteria.put(streamId, new JoinInfo(key) );
+}
+
+/**
+ * Defines the name of the output stream
+ */
+public JoinBolt withOutputStream(String streamName) {
+this.outputStreamName = streamName;
+return this;
+}
+
+/**
+ * Performs inner Join.
+ *  SQL:   from priorStream inner join newStream on newStream.key 
= priorStream.key1
+ *  same as:   new WindowedQueryBolt(priorStream,key1). 
join(newStream, key, priorStream);
+ *
+ *  Note: priorStream must be previously joined.
+ *Valid ex:new WindowedQueryBolt(s1,k1). join(s2,k2, s1). 
join(s3,k3, s2);
--- End diff --

It is necessary to specify which stream s3 should be joined with (s2 or 
s1?)  as the results will differ accordingly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Release Storm 1.1.0

[P. Taylor Goetz]

Thanks for putting this list together Jungtaek. I added a few to the 1.1 
release epic that I think are important. Feel free to do the same.

Looks like we have a few to go, but there are pull requests for them. It’s 
mostly just a matter of reviews and review responses, so I think we are close.

-Taylor

> On Feb 2, 2017, at 1:41 AM, Jungtaek Lim  wrote:
> 
> Seems like there're not blockers for 1.1.0, but some pull requests are
> worth to check.
> There're pending pull requests for storm-kafka-client waited on STORM-2225.
> Given that STORM-2225 is now merged, we might need to take a look at.
> 
> *- reviewing*
> 
> [storm-core]
> 
>> STORM-2324 : Fix deployment failure if resources directory is missing in
> topology jar
> (master) https://github.com/apache/storm/pull/1908
> (1.x) https://github.com/apache/storm/pull/1898
> 
>> STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
> (master) https://github.com/apache/storm/pull/1904
> (1.x) https://github.com/apache/storm/pull/1905
> 
> [storm-kafka]
> 
>> STORM-2270 Kafka spout should consume from latest when ZK partition
> commit offset bigger than the latest offset
> (1.x) https://github.com/apache/storm/pull/1851
> 
> [storm-kafka-client]
> 
>> STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal State
> Exception
> (1.x) https://github.com/apache/storm/pull/1902
> 
>> STORM-2315 Storm kafka client does not commit offsets when ack is disabled
> (1.x) https://github.com/apache/storm/pull/1891
> 
>> fix: KafkaSpout is blocked in AutoCommitMode
> (master) https://github.com/apache/storm/pull/1863
> 
>> STORM-2250: Kafka Spout Refactoring to Increase Modularity and Testability
> (master) https://github.com/apache/storm/pull/1832
> 
>> STORM-2014: Put logic around dropping messages into RetryService, remove
> maxRetry setting from new KafkaSpout
> (master) https://github.com/apache/storm/pull/1605
> 
>> fix NullPointException with acked.get(rtp)
> (master) https://github.com/apache/storm/pull/1807
> 
> [storm-sql]
> 
>> STORM-1443 [Storm SQL] Support customizing parallelism in StormSQL
> https://github.com/apache/storm/pull/1739
> 
> *- pending*
> 
> [storm-kafka-client]
> 
>> STORM-2296 Kafka spout no dup on leader changes
> (1.0.x) https://github.com/apache/storm/pull/1873
> (1.x) https://github.com/apache/storm/pull/1888
> 
> [storm-sql]
> 
>> STORM-2148 [Storm SQL] Trident mode: back to code generate and compile
> Trident topology
> https://github.com/apache/storm/pull/1743
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 2월 2일 (목) 오전 8:14, Harsha Chintalapani 님이 작성:
> 
>> Trying to check the status on this release of 1.1.0. Are we going to do
>> this release anytime soon?
>> 
>> 
>> On Fri, Jan 13, 2017 at 7:50 PM S G  wrote:
>> 
>>> Not sure if its a little late to include for the 1.1.0 and 1.0.3 releases
>>> now, but can we consider using zookeeper 3.4.9 for the future versions as
>>> 3.4.9 brings in a lot of stability improvements (
>>> http://zookeeper.apache.org/releases.html) and storm is still using
>> 3.4.6
>>> (
>>> https://github.com/apache/storm/blob/master/pom.xml)
>>> 
>>> On Fri, Jan 6, 2017 at 11:54 AM, P. Taylor Goetz 
>>> wrote:
>>> 
 Thanks for the update Jungtaek.
 
 I’m verifying the patches now. And they should be mergeable shortly.
 
 I think we’ll likely be ready for 1.1.0 and 1.0.3 releases next week.
 
 -Taylor
 
> On Jan 6, 2017, at 3:40 AM, Jungtaek Lim  wrote:
> 
> I just submitted a patch for STORM-2176
> . Since it's a
>> small
 fix,
> we just need to handle STORM-2228
>  to release.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 1월 5일 (목) 오후 1:43, Jungtaek Lim 님이 작성:
> 
>> Recently we receive some requests regarding release Storm 1.0.3, so
 would
>> like to bump this again.
>> 
>> Given that blocker issues for Storm 1.1.0 are also blocker for
>> 1.0.3,
 I'd
>> like to ask a favor of taking care of 'open' / 'in progress' issues
>> on
>> 1.1.0 epic.
>> https://issues.apache.org/jira/browse/STORM-1856
>> 
>> There're one 'open' issue and three 'in progress' issues. Two of
>> three
 'in
>> progress' issues are tiny fix so easy to be handled, so actual
>>> blockers
 are
>> STORM-2176  and
>> STORM-2228 .
>> 
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>> 
>> 2016년 11월 17일 (목) 오후 11:41, Satish Duggana <
>> satish.dugg...@gmail.com
 님이
>> 작성:
>> 
>>   STORM-2205: Race condition in getting nimbus summaries while ZK
>> connections are reconnected.
>> 
>> This issue seems to occur in our environments and I would like this
>> to
 be
>> part of 1.1.0.
>> 
>> Thanks,
>> Satish.
>> 
>> On Thu, Nov 

[GitHub] storm issue #1851: STORM-2270 Kafka spout should consume from latest when ZK...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1851
  
@danny0405 can you add comments in the TopicOffsetOutOfRangeException block 
on when that case will encounter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1914: STORM-2334 - Join Bolt implementation with unit tests

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/1914
  
@roshannaik Can you add some documentation for this similar to what we have 
for Windowing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1605: STORM-2014: Put logic around dropping messages into Retry...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1605
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1739: STORM-1443 [Storm SQL] Support customizing parallelism in...

[harshach]

Github user harshach commented on the issue:

https://github.com/apache/storm/pull/1739
  
+1. LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1914: STORM-2334 - Join Bolt implementation with unit tests

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/1914
  
@ptgoetz yes ... I was thinking about that myself... thinking of creating a 
Join.md under the top level docs directory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

[raghavgautam]

Github user raghavgautam commented on a diff in the pull request:

https://github.com/apache/storm/pull/1914#discussion_r99237976
  
--- Diff: storm-core/src/jvm/org/apache/storm/bolt/JoinBolt.java ---
@@ -0,0 +1,428 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.bolt;
+
+
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseWindowedBolt;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.windowing.TupleWindow;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+
+public class JoinBolt extends BaseWindowedBolt {
+
+private OutputCollector collector;
+
+// Map[StreamName -> Map[Key -> List]  ]
+HashMap >> hashedInputs = new 
HashMap<>(); // holds remaining streams
+
+// Map[StreamName -> JoinInfo]
+protected LinkedHashMap joinCriteria = new 
LinkedHashMap<>();
+protected String[][] outputKeys;  // specified via bolt.select() ... 
used in declaring Output fields
+protected String[] dotSeparatedOutputKeyNames; // flattened (de 
nested) keyNames, used for naming output fields
+protected String outputStreamName;
+
+// Use streamId, source component name OR field in tuple to 
distinguish incoming tuple streams
+public enum Selector { STREAM, SOURCE }
+protected final Selector selectorType;
+
+
+/**
+ * StreamId to start the join with. Equivalent SQL ...
+ *   select  from streamId ...
+ * @param type Specifies whether 'streamId' refers to stream 
name/source component
+ * @param streamId name of stream/source component
+ * @param key the fieldName to use as key for the stream (used for 
performing joins)
+ */
+public JoinBolt(Selector type, String streamId, String key) {
+selectorType = type;
+joinCriteria.put(streamId, new JoinInfo(key) );
+}
+
+/**
+ * Defines the name of the output stream
+ */
+public JoinBolt withOutputStream(String streamName) {
+this.outputStreamName = streamName;
+return this;
+}
+
+/**
+ * Performs inner Join.
+ *  SQL:   from priorStream inner join newStream on newStream.key 
= priorStream.key1
+ *  same as:   new WindowedQueryBolt(priorStream,key1). 
join(newStream, key, priorStream);
+ *
+ *  Note: priorStream must be previously joined.
+ *Valid ex:new WindowedQueryBolt(s1,k1). join(s2,k2, s1). 
join(s3,k3, s2);
--- End diff --

Discussed this offline with @roshannaik , this is necessary when we have 
left outer join. Consider a join of tables T1, T2, T3. If join of T1 and T2 is 
left outer join it can lead to intermediate table which looks like:

T1.K1| T2.K1
-|---
 Y | NULL 
Now consider the case, when we have to do inner join with T3.K1 (with value 
Y) the resulting table might change depending on whether it is join on T1.K1 or 
T2.K1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1904: STORM-2321 Handle blobstore zk key deletion in Key...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1904


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1842: Merge remote-tracking branch 'refs/remotes/apache/...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1842


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1898: STORM 2324 - Fix toplogy deployment failure if resources ...

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/1898
  
ok only 1 test needed fix... the others storm-core test are ok ... were 
failing  only if I ran within IDE due to relative paths issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1909: Add some scripts for help verifying the RC

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1909


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1915: STORM-2337: Broken documentation generation for storm-met...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1915
  
+1 Nice catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1743: STORM-2148 [Storm SQL] Trident mode: back to code ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1743


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Apache Storm 1.0.3 (rc1)

[Xin Wang]

+1

Ran some example topologies in 3 node cluster.

- Xin Wang (vesense)

2017-02-03 4:54 GMT+08:00 Alexandre Vermeerbergen 
:

> Hello all,
>
> Using storm 1.0.3rc1 binaries artifacts, I installed our supervision system
> (currently made of 11 topologies), with java 8 update 92 on CentOS 6.5 ;
> with a cluster type of install (2 big VMs with Supervisors, another VM with
> Nimbus, and other VMs for Zookeeper nodes).
>
> So far so good => +1 (non binding)
>
> It's been running for only 1 hour, I may update if I see issues in the
> coming days. I'll pay attention to stability & performance (too early to
> tell).
>
> Thanks,
>
> Alexandre Vermeerbergen
>
>
>
> 2017-01-31 21:55 GMT+01:00 P. Taylor Goetz :
>
> > This is a call to vote on releasing Apache Storm 1.0.3 (rc1)
> >
> > Full list of changes in this release:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > 6cb735d18ec11eccd3e80bab8f879c989a9b3967;hb=
> a81ec2580fce1f2ee6349a9028dcb7
> > 5763798bec
> >
> > The tag/commit to be voted upon is v1.0.3:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=CHANGELOG.md;h=b8adddc6aa20107288e59cba6a2976c0951742fb;hb=
> > a81ec2580fce1f2ee6349a9028dcb75763798bec
> >
> > The source archive being voted upon can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.3-rc1/apache-storm-1.0.3-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1043
> >
> > Please vote on releasing this package as Apache Storm 1.0.3.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.0.3
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>

[GitHub] storm pull request #1739: STORM-1443 [Storm SQL] Support customizing paralle...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1739


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1851: STORM-2270 Kafka spout should consume from latest ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1851#discussion_r99265466
  
--- Diff: 
external/storm-kafka/src/jvm/org/apache/storm/kafka/PartitionManager.java ---
@@ -199,7 +199,12 @@ private void fill() {
 try {
 msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, 
_partition, offset);
 } catch (TopicOffsetOutOfRangeException e) {
-offset = KafkaUtils.getOffset(_consumer, _partition.topic, 
_partition.partition, kafka.api.OffsetRequest.EarliestTime());
+long partitionLatestOffset = KafkaUtils.getOffset(_consumer, 
_partition.topic, _partition.partition, kafka.api.OffsetRequest.LatestTime());
+if (partitionLatestOffset < offset) {
+offset = partitionLatestOffset;
+} else {
+offset = KafkaUtils.getOffset(_consumer, _partition.topic, 
_partition.partition, kafka.api.OffsetRequest.EarliestTime());
+}
 // fetch failed, so don't update the fetch metrics
 
 //fix bug [STORM-643] : remove outdated failed offsets
--- End diff --

@danny0405 @harshach 
Sorry My bad. I found that I confused earliest and latest... We're good to 
go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1851: STORM-2270 Kafka spout should consume from latest when ZK...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1851
  
+1 Thanks for the great contribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1908: STORM-2324 : Fix deployment failure if resources director...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1908
  
+1 UT passed from local dev.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1898: STORM 2324 - Fix toplogy deployment failure if resources ...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1898
  
+1 UT passed from local dev.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1908: STORM-2324 : Fix deployment failure if resources d...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1908


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1898: STORM 2324 - Fix toplogy deployment failure if res...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1898


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1851: STORM-2270 Kafka spout should consume from latest when ZK...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1851
  
@harshach I found my mistake, and now I'm +1 to merge. Do you want to wait 
@danny0405 for adding comments?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1605: STORM-2014: Put logic around dropping messages into Retry...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1605
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1916: fixed storm-kafka-client test failed

[lukess]

GitHub user lukess opened a pull request:

https://github.com/apache/storm/pull/1916

fixed storm-kafka-client test failed

in order to finish tests for storm-kafka-client

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukess/storm 1.x-branch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1916


commit 0b0487be179515e2b1210aa39b71b203be6fb347
Author: lukess 
Date:   2017-02-03T02:43:50Z

fixed storm-kafka-client test failed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1605: STORM-2014: Put logic around dropping messages int...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1605


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1832: STORM-2250: Kafka Spout Refactoring to Increase Mo...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1832#discussion_r99269455
  
--- Diff: external/storm-kafka-client/pom.xml ---
@@ -52,7 +52,6 @@
 
 org.apache.kafka
 kafka-clients
-${storm.kafka.client.version}
--- End diff --

What's behind of the change? We're maintaining both `storm.kafka.version` 
and `storm.kafka.client.version`, and this line is for picking correct version.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1888: STORM-2296 Kafka spout no dup on leader changes, 1...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1888#discussion_r99283845
  
--- Diff: 
external/storm-kafka/src/test/org/apache/storm/kafka/ZkCoordinatorTest.java ---
@@ -114,6 +113,33 @@ public void testPartitionsChange() throws Exception {
 }
 }
 
+@Test
+public void testPartitionManagerRecreate() throws Exception {
+final int totalTasks = 2;
+int partitionsPerTask = 2;
+List coordinatorList = buildCoordinators(totalTasks 
/ partitionsPerTask);
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9092)));
+List> partitionManagersBeforeRefresh = 
getPartitionManagers(coordinatorList);
+waitForRefresh();
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9093)));
+List> partitionManagersAfterRefresh = 
getPartitionManagers(coordinatorList);
+assertEquals(partitionManagersAfterRefresh.size(), 
partitionManagersAfterRefresh.size());
--- End diff --

It compares exactly same thing. Did you miss here, or it's redundant line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1888: STORM-2296 Kafka spout no dup on leader changes, 1...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1888#discussion_r99284800
  
--- Diff: 
external/storm-kafka/src/test/org/apache/storm/kafka/ZkCoordinatorTest.java ---
@@ -114,6 +113,33 @@ public void testPartitionsChange() throws Exception {
 }
 }
 
+@Test
+public void testPartitionManagerRecreate() throws Exception {
+final int totalTasks = 2;
+int partitionsPerTask = 2;
+List coordinatorList = buildCoordinators(totalTasks 
/ partitionsPerTask);
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9092)));
+List> partitionManagersBeforeRefresh = 
getPartitionManagers(coordinatorList);
+waitForRefresh();
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9093)));
+List> partitionManagersAfterRefresh = 
getPartitionManagers(coordinatorList);
+assertEquals(partitionManagersAfterRefresh.size(), 
partitionManagersAfterRefresh.size());
+Iterator> iterator = 
partitionManagersAfterRefresh.iterator();
+for (List partitionManagersBefore : 
partitionManagersBeforeRefresh) {
+List partitionManagersAfter = 
iterator.next();
+for (PartitionManager before : partitionManagersBefore)
--- End diff --

I'd rather create a new hash map for before or after which key is 
partition, and compare based on the map and other list. The depth of `if` 
statement is already 4.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1888: STORM-2296 Kafka spout no dup on leader changes, 1...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1888#discussion_r99284895
  
--- Diff: 
external/storm-kafka/src/test/org/apache/storm/kafka/ZkCoordinatorTest.java ---
@@ -114,6 +113,33 @@ public void testPartitionsChange() throws Exception {
 }
 }
 
+@Test
+public void testPartitionManagerRecreate() throws Exception {
+final int totalTasks = 2;
+int partitionsPerTask = 2;
+List coordinatorList = buildCoordinators(totalTasks 
/ partitionsPerTask);
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9092)));
+List> partitionManagersBeforeRefresh = 
getPartitionManagers(coordinatorList);
+waitForRefresh();
+
when(reader.getBrokerInfo()).thenReturn(TestUtils.buildPartitionInfoList(TestUtils.buildPartitionInfo(totalTasks,
 9093)));
+List> partitionManagersAfterRefresh = 
getPartitionManagers(coordinatorList);
+assertEquals(partitionManagersAfterRefresh.size(), 
partitionManagersAfterRefresh.size());
+Iterator> iterator = 
partitionManagersAfterRefresh.iterator();
+for (List partitionManagersBefore : 
partitionManagersBeforeRefresh) {
+List partitionManagersAfter = 
iterator.next();
+for (PartitionManager before : partitionManagersBefore)
+for (PartitionManager after: partitionManagersAfter)
+if (before.getPartition().partition == 
after.getPartition().partition)
+assertStateIsTheSame(before, after);
+}
+}
+
+private void assertStateIsTheSame(PartitionManager managerBefore, 
PartitionManager managerAfter) {
+// check if state was actually moved from old PartitionManager
+assertNotSame(managerBefore, managerAfter);
+assertSame(managerBefore._waitingToEmit, 
managerAfter._waitingToEmit);
--- End diff --

Let's check all the copying fields, not only _waitingToEmit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1888: STORM-2296 Kafka spout no dup on leader changes, 1.x-bran...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1888
  
@ernisv 
Looks good. Minor nits. Build failing is unrelated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---