[GitHub] storm issue #2152: STORM-2639: Kafka Spout incorrectly computes numCommitted...
Github user askprasanna commented on the issue: https://github.com/apache/storm/pull/2152 Created a new JIRA and updated PR title as recommended. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...
Github user vesense commented on the issue: https://github.com/apache/storm/pull/2220 Looks good to me. It would be better if we can apply new code style to the changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user liu-zhaokun commented on the issue: https://github.com/apache/storm/pull/2208 @HeartSaVioR I have modified this PR according to your opinion,and it has passed all the test.Could you help me merge it?Thank you very much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Connection refused
Make sure the nimbus is running properly. Sent from my iPhone > On 18 Jul 2017, at 09:16, sam mohel wrote: > > i'm facing problem with submitting topology in distributed mode > storm-0.10.2 zookeeper-3.4.6 > > Exception in thread "main" java.lang.RuntimeException: > org.apache.thrift7.transport.TTransportException: > java.net.ConnectException: Connection refused (Connection refused) > at > backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59) > at > backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51) > at > backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) > at backtype.storm.security.auth.ThriftClient.(ThriftClient.java:72) > at backtype.storm.utils.DRPCClient.(DRPCClient.java:44) > > what should i check for fixing this problem ?
Connection refused
i'm facing problem with submitting topology in distributed mode storm-0.10.2 zookeeper-3.4.6 Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) at backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59) at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51) at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) at backtype.storm.security.auth.ThriftClient.(ThriftClient.java:72) at backtype.storm.utils.DRPCClient.(DRPCClient.java:44) what should i check for fixing this problem ?
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user liu-zhaokun commented on the issue: https://github.com/apache/storm/pull/2208 @HeartSaVioR Thanks for your reply.I will remove it from Config.java. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2208 I think the best way to avoid misleading is just removing the configuration and set the value as constant. We don't need to provide an option to user while the value is actually fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user liu-zhaokun commented on the issue: https://github.com/apache/storm/pull/2208 @harshach Thanks for your reply.You can see StormSubmitter.java,line 91.STORM_ZOOKEEPER_TOPOLOGY_AUTH_SCHEME is should always be set to digest.It can't be and won't be other value.So,I think we should declare â"storm.zookeeper.topology.auth.schemeâ should always be set to digest.We shouldn't mislead others. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2219 Close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...
Github user Ethanlm closed the pull request at: https://github.com/apache/storm/pull/2219 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/2152 @askprasanna let's just create a new JIRA and associate it with the STORM-2505 JIRA - it's a quick thing and will take care of this clean. Another minor detail we have a script to find orphan pull requests, malformed pull requests, etc. The pull request description `[STORM-2505] OffsetManager...` does not conform with the name convention. Can you please rename it to follow the pattern `STORM-2505: OffsetManager...`. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2152 @askprasanna Maybe we would need to associate this to a new issue to track your effort (given that assignee for STORM-2505 is not you and it can't be overwritten), or remove the issue at all. Except test code I think it's fairly small change which doesn't strictly require filing an issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2219 Thanks. I will submit a patch against 1.x branch. I am not very familiar with logviewer clojure implementation yet. But I will try and see what I can do. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2219: [STORM-2637] fix ClassCastException in logviewer get-log-...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2219 Thanks for the patch. I just would like to see this patch against 1.x branch rather than master, given that we should review and apply #2204 to remove Clojure implementation. Btw, if you had (or have) a chance to read logviewer clojure implementation, I would like to ask a favor with reviewing #2204. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/2219#discussion_r127842992 --- Diff: storm-core/src/clj/org/apache/storm/daemon/logviewer.clj --- @@ -311,7 +311,7 @@ (defn get-log-user-group-whitelist [fname] (let [wl-file (ServerConfigUtils/getLogMetaDataFile fname) -m (clojurify-structure (Utils/readYamlFile wl-file))] +m (clojurify-structure (Utils/readYamlFile (.toString wl-file)))] --- End diff -- Maybe better to use `getAbsolutePath` or `getCanonicalPath` to clearly represent path. I used `getAbsolutePath` from #2204 but I can change it if others claim that `getCanonicalPath` is more preferred. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2147: STORM-2538: New kafka spout emits duplicate tuples
Github user hmcl closed the pull request at: https://github.com/apache/storm/pull/2147 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/2147 Closed in favor of #2151 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2147 @hmcl @srdo I don't think we need this given this PR https://github.com/apache/storm/pull/2151 makes manual assignment as default. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2151: STORM-2542: Remove KafkaConsumer.subscribe API option, ma...
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2151 +1. Thanks @srdo this looks great. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2155: STORM-2548: Simplify KafkaSpoutConfig to avoid duplicatin...
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2155 still +1 . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2217: [1.x-branch] [STORM-2505] OffsetManager should account fo...
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2217 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2215: STORM-2548: Simplify KafkaSpoutConfig (1.x)
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2215 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2215: STORM-2548: Simplify KafkaSpoutConfig (1.x)
Github user harshach commented on a diff in the pull request: https://github.com/apache/storm/pull/2215#discussion_r127821694 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/SerializableDeserializer.java --- @@ -16,10 +16,14 @@ package org.apache.storm.kafka.spout; import java.io.Serializable; +import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.common.serialization.Deserializer; /** * @param The type this deserializer deserializes to. + * @deprecated Avoid using this class. Use {@link KafkaSpoutConfig.Builder#setProp(java.lang.String, java.lang.Object) } with + * {@link ConsumerConfig#KEY_DESERIALIZER_CLASS_CONFIG} and {@link ConsumerConfig#VALUE_DESERIALIZER_CLASS_CONFIG} instead */ -public interface SerializableDeserializer extends Deserializer, Serializable { +@Deprecated --- End diff -- why are we adding this back? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2147 @hmcl I am strongly in favor of option 1, since no one has provided a use case for/argued in favor of the subscribe API based partition assignment code. Ideally we'll deprecate (1.x) and remove (2.x) subscribe API support as well, rather than leaving it as an unsupported option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2208 @liu-zhaokun I think the comment there meant to say by default it will be "No Authentication". I.e Its users responsibility to set to digest in a secure clusters. But since the default settings for non-secure the comment looks ok to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...
Github user roshannaik commented on the issue: https://github.com/apache/storm/pull/2218 Sounds like a checkpoint mechanism limited to windowed bolts. Could you provide a design document for this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2220: STORM-2638: Add in maximum object size for Pacemak...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2220 STORM-2638: Add in maximum object size for Pacemaker Thrift objects You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2638 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2220.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2220 commit e6465b5dc5c7e31e0b92a18de578a13d05a5abf2 Author: Kyle Nusbaum Date: 2017-07-17T19:40:15Z Adding in maximum/minimum object size --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2209: [STORM-2622] Add owner resource summary on storm UI
Github user harshach commented on the issue: https://github.com/apache/storm/pull/2209 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2219: [STORM-2637] fix ClassCastException in logviewer g...
GitHub user Ethanlm opened a pull request: https://github.com/apache/storm/pull/2219 [STORM-2637] fix ClassCastException in logviewer get-log-user-group-whitelist func⦠In get-log-user-group-whitelist function, the getLogMetaDataFile function returns a File while the readYamlFile function requires a String. see https://issues.apache.org/jira/browse/STORM-2637 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Ethanlm/storm STORM-2637 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2219.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2219 commit f636fdbab9aec58bc27da3b5b8c7a1110266c05a Author: Ethan Li Date: 2017-07-17T19:02:21Z fix ClassCastException in logviewer get-log-user-group-whitelist function --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2147: STORM-2538: New kafka spout emits duplicate tuples
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/2147 @srdo @harshach @arunmahadevan in face of the Manual Partition assignment related [fixes](https://github.com/apache/storm/pull/2151/commits), there are two options. I believe in the email list we agreed to go with option #1, but I would like to make sure. 1 - We disregard and close this PR. If we do so, we are implicitly telling the user that if they decide to use dynamic partition assignment they are at their own responsibility and it won't work with multiple parallelism. Currently the only implementation we support is `ManualPartitionSubscription` 2 - We improve this code to make it work for the dynamic case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2151: STORM-2542: Remove KafkaConsumer.subscribe API opt...
Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/2151#discussion_r127773597 --- Diff: docs/storm-kafka-client.md --- @@ -240,12 +240,9 @@ streams. If you are doing this for Trident a value must be in the List returned otherwise trident can throw exceptions. -### Manual Partition Control (ADVANCED) +### Manual Partition Assigment (ADVANCED) -By default Kafka will automatically assign partitions to the current set of spouts. It handles lots of things, but in some cases you may want to manually assign the partitions. -This can cause less churn in the assignments when spouts go down and come back up, but it can result in a lot of issues if not done right. This can all be handled by subclassing -Subscription and we have a few implementations that you can look at for examples on how to do this. ManualPartitionNamedSubscription and ManualPartitionPatternSubscription. Again -please be careful when using these or implementing your own. +By default the KafkaSpout instancs will be assigned partitions by round robin assignment. If you need to customize this assignment, you can implement the `ManualPartitioner` interface. The implementation can be passed to the `ManualPartitionSubscription` constructor, and the `Subscription` can then be set in the `KafkaSpoutConfig` via the `KafkaSpoutConfig.Builder` constructor. Please take care when supplying a custom implementation, since an incorrect `ManualPartitioner` implementation could leave some partitions unread, or concurrently read by multiple spout instances. See the `RoundRobinManualPartitioner` for an example of how to implement this functionality. --- End diff -- By default the KafkaSpout instances will be assigned partitions using a round robin strategy. If you need to customize partitions assignment, you must implement the `ManualPartitioner` interface. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2150: STORM-2541: Fix storm-kafka-client manual subscrip...
Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/2150#discussion_r127769417 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicFilter.java --- @@ -0,0 +1,38 @@ +/* + * Copyright 2017 The Apache Software Foundation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.storm.kafka.spout; + +import java.io.Serializable; +import java.util.List; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.common.TopicPartition; + +public interface TopicFilter extends Serializable { + +/** + * Get the Kafka TopicPartitions passed by this filter. + * @param consumer The Kafka consumer to use to read the list of existing partitions + * @return The Kafka partitions passed by this filter. + */ +List getFilteredTopicPartitions(KafkaConsumer consumer); + +/** + * @return A human-readable string representing the subscribed topics. --- End diff -- NIT: representing the topics that pass the filter --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2150: STORM-2541: Fix storm-kafka-client manual subscription no...
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/2150 +1. @harshach @priyank5485 can you please take one final look. If you don't have any objection, I suggest that we merge this patch in the next day or so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2155: STORM-2548: Simplify KafkaSpoutConfig to avoid duplicatin...
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/2155 +1. @harshach @priyank5485 can you please take one final look. If you don't have any objection, I suggest that we merge this patch in the next day or so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...
GitHub user arunmahadevan opened a pull request: https://github.com/apache/storm/pull/2218 STORM-2614: Enhance stateful windowing to persist the window state Right now the tuples in window are stored in memory. This limits the usage to windows that fit in memory. Also the source tuples cannot be acked until the window expiry. By persisting the window transparently in the state backend and caching/iterating them as needed, we could support larger windows and also support windowed bolts with user/application state. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arunmahadevan/storm STORM-2614 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2218 commit 376d3120a5075d9687f63a587dca5e040a5aa6eb Author: Arun Mahadevan Date: 2017-06-28T16:13:26Z STORM-2614: Enhance stateful windowing to persist the window state Right now the tuples in window are stored in memory. This limits the usage to windows that fit in memory. Also the source tuples cannot be acked until the window expiry. By persisting the window transparently in the state backend and caching/iterating them as needed, we could support larger windows and also support windowed bolts with user/application state. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2199: [STORM-2201] Add dynamic scheduler configuration loading
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2199 Hi @HeartSaVioR @revans2, could you please review this when you get a chance? Thanks very much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2209: [STORM-2622] Add owner resource summary on storm UI
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2209 Added some unit tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2152: [STORM-2505] OffsetManager should account for offset void...
Github user askprasanna commented on the issue: https://github.com/apache/storm/pull/2152 Squashed the commits. Also opened https://github.com/apache/storm/pull/2217 for 1.x branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2217: [1.x-branch] [STORM-2505] OffsetManager should acc...
GitHub user askprasanna opened a pull request: https://github.com/apache/storm/pull/2217 [1.x-branch] [STORM-2505] OffsetManager should account for offset voids when retur⦠â¦ning the count of committed offsets You can merge this pull request into a Git repository by running: $ git pull https://github.com/askprasanna/storm 1.x-2505 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2217 commit b661c3b2f0ef529ca33c3ba142f9440f39b8ef4f Author: Prasanna Ranganathan Date: 2017-06-07T12:33:09Z [STORM-2505] OffsetManager should account for offset voids when returning the count of committed offsets --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2152: [STORM-2505] OffsetManager should account for offs...
Github user askprasanna commented on a diff in the pull request: https://github.com/apache/storm/pull/2152#discussion_r127695589 --- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/KafkaSpoutCommitTest.java --- @@ -0,0 +1,146 @@ +/* + * Copyright 2017 The Apache Software Foundation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.storm.kafka.spout; + +import static org.apache.storm.kafka.spout.builders.SingleTopicKafkaSpoutConfiguration.getKafkaSpoutConfigBuilder; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.mockito.Matchers.*; +import static org.mockito.Mockito.*; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.apache.kafka.clients.consumer.*; +import org.apache.kafka.common.TopicPartition; +import org.apache.storm.kafka.spout.builders.SingleTopicKafkaSpoutConfiguration; +import org.apache.storm.kafka.spout.internal.KafkaConsumerFactory; +import org.apache.storm.spout.SpoutOutputCollector; +import org.apache.storm.task.TopologyContext; +import org.apache.storm.utils.Time; +import org.apache.storm.utils.Time.SimulatedTime; +import org.junit.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.Captor; +import org.mockito.InOrder; +import org.mockito.MockitoAnnotations; + +public class KafkaSpoutCommitTest { + +private final long offsetCommitPeriodMs = 2_000; +private final TopologyContext contextMock = mock(TopologyContext.class); +private final SpoutOutputCollector collectorMock = mock(SpoutOutputCollector.class); +private final Map conf = new HashMap<>(); +private final TopicPartition partition = new TopicPartition(SingleTopicKafkaSpoutConfiguration.TOPIC, 1); +private KafkaConsumer consumerMock; +private KafkaSpout spout; +private KafkaSpoutConfig spoutConfig; + +@Captor +private ArgumentCaptor> commitCapture; + +private void setupSpout(Set assignedPartitions) { +MockitoAnnotations.initMocks(this); +spoutConfig = getKafkaSpoutConfigBuilder(-1) +.setOffsetCommitPeriodMs(offsetCommitPeriodMs) +.build(); + +consumerMock = mock(KafkaConsumer.class); +KafkaConsumerFactory consumerFactory = (kafkaSpoutConfig) -> consumerMock; + +//Set up a spout listening to 1 topic partition +spout = new KafkaSpout<>(spoutConfig, consumerFactory); + +spout.open(conf, contextMock, collectorMock); +spout.activate(); + +ArgumentCaptor rebalanceListenerCapture = ArgumentCaptor.forClass(ConsumerRebalanceListener.class); +verify(consumerMock).subscribe(anyCollection(), rebalanceListenerCapture.capture()); + +//Assign partitions to the spout +ConsumerRebalanceListener consumerRebalanceListener = rebalanceListenerCapture.getValue(); +consumerRebalanceListener.onPartitionsAssigned(assignedPartitions); +} + +@Test +public void testCommitSuccessWithOffsetVoids() { +//Verify that the commit logic can handle offset voids +try (SimulatedTime simulatedTime = new SimulatedTime()) { +setupSpout(Collections.singleton(partition)); +Map>> records = new HashMap<>(); +List> recordsForPartition = new ArrayList<>(); +// Offsets emitted are 0,1,2,3,4,...,8,9 --- End diff -- will tweak the comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Release Planning for 1.1.1 and others?
Hi devs, We released Storm 1.1.0 at the late March, and we didn't have short plan for next release. (We discussed Storm 2.0.0, and I think we're getting closer, a bit slowly but steady.) While we still have some issues and pull requests for storm-kafka-client which I think should be included to 1.0.1, but maybe we can start planning 1.1.1, having epic issue for release as we did for prior releases, and also other versions as well (1.2.0 or 2.0.0). Unless I'm missing something, looks like there're no critical bugs except storm-kafka-client for 1.1.1 candidate. So I think we could roll out 1.1.1 fairly soon after sorting out current issue on storm-kafka-client. I'm not familiar with storm-kafka-client module hence I'd like to rely on Hugo or Stig or other committers/contributors to sort out, but may spend some time to be familiar and participate if its progress is going to be slow. Looking forward to hear your opinions. Thanks, Jungtaek Lim (HeartSaVioR)
[GitHub] storm issue #2208: [STORM-2627] The annotation of storm.zookeeper.topology.a...
Github user liu-zhaokun commented on the issue: https://github.com/apache/storm/pull/2208 @haitaoyao I am so sorry to bother you.Do you have time to help me review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2207: [STORM-2626] Provided a template for drpc-auth-acl.yaml
Github user liu-zhaokun commented on the issue: https://github.com/apache/storm/pull/2207 @harshach I am so sorry to bother you.Do you have time to help me review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---