[GitHub] storm issue #1891: STORM-2315 Storm kafka client does not commit offsets whe...

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1891 @liurenjie1024 I am not sure how this code changes will help though. If acking is not enabled adding auto.commit.enable to true on producer properties should solve this right? --- If your project

problem in running topology in local mode

2017-01-23 Thread sam mohel
I got this error when i ran my topology in local mode clojure.tools.logging$eval1$fn__7 invoke SEVERE: java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V at org.apache.log4j.Categor

[GitHub] storm pull request #1881: STORM-2298 Don't kill Nimbus when ClusterMetricsCo...

2017-01-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1881 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: Local BlobStore and Nimbus H/A

2017-01-23 Thread Jungtaek Lim
More details: there were three Nimbuses N1, N2, N3 and N1 was the leader. We submitted topology T1 and after submission we restarted N1. N2 got leader and we killed T1. While N1 is initializing and syncing up its topology blobs, N2 concurrently removes the ZK path and also max sequence number path

[GitHub] storm pull request #1885: STORM-2301 [storm-cassandra] upgrade cassandra dri...

2017-01-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1885 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] storm issue #1832: STORM-2250: Refactor new Kafka spout so it is easier to r...

2017-01-23 Thread hmcl
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1832 @srdo What about calling the JIRA/PR Summary/Title: **Kafka Spout Refactoring to Increase Modularity and Testability** --- If your project is set up for it, you can reply to this email and have your r

[GitHub] storm issue #1845: STORM-2320: DRPC client printer class reusable for local ...

2017-01-23 Thread hmcl
Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1845 @ptgoetz I have filed a [JIRA](https://issues.apache.org/jira/browse/STORM-2320) which describes the motivation and necessity for this patch. Let's let me know if you need further clarification. ---

[GitHub] storm issue #1891: STORM-2315 Storm kafka client does not commit offsets whe...

2017-01-23 Thread liurenjie1024
Github user liurenjie1024 commented on the issue: https://github.com/apache/storm/pull/1891 @harshach Yes that's the case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] storm pull request #1851: STORM-2270 Kafka spout should consume from latest ...

2017-01-23 Thread HeartSaVioR
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/1851#discussion_r97461150 --- Diff: external/storm-kafka/src/jvm/org/apache/storm/kafka/PartitionManager.java --- @@ -199,7 +199,12 @@ private void fill() { try {

[GitHub] storm pull request #1851: STORM-2270 Kafka spout should consume from latest ...

2017-01-23 Thread harshach
Github user harshach commented on a diff in the pull request: https://github.com/apache/storm/pull/1851#discussion_r97459031 --- Diff: external/storm-kafka/src/jvm/org/apache/storm/kafka/PartitionManager.java --- @@ -199,7 +199,12 @@ private void fill() { try {

[GitHub] storm issue #1881: STORM-2298 Don't kill Nimbus when ClusterMetricsConsumer ...

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1881 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm issue #1868: STORM-2225: change spout config to be simpler. (1.x)

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1868 +1. @revans2 can you squash the commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] storm issue #1891: STORM-2315 Storm kafka client does not commit offsets whe...

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1891 @liurenjie1024 If I understand correctly this bug happens when user runs topology.num.ackers=0 ? --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] storm issue #1885: STORM-2301 [storm-cassandra] upgrade cassandra driver to ...

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1885 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] storm issue #1885: STORM-2301 [storm-cassandra] upgrade cassandra driver to ...

2017-01-23 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1885 @harshach @ptgoetz Gently ping sponsors for reviewing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not hav

[GitHub] storm issue #1739: STORM-1443 [Storm SQL] Support customizing parallelism in...

2017-01-23 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1739 Rebased onto master since STORM-1446 is merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] storm issue #1743: STORM-2148 [Storm SQL] Trident mode: back to code generat...

2017-01-23 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1743 Rebased onto master since STORM-1446 is merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does no

[GitHub] storm pull request #1889: STORM-2305: STORM-2279 calculates task index diffe...

2017-01-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1889 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] storm pull request #1736: STORM-1446 [Storm SQL] Compile the Calcite logical...

2017-01-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1736 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97418597 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/BranchExample.java --- @@ -0,0 +1,72 @@ +/** + * Licensed to the Apache Sof

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97423786 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/StatefulWordCount.java --- @@ -0,0 +1,87 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97418547 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/BranchExample.java --- @@ -0,0 +1,72 @@ +/** + * Licensed to the Apache Sof

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97423628 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/StatefulWordCount.java --- @@ -0,0 +1,87 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97417642 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/AggregateExample.java --- @@ -0,0 +1,96 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97424063 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/TypedTupleExample.java --- @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97422636 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/JoinExample.java --- @@ -0,0 +1,110 @@ +/** + * Licensed to the Apache Soft

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97427695 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/StateQueryExample.java --- @@ -0,0 +1,110 @@ +/** + * Licensed to the Apach

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97423227 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/StateQueryExample.java --- @@ -0,0 +1,110 @@ +/** + * Licensed to the Apach

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97423023 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/StateQueryExample.java --- @@ -0,0 +1,110 @@ +/** + * Licensed to the Apach

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97424682 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/WordCountToBolt.java --- @@ -0,0 +1,106 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97419171 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/GroupByKeyAndWindowExample.java --- @@ -0,0 +1,109 @@ +/** + * Licensed to

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97424356 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/TypedTupleExample.java --- @@ -0,0 +1,64 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97420280 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/GroupByKeyAndWindowExample.java --- @@ -0,0 +1,109 @@ +/** + * Licensed to

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97420102 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/GroupByKeyAndWindowExample.java --- @@ -0,0 +1,109 @@ +/** + * Licensed to

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r9741 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/GroupByKeyAndWindowExample.java --- @@ -0,0 +1,109 @@ +/** + * Licensed to

[GitHub] storm pull request #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1693#discussion_r97424491 --- Diff: examples/storm-starter/src/jvm/org/apache/storm/starter/streams/WindowedWordCount.java --- @@ -0,0 +1,80 @@ +/** + * Licensed to the Apache

[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

2017-01-23 Thread srdo
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1808#discussion_r97367473 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java --- @@ -1,16 +1,35 @@ +/* + * Licensed to

[GitHub] storm issue #1736: STORM-1446 [Storm SQL] Compile the Calcite logical plan t...

2017-01-23 Thread harshach
Github user harshach commented on the issue: https://github.com/apache/storm/pull/1736 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Thrift Access Logger deacivation

2017-01-23 Thread tsantalos-spamia
 Does anyone know how to deactivate the ThriftAccessLogger for localhost mode (storm 1.0.2)?? It constantly shows debugging information that makes debugging impossible.

[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1808#discussion_r97334512 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -447,7 +440,10 @@ private void shutdown() {

[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1808#discussion_r97333183 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/bolt/selector/FieldNameTopicSelector.java --- @@ -0,0 +1,49 @@ +/** + * Li

[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1808#discussion_r97331194 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/TopicPartitionComparator.java --- @@ -1,16 +1,35 @@ +/* + * Licensed

[GitHub] storm pull request #1808: STORM-2225: change spout config to be simpler.

2017-01-23 Thread revans2
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1808#discussion_r97332427 --- Diff: external/storm-kafka/src/jvm/org/apache/storm/kafka/bolt/KafkaBolt.java --- @@ -51,7 +51,9 @@ * This bolt uses 0.8.2 Kafka Producer API.

Re: Local BlobStore and Nimbus H/A

2017-01-23 Thread Bobby Evans
HA for the blobstore was set up so that ZK would hold the source of truth and then the other nimbus nodes would be eventually consistent with each other.  I'm not totally sure of the issue, because I don't understand if this is happening in the context of a follower trying to keep up to date, or

[GitHub] storm issue #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread revans2
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1693 @arunmahadevan I will try to find time to take a look at it soon. Sorry for the delay. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] storm issue #1693: [STORM-1961] Stream api for storm core use cases

2017-01-23 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/1693 @revans2 @ptgoetz pinging again for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] storm issue #1889: STORM-2305: STORM-2279 calculates task index different fr...

2017-01-23 Thread arunmahadevan
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/1889 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

Re: simple question about grouping

2017-01-23 Thread Arun Mahadevan
Grouping makes sense only when you have more than one task for a bolt. If your bolt has more than one task, then the grouping will decide how the tuples from the spout are distributed to the individual tasks of the bolt. (shuffe = random, fields = keyed on some field and so on). See http

Re: simple question about grouping

2017-01-23 Thread sam mohel
i hope someone can help . why the group is inactive ? On Mon, Jan 23, 2017 at 11:39 AM, sam mohel wrote: > i have text file contains data . size of this file is 3.5 MB . My topology > consists of one spout and one bolt so is that possible to make all > processing in one bolt and in this case wha

[GitHub] storm issue #1736: STORM-1446 [Storm SQL] Compile the Calcite logical plan t...

2017-01-23 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1736 Addressed the change with master: update to Calcite 1.11.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

simple question about grouping

2017-01-23 Thread sam mohel
i have text file contains data . size of this file is 3.5 MB . My topology consists of one spout and one bolt so is that possible to make all processing in one bolt and in this case what is the role of grouping here ? Thanks in advance

[GitHub] storm pull request #1891: STORM-2315 Storm kafka client does not commit offs...

2017-01-23 Thread liurenjie1024
GitHub user liurenjie1024 opened a pull request: https://github.com/apache/storm/pull/1891 STORM-2315 Storm kafka client does not commit offsets when ack is disabled. Here is why this bug happens: when ack is disabled, the ack method of spout will be call in the emit method, and tha

Local BlobStore and Nimbus H/A

2017-01-23 Thread Jungtaek Lim
Hi devs, I've been struggling to resolve specific scenario, and found Local BlobStore cares about Nimbus failure scenarios, but not about removing keys. For example, I heard that Nimbus crashed in specific scenario, and error stack trace pointed to below code: https://github.com/apache/storm/blob