Re: Custom Grouping using flux

Is numberOfTasks size of the array you receive as third argument in your prepare? In that case change argument to boltIds.add method to targetTasks.get((int) mod); What you receive in prepare method as the third argument is the list of integers that represent all the tasks for the target. And i

Re: addConfiguration in flux

. Can we do the same in flux ? Regards, Rajeev. On Tue, 30 Jan 2018 at 03:32, Priyank Shah mailto:ps...@hortonworks.com>> wrote: Check out http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html and search for configMethods on that page. It has an example for how to call a withFoo method

Re: Number of Tasks in Flux yaml file

I misread your first question in previous email. Does not look like setting number of tasks is supported(or straightforward) using flux yaml. Number of executors is supported From: Priyank Shah Reply-To: "user@storm.apache.org" Date: Monday, February 5, 2018 at 10:56 AM

Re: Number of Tasks in Flux yaml file

Hi, Please find answers inline and check out http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html for more information From: Rajeev Reply-To: "user@storm.apache.org" Date: Monday, February 5, 2018 at 2:15 AM To: "user@storm.apache.org" Subject: Number of Tasks in Flux yaml file Hi, 1

Re: addConfiguration in flux

Check out http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html and search for configMethods on that page. It has an example for how to call a withFoo method on the bolt instance. You can do the same on MessagingQueueSource instance that you can create using component definition. From: Raje

Re: storm flux API - emit to specific streamId

You can provide it under grouping. Check GroupingDef. Pasting an example below. streams: - name: "kafka --> split" # name isn't used (placeholder for logging, UI, etc.) from: "kafka-spout" to: "splitsentence" grouping: type: SHUFFLE streamId: “mystream” From: "Huang, Z

Re: Kafka Spouts Lag via REST API

Hi SG, You will need to update your storm-kafka dependency to latest storm release version for you to be able to see the spout lag From: S G Reply-To: "user@storm.apache.org" Date: Thursday, February 23, 2017 at 8:53 AM To: "user@storm.apache.org" Subject: Re: Kafka Spouts Lag via REST API H

Re: Storm Kafka offset monitoring

etting commit failed exception due to rebalance Can you point out what's the proper way for implementing Kafka spout. In storm 1.0.3 docs I have seen the way using zookeeper broker hosts Regards Pradeep S On Tue, Feb 21, 2017 at 2:35 PM Priyank Shah mailto:ps...@hortonworks.com>> wrote: Hi Prad

Re: Storm Kafka offset monitoring

o find 1.1.0 version of storm Can you please point to that.also can you please confirm on what specific spout changes to make. Regards Pradeep S On Tue, Feb 21, 2017 at 10:54 AM Priyank Shah mailto:ps...@hortonworks.com>> wrote: Hi Pradeep, If you upgrade your spout in the topology and storm

Re: Storm Kafka offset monitoring

Hi Pradeep, If you upgrade your spout in the topology and storm code to a later version(I checked v1.1.0 and it has the tool) you will get a table in storm ui which show you offsets. If you cannot upgrade then I think you will have to do it manually. From: pradeep s Reply-To: "user@storm.apach

Re: Kafka monitor unable to get offset lag

r unable to get offset lag Yes running from command line gives me json with topic lag. What is secured setup? On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah mailto:ps...@hortonworks.com>> wrote: Hi Igor, Just to clarify, are you saying that running storm-kafka-monitor from command line wi

Re: Kafka monitor unable to get offset lag

ingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal: 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal: On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah ma

Re: Kafka monitor unable to get offset lag

Hi Igor, When you say topology is working fine do you mean you see data flowing? Can you try to look up the logs for ui server and paste relevant lines here if any? Priyank Sent from my iPhone On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko mailto:f1she...@gmail.com>> wrote: I've launched topolog

Re: Is there any Kinesis Spout for Storm other than awslab?

Yes, there is. You can check out https://github.com/apache/storm/tree/master/external/storm-kinesis From: Noppanit Charassinvichai Reply-To: "user@storm.apache.org" Date: Wednesday, October 12, 2016 at 3:33 AM To: "user@storm.apache.org" Subject: Is there any Kinesis Spout for Storm other than

Re: Debugging tuples being replayed despite acks

You mentioned that you are making sure that its acking everything in first bolt. Is there any other bolt downstream that could be failing the tuples? From: Jason Kania Reply-To: "user@storm.apache.org", Jason Kania Date: Tuesday, August 9, 2016 at 10:28 AM To: User S

Re: Storm bolt instances - limits

Hi Kashyap, Please check out http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 Priyank From: Kashyap Mhaisekar Reply-To: "user@storm.apache.org" Date: Tuesday, January 5, 2016 at 1:21 PM To: "user@storm.apache.org

Re: multiple fields grouping in storm

ember 3, 2015 at 7:37 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" Subject: Re: multiple fields grouping in storm Thanks Priyank! Are there any method to guarantee that same task just receives the tuples with same fields? On Wed, Nov 4, 2015 at 11:21 AM, Priyank Sh

Re: multiple fields grouping in storm

Hi Shuo, Seeing a lot of group errors in log file is expected. From http://storm.apache.org/documentation/Concepts.html the description of Field Grouping says 1. Fields grouping: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the

Re: Storm and enviroment variables

look for storm.yaml in that dir and this fixed my issue. From: Priyank Shah [mailto:ps...@hortonworks.com] Sent: Wednesday, October 07, 2015 5:31 PM To: user@storm.apache.org<mailto:user@storm.apache.org> Subject: Re: Storm and enviroment variables I think you can specify a config file that

Re: Storm and enviroment variables

ober 7, 2015 at 2:15 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" Subject: RE: Storm and enviroment variables Ok so an additional question here… Can I tell storm on startup to use a storm.yaml file that I specify with a full path, or will it only look in /etc/storm

Re: Storm and enviroment variables

Hi Joseph, I think storm core components don’t support environment variables. It either has to be a config yaml file or a –c option. For –c option, its a key value pair of the form key=value. I checked the code and key should be same as key in yaml. It should work for simple strings. But not s