unsubscribe

unsubscribe

flux with hdfsbolt

Hi All, Two questions: 1) what version of hadoop does flux work with? My assumption was that it should not matter, but I am getting the following error when I use a hdfsbolt with flux which I assume implies some version mismatch. I don't have this issue if build a topology straightforward that uses

Re: flux

, [mike, 113] > 96763 [Thread-11-bolt-2] INFO > org.apache.storm.flux.wrappers.bolts.LogInfoBolt - source: bolt-1:2, > stream: default, id: {}, [nathan, 102] > > Can you post the output you are seeing where it just exists? > > -Taylor > > On May 26, 2015, at 12:05 PM, c

Re: flux

tring id) > > > -Taylor > > > On May 26, 2015, at 7:35 AM, clay teahouse wrote: > > Hi All, > > I am trying to test flux module for writing template driven topologies. I > setting the topologies in localcluster mode. > 1) using template simple_wordcount.yaml >

flux

Hi All, I am trying to test flux module for writing template driven topologies. I setting the topologies in localcluster mode. 1) using template simple_wordcount.yaml 2) using kafka_spout.yaml (using TestBolt) With (1) I don't get any output form TestBolt and the topology exits. With (2), I get t

Zookeeper ConnectionLossException

Hi All, What would be the reason for getting this exception while running a topology? Everything works fine for a while, and then I get this error and the topology dies. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /storm/partition_6 ... ...

Re: local cluster with external zookeeper

I got it. The issue was that the constructor expects the port number as long. On Tue, May 5, 2015 at 10:46 AM, clay teahouse wrote: > Hello Florian, > From which version of storm, this feature is available? I am running storm > .0.9.3 and I am getting no suitable constructor err

Re: storm logback freezing

ed the action and the logging level is > set to info > * the supervisor logs - if the logging level is set to info > * count and record the # of worker processes on your nodes > * the storm metric - new worker events - has not to me proved to be > accurate. I do not know why. > >

Re: local cluster with external zookeeper

Hello Florian, >From which version of storm, this feature is available? I am running storm .0.9.3 and I am getting no suitable constructor error when I compile with the topology with new LocalCluster("localhost", 2181). thanks On Tue, May 5, 2015 at 8:02 AM, clay teahouse wro

Re: storm logback freezing

ryone should have: > * nimbus > * worker_{port} > * supervisor > > Which of these is “freezing”? > > What version storm? > > When a worker starts, does writing to its log work? > > Have you made any changes to your logback configuration? > > > > From: clay

Re: local cluster with external zookeeper

Thanks Florian. I'll give it a try. On Tue, May 5, 2015 at 7:59 AM, Florian Hussonnois wrote: > Hi, > > You can create a LocalCluster with an external zookeeper as follows : new > LocalCluster("localhost", 2181) > > 2015-05-05 14:23 GMT+02:00 clay teahouse :

local cluster with external zookeeper

Hi All, Is it possible to run a topology in local cluster mode, with an external zookeeper? I see the following jira which seems to have been submitted to allow for an external zookeeper with local cluster topology. https://issues.apache.org/jira/browse/STORM-213 thank you Clay

storm logback freezing

Hi all, Has anyone experienced a case where storm logback freezes? The topology seems to be functioning without an issue (I can see the results, in the destination consumers), but the storm log shows no progress. This usually happens a couple of hours after the topology starts, and not right away.

streams in storm and memory usage

Hi all, I have a simple question which probably has been asked before, but I cannot find a concrete answer. I'd appreciate your feedback. Assume I have a spout that emit tuples which are consumed by 4 different bolts . Spout A ---> Bolt B ---> Bolt C ---> Bolt D

Re: template driven topology

mind, but any feedback is more than > welcome. > > -Taylor > > > > On Mar 27, 2015, at 7:56 PM, clay teahouse > wrote: > > > > Hi All, > > > > Is there anything out there for building topologies based on templates; > that is specifying the compone

template driven topology

Hi All, Is there anything out there for building topologies based on templates; that is specifying the components in a template and just have a simple framework that builds the topology based on that template. Sorry if this topic has been discussed before. I couldn't find anything related. thanks

reasons for topology hanging

Hi All, What could be the reasons for a topology hanging, under a somewhat heavy load (a few hundred mb per minute)? There is no error in the logs. I am using kafkaspout to pull data from kafka and a simple bolt to stream the data. My spout max pending is set to 1024. My topology is running in loc

Re: How to clean up /tmp in running topology

Hello Bryan Have you gotten any feedback? You can have the logs generated in a different directory by setting -Djava.io.tmpdir on the command line (if your issue is with /tmp getting filled up), but I'd like to know how to manage these directories regardless of the location. Clay On Tue, Mar 3,

cleaning up storm generated files

Hi all, How do I remove storm generated system files programmatically without stepping over the files that I shouldn't be deleting? I need this to cleanup the files left behind from the aborted topologies. I tried fuser and lsof, but with no luck. For example, fuser shows stormconf.ser and stormc

HdfsBolt and hdfs in HA mode

Hi All, Has anyone used HdfsBolt with hdfs in HA mode? How would you determine which hdfs node is the active node? thanks Clay

SSL_OPTS

Hi All, I have set and exported SSL_OPTS but it is not being picked up while running storm. I need to use client certificate with the http client connections originated from a bolt. I have set the options on the storm command as well, but still the same issue. thanks Clay

Re: Do I need to synchronize emits?

t a typical use case so we do > not synchronize by default. > > - Bobby > > > On Sunday, February 8, 2015 4:01 AM, clay teahouse < > clayteaho...@gmail.com> wrote: > > > Hi All, > > I emit my tuples in batches. Do I need to put the emit in a synchroni

Do I need to synchronize emits?

Hi All, I emit my tuples in batches. Do I need to put the emit in a synchronized block? The reason I am asking, I am getting the IndexOutOfBoundsException error once in a while, especially with a high paralleliziation hint. According to this link, it is a bug in storm, but I am using the latest s

Re: kafkaspout stops fetching

le of hundred meg and the max > spout pending is 1024" > your fetch.size probably too large as it trying to fetch 200mb of data at > a time and your topic might not have sufficient data. > > > On Fri, Feb 6, 2015, at 06:03 AM, clay teahouse wrote: > > Hi all, > &

kafkaspout stops fetching

Hi all, My kafkaspout stops fetching after the first couple of hundred tuples. My fetch and buffer are set to a couple of hundred meg and the max spout pending is 1024. I don't see anything unusual in the log. Any idea what could be the cause? thanks Clay

Re: kafkaspout is very slow

cts, Updated and In One Place. > Try FullContact for Free > <https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures> > > On Wed, Feb 4,

Re: kafkaspout is very slow

anything else that I can try? On Wed, Feb 4, 2015 at 6:51 PM, clay teahouse wrote: > 100,000 records is about 12MB. > I'll try bumping the numbers, by 100 fold to see if it makes any > difference. > thanks, > -Clay > > On Wed, Feb 4, 2015 at 5:47 PM, Filipa Moura > w

Re: kafkaspout is very slow

topology.executor.buffer.size: 65536 > topology.receiver.buffer.size: 16 > topology.executor.send.buffer.size: 65536 > > kafka.fetch.size.bytes: 102400 > kafka.buffer.size.bytes: 102400 > > thanks > Clay > > On Wed, Feb 4, 2015 at 4:24 PM, Filipa Moura > wrote: > >> c

Re: kafkaspout is very slow

r.size: 16 topology.executor.send.buffer.size: 65536 kafka.fetch.size.bytes: 102400 kafka.buffer.size.bytes: 102400 thanks Clay On Wed, Feb 4, 2015 at 4:24 PM, Filipa Moura wrote: > can you share a screenshot of the Storm UI for your spout? > > On Wed, Feb 4, 2015 at 9:5

Re: kafkaspout is very slow

UM_TASKS_KAFKA_SPOUT) >> //the maximum parallelism you can have on a KafkaSpout is the >> number of partitions >> .setMaxSpoutPending(*TOPOLOGY_MAX_SPOUT_PENDING*); >> >> -- >> Andrey Yegorov >> >> On Tue, Feb 3, 2015 at

Re: the external Zookeeper in Local mode

slow. thanks Clay On Wed, Feb 4, 2015 at 4:04 AM, clay teahouse wrote: > Hi, > Even if I use the local cluster with the constructor that specified the > external zookeeper's address, I still see: > backtype.storm.zookeeper - Starting inprocess zookee

Re: the external Zookeeper in Local mode

storm in local mode, it spins up an in-memory zookeeper for > > storm where it keeps data related to nimbus/supervisors, etc. > > > > Thanks, > > Shivendra > > > > > > > On Feb 3, 2015, at 7:23 PM, clay teahouse > wrote: > > > > > &

the external Zookeeper in Local mode

Hi, I have a topology running in local mode. The topology uses kafkaspout and is configured to use the external zookeeper. But when I start the topology, I see the following: org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000 I also messages like t

kafkaspout is very slow

Hi all, In my topology, kafka spout is responsible for over 85% of the latency. I have tried different spout max pending and played with the buffer size and fetch size, still no luck. Any hint on how to optimize the spout? The issue doesn't seem to be with the kafka side, as I see high throughput

Re: questions on task, threads and workers

ichael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ > is > a pretty good introduction to these concepts > > On Mon, Feb 2, 2015 at 9:54 AM, clay teahouse > wrote: > >> Hi, >> I have a few simple questions. >> 1)In storm .9.x, what i

questions on task, threads and workers

Hi, I have a few simple questions. 1)In storm .9.x, what is the default value for the bolt num tasks? According to the docs, the parallelism hint no longer sets the number of tasks, but the number of executor threads. 2)What happens if the number of tasks is less than the number of threads? Should

Re: zookeeper questions

; > On 25/01/15 23:14, clay teahouse wrote: > > I meant zookeeper is external to storm. I running the zookeeper that comes > with kafka. > > On Sun, Jan 25, 2015 at 3:11 PM, Margus Roo wrote: > >> So your storm's zookeeper is external. But how about Kafka's? &

Re: zookeeper questions

+372 51 480 > > On 25/01/15 23:08, clay teahouse wrote: > > Thanks Margus. > 1) I don't do anything (at least explicitly) with zookeeper settings or > the offsets. All I do in the topology, I specify the address of the > zookeeper (ZkHosts). Here is some th

Re: zookeeper questions

wrote it in your topology code? > zk path in example /storm is a place where storm keeps statuses about > storm in example. > > Margus (margusja) Roohttp://margus.roo.ee > skype: margusja > +372 51 480 > > On 25/01/15 22:16, clay teahouse wrote: > > Hi All, >

zookeeper questions

Hi All, A couple of questions: 1) What does storm.zookeeper.root point to? The definition says storm root directory location. The default value is /storm. But I am not sure I understand what this means. If I have external zookeeper running, what should this configuration point to? 2) I am running

Re: passing objects to bolts through Config

be an overkill. On Sat, Jan 24, 2015 at 10:28 AM, Irek Khasyanov wrote: > Why you need to use storm's config? Use it only for storm configuration, > not for your topology. > > On 24 January 2015 at 19:17, clay teahouse wrote: > >> Thanks Irek. So you are passin

Re: passing objects to bolts through Config

torm's config? Use it only for storm configuration, >> not for your topology. >> >> On 24 January 2015 at 19:17, clay teahouse >> wrote: >> >>> Thanks Irek. So you are passing the config through the bolt's >>> constructor. I was trying to

Re: passing objects to bolts through Config

it.SECONDS), 1) > > TopologyConfig is just generic class with my configuration > > On 24 January 2015 at 18:59, clay teahouse wrote: > >> Any feedback on what the issue could be would be appreciated. >> I am getting an instance of Config and add my static objects to it and

Re: passing objects to bolts through Config

t or HashMap. On Sat, Jan 24, 2015 at 12:01 AM, clay teahouse wrote: > Hi, > > I am trying to pass some objects to the bolts through config, but I am not > having much success. These objects are hashmap and arrarylists. I am > assuming these are serializable. Any idea what could be wrong? > > thanks, > Clay > >

passing objects to bolts through Config

Hi, I am trying to pass some objects to the bolts through config, but I am not having much success. These objects are hashmap and arrarylists. I am assuming these are serializable. Any idea what could be wrong? thanks, Clay

share singleton between topology and multiple bolts

Hi All, My topology initializes a singleton containing a s set of static objects. These objects can be complex but are static. I want all the bolts to have access to this singleton. It seems that the bolts can access the primitives such as strings in this singleton, but cannot access more complex o

Re: loading jars

through the > paths at which it was expecting jars, there were no jars there. So I copied > it there and then it was okay. > > On Thu, Jan 22, 2015 at 7:04 PM, clay teahouse > wrote: > >> Hi, >> I am trying to use storm in cluster mode. I've started nimbus and &g

loading jars

Hi, I am trying to use storm in cluster mode. I've started nimbus and supervisor, but when I try to run the topology, I get the error it cannot find or load some jar in $STORM_HOME/lib. All the jars it is complaining about do exist there. When I do storm classpath, I see the jars that it is complai

Re: worker logs

effect > in distributed mode. > > Thanks > Parth > > On Jan 19, 2015, at 11:34 AM, clay teahouse > wrote: > > I am using 0.9.3. I did set it in storm.yaml, and the console log shows > it as set, but the worker logs still end up in /tmp. > I am running storm in loca

Re: worker logs

of storm are you using? I just tried this locally and it seem > to be working on the master branch right now. > > You can try to set the “storm.log.dir” in your storm config and see if > that takes effect or not. > > Thanks > Parth > > On Jan 19, 2015, at 9:55 AM, clay te

Re: worker logs

e.name}. You can find the file under > STORM_HOME/logback/. > > Thanks > Parth > > > > On Jan 19, 2015, at 7:34 AM, Susana González wrote: > > I'm afraid I haven't specified them. That location is where the logs were > created by default. > > On Mon,

Re: worker logs

e e.g. a file > called "storm" in the folder /etc/logrotate.d with something like the > following: > > /opt/storm/logs/worker-*.log { > daily > rotate 5 > compress > missingok > notifempty > copytruncate > create 640 root root > }

worker logs

Hi All, Sorry if these questions have asked before. I cannot find the answers. 1) Where do I specify the location of the worker logs? The default seems to be /tmp. 2) How do I rotate these logs? thanks, Clay

Re: null pointer exception

> There is a known issue where this can happen if two shell bolts share an > executor, because they are multi-threaded. > > - Bobby > > > On Sunday, January 4, 2015 10:19 PM, clay teahouse < > clayteaho...@gmail.com> wrote: > > > Hi All, > I have the fol

null pointer exception

Hi All, I have the following topology spout -> Bolt1 --> Bolt2 Neither bolts are async or multi-threaded. Bolt2 uses http client to make post/put requests to a web server. Both bolts ack the tuples before exiting the execute. The topology runs fine for a while under a load of about 50MB/minute. A

hdfsbolt

Hi All, Why HdfsBolt doesn't retry when the hadoop node is down or not accessible and dies brings down the topology with it too? I can catch the run time exception and keep the topology going, but was wondering why the retry is not built into HdfsBolt. thank you Clay

Re: bolt stop receiving tuples

to take a thread dump of the java process when it is in hung > state. That will clearly tell you what the problem is. > > For easy diagonosis set the worker to one and possibly set the number of > tasks of spout/bolt to 1. > > Thanks and Regards, > Devang > On 4 Dec 2014 21

Re: bolt stop receiving tuples

the code in execute method of bolt B with a log > statement and check if it's still an issue. > > Thanks and Regards, > Devang > On 4 Dec 2014 19:28, "clay teahouse" wrote: > >> This is a local cluster. I don't see anything interesting in the logs >>

Re: bolt stop receiving tuples

nd worker > > On Wed, Dec 3, 2014 at 6:09 PM, clay teahouse > wrote: > >> Hello All, >> >> I have this configuration: >> >> spout -> Bolt A (emits tuples) -> Bolt B >> >> Bolt A emits tuples successfully but bolt B stops receiving t

bolt stop receiving tuples

Hello All, I have this configuration: spout -> Bolt A (emits tuples) -> Bolt B Bolt A emits tuples successfully but bolt B stops receiving tuples after the first time (it never enters the execute after the first time). The first time execution seems to be successful. Any idea what the issue coul

Re: acking and offsets

aseBasicBolt which does it for you. > > Please see the following link for detailed explanation > > https://storm.apache.org/documentation/Guaranteeing-message-processing.html > > Kobi > > On Wed, Nov 19, 2014 at 6:05 AM, clay teahouse > wrote: > >> 1) If in case

Re: acking and offsets

ll retry it from > the start, it will not retry only the bolts that were not acking. > > You can try to save the IDs in each bolt and ignore them if they reappear. > > Kobi > > On Mon, Nov 17, 2014 at 9:40 PM, clay teahouse > wrote: > >> Just to clarify my question

Re: acking and offsets

. On Mon, Nov 17, 2014 at 12:49 PM, clay teahouse wrote: > Hello All, > I am using kafka spout that comes with storm 0.9.3 ( > https://github.com/apache/storm). > I am having several different bolts consuming the same tuples from the > spout (in the same topology). These bolts pro

acking and offsets

Hello All, I am using kafka spout that comes with storm 0.9.3 ( https://github.com/apache/storm). I am having several different bolts consuming the same tuples from the spout (in the same topology). These bolts process the tuples and send the output to different destinations. I have a couple of bas

in bolt caching and joining streams in bolts

Hello All, I've asked this question before in a different form but have not gotten any feedback. I'd appreciate if you have examples of the following use cases that you could share. 1) in-bolt caching 2) a bolt receiving multiple streams 3) joining the multiple streams (in bolts). thanks, Clay

Re: join and in-bolt caching

Hello All, I'd appreciate your input/pointers regarding my post on join and in-blot caching. thanks, Clay On Tue, Nov 11, 2014 at 5:09 AM, clay teahouse wrote: > Hello All, > I need to > 1) Look up some values from a source and cache them in a bolt > 2) Filter a second st

join and in-bolt caching

Hello All, I need to 1) Look up some values from a source and cache them in a bolt 2) Filter a second stream based on the looked up values in (1). Example: 1) Area Codes = 712, 915 -- these area codes might change from time to time 2) People living in different addresses Emit: People leaving in a

Re: emitting batches of tuples

r to a db column > join. > > > > https://storm.incubator.apache.org/documentation/Trident-API-Overview.html > > > > *From:* clay teahouse [mailto:clayteaho...@gmail.com] > *Sent:* Monday, November 03, 2014 9:41 AM > > *To:* user@storm.apache.org > *Subject:* Re: emitting

Re: emitting batches of tuples

I meant to address Bill. Sorry for the mix up. Clay. On Mon, Nov 3, 2014 at 8:41 AM, clay teahouse wrote: > Thanks Andrew. How would I chain the streams in trident? I want to pipe > the output of one stream to another stream. Can I have an hierarchy of > streams with trident? > &

Re: emitting batches of tuples

our spout > “A”, you can just do > > > > val stream1 = A.each() > > val stream2 = A.each() > > > > and now you have 2 streams from your spout. You can then join or merge > the streams later. > > > > *From:* clay teahouse [mailto:clayteaho...@gmail.com

Re: emitting batches of tuples

looking for. > > Cheers. > > Kindly yours, > > Andrew Grammenos > > -- PGP PKey -- > ​ <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt> > https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt > > On Mon, Nov 3, 2014 at 1:42 PM, clay teahouse > wrote:

emitting batches of tuples

Hello All, Is it possible emit batches of tuples, as opposed to one tuple at a time? In other word, is it possible to batch the tuples before emitting them? An application for batching the tuples is for example for writing the tuples to a tcp socket but not wanting to do a flush after each tuple i

conditional streaming

Hello All, My data source can have multiple formats, except that all records share the first field. Based on the value of this field, I want to generate a separate stream that goes to a particular bolt for special processing. Now my question is how one does conditional streaming, based on a partic

multi-stream example

Hello All, Can someone share an example of a bolt with multi stream output, with each particular output stream going to a particular bolt? Bolt A =>stream 1 => Bolt B Bolt A => stream 2 => Bolt C Bolt A => stream 3 => Bolt D thanks, Clay

TOPOLOGY_ACKER_EXECUTORS

Hello, I am trying to turn off acking by settingTOPOLOGY_ACKER_EXECUTORS to 0. But when I do that my trident topology fails with the following error and subsequently the worker dies. java.lang.RuntimeException: backtype.storm.topology.FailedException: Received commit for different transaction atte

tcp socket spout

Hi, Is there a tcp socket spout out there? thanks, Clay

subscribe

running the test topologies in storm-kafka and storm-hdfs

Hi, Can someone help by showing how to run the test topologies that are in storm-hdfs and storm-kafka, that come with https://github.com/apache/storm , storm-0.9.3-incubating? I have spent lots of time on this and still not able to get them working. I don't have problem running the storm-kafka tes

new topology

Hello All, I am using https://github.com/apache/storm, storm-0.9.3-incubating, working through the storm-kafka examples. TridentKafkaToplogy when it tries to get a new topology instance. I have had the same experience in other cases too. Any clue what could be wrong? thanks, Clay

storm-hdfs

Hello, I am trying to run the examples that come with external/storm-hdfs ( https://github.com/apache/storm, storm-0.9.3-incubating) but I am not able to load any data into hdfs. I've tried HdfsFileTopology and TridentFileTopology with the url to my hfds as input. I don't get any error. Any idea