Re: How to clean up /tmp in running topology

No, no one has written but you (Thank you!). I could write them to a bigger drive, but are they supposed to expand indefinitely, or what are they? Perhaps the reason they've grown so big is because I have started and restarted many topologies many times without properly shutting them down (kill -

Re: Exactly once transactions and storm

Hi, I've checked with the data team and it is not possible to have the database provide us with the "error as duplicate check" strategy (there is a link between the incoming message id and the persisted row, but the actual key is not unique based on it, so it wouldn't create this clash). We're lo

Re: How to clean up /tmp in running topology

Hello Bryan Have you gotten any feedback? You can have the logs generated in a different directory by setting -Djava.io.tmpdir on the command line (if your issue is with /tmp getting filled up), but I'd like to know how to manage these directories regardless of the location. Clay On Tue, Mar 3,

Re: Error 404 to Storm Ui

Hello, Nicolas. Did you make apache2 configuration for PROXY to Storm UI? And, you should remember Storm UI uses port 8080. > Hello, > > I would install Storm on Ubuntu 14.04 (on a virtual machine)... > > I also install apache2, and create a link in this directory with the *a2ensite > storm.con

Re: KryoDecorator not working when setNumWorkers > 1

Your discovery seems to have helped me, too. I'm not sure yet exactly what my problem was, but adding the chill-storm dependency and registering the BlizzardKryoFactory seems to have made it work. Now to go back and see if there's anything in my configuration that I added in desperation that's no

Re: KryoDecorator not working when setNumWorkers > 1

I was able to solve my issue. Once I verified that all my simulation logic was valid, I started looking for reasons why the registrations in my decorator weren't being picked up. Knowing that this was a consistent behavior, as opposed to what I originally thought, helped greatly (thanks Bill). Ul

cleaning up storm generated files

Hi all, How do I remove storm generated system files programmatically without stepping over the files that I shouldn't be deleting? I need this to cleanup the files left behind from the aborted topologies. I tried fuser and lsof, but with no luck. For example, fuser shows stormconf.ser and stormc

Re: Cleanup Cassandra session

Thank Dan, I implemented my code as per you suggestions. But I still need to wait and see if it is working as expected. If there is any other way anyone has experienced would be highly appreciated. -- Kushan Maskey 817.403.7500 M. Miller & Associates kushan.mas..

Re: Best way to clean-up log files in worker nodes

Hi, Have you try to edit the logback/cluster.xml file to enable logs rotation ? This file is periodically scanned so you don't need to restart your supervisors. 2015-03-03 19:39 GMT+01:00 Nick R. Katsipoulakis : > Hello, > > I have had my cluster running for a while and my worker nodes' log file

Re: Cleanup Cassandra session

Hi Kushan, I've blasted production Cassandra rings in the past from Storm, and I have not had much difficulty with too many open connections or anything else if I cleanup. Please cf the CassandraWriterBolt.java code below (for CQL prepared statements). Hope this helps, -Dan

Cleanup Cassandra session

I have bolt that inserts data into Cassandra database. When I kill the topology on test and production server, how can I make sure that the Cassandra session. The reasons I am asking is sometime I get too many files open exception thrown which tells me that there are number of open connections. Ple

Re: Exactly once transactions and storm

I am not really familiar with cassandra but I think they do support conditional insert/update. Something like Insert into my_table (col1) values ('val1') if not exists;. See if it actually does support conditional insert/update and if you can use this feature. Thanks Parth From: Javier Gonzal

Re: Exactly once transactions and storm

Hi Parth, Thanks for your reply. This could work, as the problem domain has a unique Id in the incoming stream, but I believe the db will be Cassandra, which updates instead of throwing errors when inserting a duplicate key. So I can't rely on that. Best regards, JG On Mar 3, 2015 12:45 PM, "Part

Best way to clean-up log files in worker nodes

Hello, I have had my cluster running for a while and my worker nodes' log files are getting really big in size. What is the best way to clear up some space? Should I just erase them and move them back them up somewhere else? Thanks, Nikos -- Nikolaos Romanos Katsipoulakis, University of Pittsbur

Re: Is there a way to find the details of the target bolt/bolts and the source bolt/bolts in topology.

You can have cycles in storm. As an example: Bolt A -> Bolt B Bolt B -> Bolt C Bolt B (non-default stream) -> Bolt A By using a different stream you can avoid infinite looping. On Tue, Mar 3, 2015 at 12:51 PM, Susheel Kumar Gadalay wrote: > You want to share some info with the parent bolt m

Re: Is there a way to find the details of the target bolt/bolts and the source bolt/bolts in topology.

You want to share some info with the parent bolt means you want to emit some tuples to the parent bolt. Will it not lead to cyclic. As far as I know it has to DAG (Directed Acyclic Graph). On 3/3/15, Ravali Kandur wrote: > Sure Vineet, > > I have a use case where I need to keep a track of the im

Re: Exactly once transactions and storm

Do you have some uniqueness in messages based on which you can define a DB constraint ? If there is one you define a unique constraint in DB, if the spout retries the bolt writing to DB will fail with constraint violation and the exception should also tell you which constraint was violated, you

Exactly once transactions and storm

Hi guys, We're looking at storm to solve a message processing scenario that needs to be horizontally scalable for high projected volume. The use case goes like this: 1.- receive messages from external source. 2.- generate a set of messages from this external input, based on rules. 3.- persi

Re: Setting heap size parameters by workers.childopts and supervisor.childopts

Thanks Parth for the clarification! But this concludes that SUPERVISOR_CHILDOPTS is useless in a topology specific config? It's good to know that a topology specific value does not affect a supervisor at all. 2015-03-03 18:06 GMT+01:00 Parth Brahmbhatt : > The setting in topology configs are sup

Re: Is there a way to find the details of the target bolt/bolts and the source bolt/bolts in topology.

In the initialize() method for your component you will receive a TopologyContext object reference from which you can access this information: https://nathanmarz.github.io/storm/doc/backtype/storm/task/TopologyContext.html On Tue, Mar 3, 2015 at 11:16 AM, Ravali Kandur wrote: > Sure Vineet, > >

Re: Setting heap size parameters by workers.childopts and supervisor.childopts

The setting in topology configs are suppose to be topology specific. When you set WORKER_CHILDOPTS those values are read by supervisor and passed to worker command when starting worker processes. However, SUPERVISOR_CHILDOPTS will have no effect as supervisors are already running and when they

Re: Is there a way to find the details of the target bolt/bolts and the source bolt/bolts in topology.

Sure Vineet, I have a use case where I need to keep a track of the immediate parent of the bolt to convey some information regarding the messages received. For example, when I receive a message from BoltB to BoltC, once BoltC processes the message, it needs to share some information with BoltB wi

Setting heap size parameters by workers.childopts and supervisor.childopts

Hello, I had a heap size problem within the *prepare* method of my bolts. Therefore, I tried to increase the heap size by setting the values in the topology config: conf.put(Config.WORKER_CHILDOPTS, "-Xmx4g"); conf.put(Config.SUPERVISOR_CHILDOPTS, "-Xmx4g"); But the out of memory exception rema

Error 404 to Storm Ui

Hello, I would install Storm on Ubuntu 14.04 (on a virtual machine)... I also install apache2, and create a link in this directory with the *a2ensite storm.conf* command. With this access, I can go to my storm directory by tapping : 172.1.24.18/ But, I have 404 errors : example : : GET http://1

Re: kafka-storm-cassandra example

I agree with Vineet’s sentiment. If you want to forge ahead, you might want to have a look at the example topology’s in storm-cassandra-cql: https://github.com/hmsonline/storm-cassandra-cql/tree/master/src/test/java/com/hmsonline/trident/cql/example

How to clean up /tmp in running topology

Greetings Storm Users, Does anyone know how to handle the large volume of files written to /tmp in running a storm topology. My topology is writing GBs worth of data to /tmp and it's filling up the drive. drwxrwxr-x 3 ubuntu ubuntu 4.0K Mar 3 09:33 1484627e-cce2-4055-8b71-04b681b928ad/ drwxrwx

Re: kafka-storm-cassandra example

Key, These frameworks are designed to have business value use case with them and without having the knowledge of coding and framework architecture I don't think its a smart way to move forward. Adding up, these tech. stack are not so hard to code or understand as it doesn't involve high level of

Re: kafka-storm-cassandra example

Hi Vineet, Thank you very much for your advice, but the problem is that I've never worked with java, scala. And all these tutorials looks like http://knowyourmeme.com/memes/how-to-draw-an-owl for me. I really hope that there is a simple working example somewhere which shows how to read data from k

Re: kafka-storm-cassandra example

I guess you can create your own topology with the reference link below. https://storm.apache.org/documentation/Tutorial.html Moreover Cassandra will just serve you as persistence layer which you can customize in your code as well. You can probably go with the default implementation of the Storm

Re: kafka-storm-cassandra example

Hi Vineet, All ports are opened. It seems that these examples try to create their own local storm cluster. What I need is working example which I can alter for my needs. Regards. On Tue, Mar 3, 2015 at 8:46 AM, Vineet Mishra wrote: > Hi Kay, > > It seems like your daemon connectivity is lacking