unsubscribe

On Wed, May 27, 2015 at 1:51 AM, Tousif tousif.pa...@gmail.com wrote: Hi , Is there a way to share a resource file across all workers similar to hdfs. That resource/config file will have to be updated run time. i'm not looking at using hdfs for now. -- Regards Tousif Khazi --

Status of running storm on yarn (the yahoo project)

Hello! I'm interesting in running the storm topologies on yarn. I was looking at the yahoo project https://github.com/yahoo/storm-yarn, and I could observed that there is no activity since 7 months ago. Also, the issues and requests lists are not updated. Therefore I have some questions: 1. Is

Re: unsubscribe

Hi , If file is small you can pass them as json serialized object in storm config while submiting topology and get from config map at spout and bolt. Thanks, _Nipur On May 27, 2015 3:22 PM, Chris Bedford ch...@buildlackey.com wrote: On Wed, May 27, 2015 at 1:51 AM, Tousif

sharing a resource across all workers

Hi , Is there a way to share a resource file across all workers similar to hdfs. That resource/config file will have to be updated run time. i'm not looking at using hdfs for now. -- Regards Tousif Khazi

Re: Status of running storm on yarn (the yahoo project)

I have heard Nathan Marz mention Mesos. How is yarn / storm-yarn / slider-yarn different from Mesos? These are the links I found to Mesos: https://github.com/mesos/storm https://github.com/nathanmarz/storm-mesos http://mesos.apache.org/ Thank you for your time! + Jeff

Re: Status of running storm on yarn (the yahoo project)

Mesos is very similar to YARN.  It is a resource scheduler.  Storm in the past had support for mesos, through a separate repo https://github.com/nathanmarz/storm-mesos it might still work with the latest versions of storm.  I don't know.  The concept here is that there was a special layer

Re: Status of running storm on yarn (the yahoo project)

storm-yarn was originally done as a proof of concept.  We had plans to take it further, but the amount of work required to make it production ready on a very heavily used cluster was more then we were willing to invest at the time.  Most of that work was around network scheduling, isolation and

Re: Status of running storm on yarn (the yahoo project)

I also developed a prototype/proof-of-concept (read: duck tape and bailer twine) for running Storm on YARN. I took a slightly different approach than Yahoo’s storm-yarn and Slider which from a high level allow you to spin up a Storm cluster on top of YARN. In my PoC a topology is treated as a

RE: Status of running storm on yarn (the yahoo project)

Dell - Internal Use - Confidential Thanks Bobby, for the detailed answer. So it sounds like , it is better not to combine Storm with batch workloads at this point (yarn, mesos or ec2), due to the network saturation and timeout threats. Is this behavior also seen in other streaming frameworks

RE: Status of running storm on yarn (the yahoo project)

Hi Nathan, I want to do real time computation using storm, which one is best storm or trident. i need to handle huge amount of data , exactly once please help me Thanks! Date: Wed, 27 May 2015 12:40:43 -0400 Subject: Re: Status of running storm on yarn (the yahoo project) From:

Re: Status of running storm on yarn (the yahoo project)

Mesosphere has official support for Storm on Mesos: https://github.com/mesos/storm On Wed, May 27, 2015 at 11:14 AM, rajesh_kall...@dellteam.com wrote: *Dell - Internal Use - Confidential * Thanks Bobby, for the detailed answer. So it sounds like , it is better not to combine Storm with

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Not sure if you fixed the issue but I think the problem may come from the max spout pending. You are using Trident and this value is the max number of pending BATCHES and not number of tuples so let say your topic has 10 partitions and max spout pending is set to 10 and the max fetch size is set

Re: Load /Cache Data inside Bolt

If it is really static thing, you can serialize it and pass to Bolt's constructor. If you don't want to spend serialization cost, you can store it to file (and include it to resources of jar) and load it from preparation of Bolt. If you want to have small jar, you can use external storage. (RDB,

SF / East Bay Area Stream Processing Meetup next Thursday (6/4)

http://www.meetup.com/Bay-Area-Stream-Processing/events/219086133/ Thursday, June 4, 2015 6:45 PM TubeMogul http://maps.google.com/maps?f=qhl=enq=1250+53rd%2C+Emeryville%2C+CA%2C+94608%2C+us 1250 53rd St #1 Emeryville, CA 6:45PM to 7:00PM - Socializing 7:00PM to 8:00PM - Talks 8:00PM to

RE: java.lang.OutOfMemoryError: GC overhead limit exceeded

Dell - Internal Use - Confidential The previous reply was from March 10th so Bill may have resolved this issue but here is some info that can be useful to other folks in the future. You can set the JVM flags to do a heapdump on OOM using -XX:+HeapDumpOnOutOfMemoryError. You can then analyze

What will Storm do when a supervisor is manually killed?

​Hi Storm fellows, ​I've got a simple question and would like to have a quick answer. Let's say a storm topology is running on a cluster without any supervision, at the beginning it is behaving properly and have a balanced distribution. But you know, sometimes errors may occur and bring down the