Re: Storm Shared Memory

Now Storm no concept like 'shared memory' you should use external resources. As a reference, you can take a look at `storm-redis` https://github.com/apache/storm/tree/master/external/storm-redis. 2016-07-13 21:21 GMT+08:00 Stephen Powis : > You'd need to use some external

Re: Storm Shared Memory

You'd need to use some external data store (such as redis) to maintain state that exists across multiple JVMs On Wed, Jul 13, 2016 at 4:22 AM, jimmy tekli wrote: > Hello , > I developed a java project based on the storm topology that process > incoming tuples in a

RE: How to use KafkaSpout to consume kafka cluster secured with kerberos

Hi everybody, I am facing the same problem (with Storm 1.0.1 and Kafka 0.9.0.1). Any ideas? Thanks, Tom From: Hao Chen [mailto:h...@apache.org] Sent: Dienstag, 12. Juli 2016 02:44 To: user@storm.apache.org Subject: How to use KafkaSpout to consume kafka cluster secured with kerberos Hi Team,

Re: Allocating separate memory and workers to topologies of a single jar?

*Solved:* All of your solutions worked. Thanks. Mentioning the solution more elaborately for posterity: According to NathamMarz : *The slots on each machine is determined by the storm.yaml that the supervisor has, not the storm.yaml

Re: storm cgroup question

thanks 2016-07-13 4:26 GMT+08:00 Bobby Evans : > cgroups is still a work in progress even if the code is in, it is not > fully tested and ready to go. > > - Bobby > > > On Tuesday, July 12, 2016 9:12 AM, 李岩 wrote: > > > Hi All > > I have a question:

Re: Allocating separate memory and workers to topologies of a single jar?

Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override WORKER_CHILDOPTS for a topology. You can configure any java options like memory, gc etc In your case it can be config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g"); Thanks, Satish. On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe

Re: MasterBatchCoordinator has become the bottleneck in my trident topology, can the number of MasterBatchCoordinator task larger than 1?

AFAIK, there can be only one master-batch-coordinator for each batch group. You should look at increasing parallelism of trident operations to get better latencies. Thanks, Satish. On Wed, Jul 13, 2016 at 12:26 PM, hong mao wrote: > I get the resaon, there is only one

Re: Storm monitoring

Hi, I had written a MetricsConsumer for CloudWatch some time ago. The code would need reworking a bit, I might release it if I find the time. For StormCrawler topologies, our Elasticsearch module

Storm Shared Memory

Hello ,I developed a java project based on the storm topology that process incoming tuples in a certain way.I tested it locally and it worked perfectly in the context of streaming tuples of course. My topology is formed of one spout and three bolts.In my code I used Data Structures (such as

Re: Allocating separate memory and workers to topologies of a single jar?

Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this error: *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must be an Iterable but was a class java.lang.Integer* A bit of looking around showed me that this might be the right syntax:

Re: Required field 'nimbus_uptime_secs' is unset!

Hi! For me it seems that you have to pass the classpath or to build a fat jar. Please have a look at this post: http://stackoverflow.com/questions/32976198/deploy-storm-topology-remotely-using-storm-jar-command-on-windows Florin On Wed, Jul 13, 2016 at 8:09 AM, ram kumar

Re: Allocating separate memory and workers to topologies of a single jar?

Hello! For the the topology that you have 0MB allocated, for me it seems that you don't have enough slots available. Check out the storm.yaml file (on your worker machines) how many slots you have allocated. (by default the are 4 slots available supervisor.slots.ports: - 6700 - 6701

Re: MasterBatchCoordinator has become the bottleneck in my trident topology, can the number of MasterBatchCoordinator task larger than 1?

I get the resaon, there is only one MasterBatchCoordinator for each trident spout, so setting the thrid parameter (parallelism) takes no effect. 2016-07-12 0:08 GMT+08:00 hong mao : > Hi all, > We are using trident topology to pull messages from kafka and store >

Re: Allocating separate memory and workers to topologies of a single jar?

I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and now *all topologies* Assigned memory is 0MB. Does anyone know why it works this way? How do we assign memory to topologies in this situation? On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe