Hello!
For the the topology that you have 0MB allocated, for me it seems that
you don't have enough slots available. Check out the storm.yaml file (on
your worker machines) how many slots you have allocated.
(by default the are 4 slots available supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703) You have 5 topologies, therefore one is not ran.
Regarding the memory allocation, you allocate memory per each worker (slot
available), not per topology. If you set up for your topology a number of
workers equal to 1, then you topology will run on a single worker
(available slot) and will receive the configuration that you gave for your
worker. If you configure to spread your spout and bolts to multiple
workers,(that are available in as configured slots) then all the workers
will receive the same amount of memory configured globally via
worker.childopts property in the storm yaml . So you don't configure the
meory per topology but per worker.
If you want use different memory allocation for workers depending on your
topology components memory consumption, then you should use the property
stormConfig.put("topology.worker.childopts",1024)
I hope it helps.
Regards,
Florin
On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <navin....@searchlighthealth.com>
wrote:
> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
> now *all topologies* Assigned memory is 0MB.
> Does anyone know why it works this way? How do we assign memory to
> topologies in this situation?
>
> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
> navin....@searchlighthealth.com> wrote:
>
>> Hi,
>>
>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>> and using *stormSubmitter* to submit it to Storm. The storm UI shows the
>> assigned memory for each topology as:
>> *Assigned Mem (MB)*
>> 832
>> 0
>> 832
>> 832
>> 832
>>
>> One of the topologies was assigned 0MB. Assuming that it was because of
>> setting a single Config for all of them, I gave them separate configs (as
>> below), but nothing changed.
>>
>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>
>>
>>
>>
>>
>>
>> *StormSubmitter.submitTopology(upTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>> stormConfigFactory.GetNewConfig(),
>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *And the StormConfigFactory class:public class StormConfigFactory {
>> public Config GetNewConfig() { Config stormConfig = new
>> Config(); stormConfig.setNumWorkers(1);
>> stormConfig.setNumAckers(1);
>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>> 65536);
>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>> Arrays.asList(new String[]{"localhost"}));
>> return stormConfig; }}*
>>
>>
>> *How do I allocate separate memory and workers for each topology?*
>>
>> --
>> Regards,
>> Navin
>>
>
>
>
> --
> Regards,
> Navin
>
