Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this
error:
*java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must
be an Iterable but was a class java.lang.Integer*
A bit of looking around showed me that this might be the right syntax:
*config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
SOME_OPTS);*
But couldn't find any example on what Storm expects as SOME_OPTS.
On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <spicoflo...@gmail.com>
wrote:
> Hello!
> For the the topology that you have 0MB allocated, for me it seems that
> you don't have enough slots available. Check out the storm.yaml file (on
> your worker machines) how many slots you have allocated.
> (by default the are 4 slots available supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703) You have 5 topologies, therefore one is not ran.
>
> Regarding the memory allocation, you allocate memory per each worker (slot
> available), not per topology. If you set up for your topology a number of
> workers equal to 1, then you topology will run on a single worker
> (available slot) and will receive the configuration that you gave for your
> worker. If you configure to spread your spout and bolts to multiple
> workers,(that are available in as configured slots) then all the workers
> will receive the same amount of memory configured globally via
> worker.childopts property in the storm yaml . So you don't configure the
> meory per topology but per worker.
>
> If you want use different memory allocation for workers depending on your
> topology components memory consumption, then you should use the property
>
> stormConfig.put("topology.worker.childopts",1024)
>
> I hope it helps.
>
> Regards,
> Florin
>
> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
> navin....@searchlighthealth.com> wrote:
>
>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
>> now *all topologies* Assigned memory is 0MB.
>> Does anyone know why it works this way? How do we assign memory to
>> topologies in this situation?
>>
>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>> navin....@searchlighthealth.com> wrote:
>>
>>> Hi,
>>>
>>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows
>>> the assigned memory for each topology as:
>>> *Assigned Mem (MB)*
>>> 832
>>> 0
>>> 832
>>> 832
>>> 832
>>>
>>> One of the topologies was assigned 0MB. Assuming that it was because of
>>> setting a single Config for all of them, I gave them separate configs (as
>>> below), but nothing changed.
>>>
>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>
>>>
>>>
>>>
>>>
>>>
>>> *StormSubmitter.submitTopology(upTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>> stormConfigFactory.GetNewConfig(),
>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *And the StormConfigFactory class:public class StormConfigFactory {
>>> public Config GetNewConfig() { Config stormConfig = new
>>> Config(); stormConfig.setNumWorkers(1);
>>> stormConfig.setNumAckers(1);
>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>> 65536);
>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>> Arrays.asList(new String[]{"localhost"}));
>>> return stormConfig; }}*
>>>
>>>
>>> *How do I allocate separate memory and workers for each topology?*
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
>
--
Regards,
Navin
