Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override
WORKER_CHILDOPTS for a topology. You can configure any java options like
memory, gc etc
In your case it can be
config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g");
Thanks,
Satish.
On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe <navin....@searchlighthealth.com>
wrote:
> Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this
> error:
> *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must
> be an Iterable but was a class java.lang.Integer*
>
> A bit of looking around showed me that this might be the right syntax:
> *config.put(Config.TOPOLOGY_WORKER_CHILDOPTS,
> SOME_OPTS);*
>
> But couldn't find any example on what Storm expects as SOME_OPTS.
>
> On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <spicoflo...@gmail.com>
> wrote:
>
>> Hello!
>> For the the topology that you have 0MB allocated, for me it seems that
>> you don't have enough slots available. Check out the storm.yaml file (on
>> your worker machines) how many slots you have allocated.
>> (by default the are 4 slots available supervisor.slots.ports:
>> - 6700
>> - 6701
>> - 6702
>> - 6703) You have 5 topologies, therefore one is not ran.
>>
>> Regarding the memory allocation, you allocate memory per each worker
>> (slot available), not per topology. If you set up for your topology a
>> number of workers equal to 1, then you topology will run on a single worker
>> (available slot) and will receive the configuration that you gave for your
>> worker. If you configure to spread your spout and bolts to multiple
>> workers,(that are available in as configured slots) then all the workers
>> will receive the same amount of memory configured globally via
>> worker.childopts property in the storm yaml . So you don't configure the
>> meory per topology but per worker.
>>
>> If you want use different memory allocation for workers depending on your
>> topology components memory consumption, then you should use the property
>>
>> stormConfig.put("topology.worker.childopts",1024)
>>
>> I hope it helps.
>>
>> Regards,
>> Florin
>>
>> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe <
>> navin....@searchlighthealth.com> wrote:
>>
>>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); and
>>> now *all topologies* Assigned memory is 0MB.
>>> Does anyone know why it works this way? How do we assign memory to
>>> topologies in this situation?
>>>
>>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe <
>>> navin....@searchlighthealth.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a program *MyProg.java* inside which I'm creating 5 topologies
>>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows
>>>> the assigned memory for each topology as:
>>>> *Assigned Mem (MB)*
>>>> 832
>>>> 0
>>>> 832
>>>> 832
>>>> 832
>>>>
>>>> One of the topologies was assigned 0MB. Assuming that it was because of
>>>> setting a single Config for all of them, I gave them separate configs (as
>>>> below), but nothing changed.
>>>>
>>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *StormSubmitter.submitTopology(upTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName,
>>>> stormConfigFactory.GetNewConfig(),
>>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName,
>>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *And the StormConfigFactory class:public class StormConfigFactory {
>>>> public Config GetNewConfig() { Config stormConfig = new
>>>> Config(); stormConfig.setNumWorkers(1);
>>>> stormConfig.setNumAckers(1);
>>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false);
>>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64);
>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,
>>>> 65536);
>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,
>>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,
>>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS,
>>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS,
>>>> Arrays.asList(new String[]{"localhost"}));
>>>> return stormConfig; }}*
>>>>
>>>>
>>>> *How do I allocate separate memory and workers for each topology?*
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>
>
> --
> Regards,
> Navin
>
