The new default is equivalent to the previous "streaming mode". The
community decided to get rid of this distinction, because it was
confusing to users.
The difference between "streaming mode" and "batch mode" was how
Flink's managed memory was allocated, either lazily when required
('streaming mode") or eagerly on task manager start up ("batch mode").
Now it's lazy by default.
This is not something you need to worry about, but if you are mostly
using the DataSet API where pre allocation has benefits, you can get
the "batch mode" behaviour by using the following configuration key:
taskmanager.memory.preallocate: true
But you are using the DataStream API anyways, right?
– Ufuk
On Tue, Feb 23, 2016 at 6:36 AM, Welly Tambunan <if05...@gmail.com> wrote:
> Hi Fabian,
>
> Previously when using flink 0.9-0.10 we start the cluster with streaming
> mode or batch mode. I see that this one is gone on Flink 1.00 snapshot ? So
> this one has already taken care of the flink and optimize by runtime >
>
> On Mon, Feb 22, 2016 at 5:26 PM, Fabian Hueske <fhue...@gmail.com> wrote:
>>
>> Hi Welly,
>>
>> sorry for the late response.
>>
>> The number of network buffers primarily depends on the maximum parallelism
>> of your job.
>> The given formula assumes a specific cluster configuration (1 task manager
>> per machine, one parallel task per CPU).
>> The formula can be translated to:
>>
>> taskmanager.network.numberOfBuffers: p ^ 2 * t * 4
>>
>> where p is the maximum parallelism of the job and t is the number of task
>> manager.
>> You can process more than one parallel task per TM if you configure more
>> than one processing slot per machine ( taskmanager.numberOfTaskSlots). The
>> TM will divide its memory among all its slots. So it would be possible to
>> start one TM for each machine with 100GB+ memory and 48 slots each.
>>
>> We can compute the number of network buffers if you give a few more
>> details about your setup:
>> - How many task managers do you start? I assume more than one TM per
>> machine given that you assign only 4GB of memory out of 128GB to each TM.
>> - What is the maximum parallelism of you program?
>> - How many processing slots do you configure for each TM?
>>
>> In general, pipelined shuffles with a high parallelism require a lot of
>> memory.
>> If you configure batch instead of pipelined transfer, the memory
>> requirement goes down
>> (ExecutionConfig.setExecutionMode(ExecutionMode.BATCH)).
>>
>> Eventually, we want to merge the network buffer and the managed memory
>> pools. So the "taskmanager.network.numberOfBuffers" configuration whill
>> hopefully disappear at some point in the future.
>>
>> Best, Fabian
>>
>> 2016-02-19 9:34 GMT+01:00 Welly Tambunan <if05...@gmail.com>:
>>>
>>> Hi All,
>>>
>>> We are trying to running our job in cluster that has this information
>>>
>>> 1. # of machine: 16
>>> 2. memory : 128 gb
>>> 3. # of core : 48
>>>
>>> However when we try to run we have an exception.
>>>
>>> "insufficient number of network buffers. 48 required but only 10
>>> available. the total number of network buffers is currently set to 2048"
>>>
>>> After looking at the documentation we set configuration based on docs
>>>
>>> taskmanager.network.numberOfBuffers: # core ^ 2 * # machine * 4
>>>
>>> However we face another error from JVM
>>>
>>> java.io.IOException: Cannot allocate network buffer pool: Could not
>>> allocate enough memory segments for NetworkBufferPool (required (Mb): 2304,
>>> allocated (Mb): 698, missing (Mb): 1606). Cause: Java heap space
>>>
>>> We fiddle the taskmanager.heap.mb: 4096
>>>
>>> Finally the cluster is running.
>>>
>>> However i'm still not sure about the configuration and fiddling in task
>>> manager heap really fine tune. So my question is
>>>
>>> Am i doing it right for numberOfBuffers ?
>>> How much should we allocate on taskmanager.heap.mb given the information
>>> Any suggestion which configuration we need to set to make it optimal for
>>> the cluster ?
>>> Is there any chance that this will get automatically resolve by
>>> memory/network buffer manager ?
>>>
>>> Thanks a lot for the help
>>>
>>> Cheers
>>>
>>> --
>>> Welly Tambunan
>>> Triplelands
>>>
>>> http://weltam.wordpress.com
>>> http://www.triplelands.com
>>
>>
>
>
>
> --
> Welly Tambunan
> Triplelands
>
> http://weltam.wordpress.com
> http://www.triplelands.com
