How can i set the value of taskmanager.network.numberOfBuffers ?

2020-03-19 Thread forideal
Hi community This parameter makes me confused. |taskmanager.network.numberOfBuffers| 70 | In my job, i use 700 slots, but ,i have to set the this parameter to 70.If not,i will get a exception. java.io.IOException: Insufficient n

Re: How can i set the value of taskmanager.network.numberOfBuffers ?

2020-03-20 Thread Xintong Song
Hi Forideal, Do you mean you have 700 slots per TM or in total? How many TMs do you have? And how many slots do you have per TM? Also, when is the screenshot taken? It is after the job is fully initiated? It seems you only need 1k+ network buffers. Thank you~ Xintong Song On Fri, Mar 20, 202

Re:Re: How can i set the value of taskmanager.network.numberOfBuffers ?

2020-03-23 Thread forideal
Hi Xintong, Thank you for your reply. Do you mean you have 700 slots per TM or in total? How many TMs do you have? And how many slots do you have per TM? I have a Flink Cluster with 35 TMs,each TM has 16 slots. cluster info: total TMs=35 ,total slots=560 Job info: request slot 400 It is after

Re: Re: How can i set the value of taskmanager.network.numberOfBuffers ?

2020-03-23 Thread Arvid Heise
Hi Forideal, which Flink version are you using? If you using 1.9 or older, have a look at the memory setup [1] and config docs [2]. If you are using 1.10, it should be enough to increase* taskmanager.network.memory.**fraction* and *taskmanager.network.memory.**max*. You shouldn't use *taskmanager.