Hi Pritam, The legacy config option `taskmanager.network.numberOfBuffers` is deprecated and will be ignored, so please do not refer to it. The exact size of network memory calculated through `taskmanager.memory.network.fraction`, while also ensuring that it is within the constraints of min and max value. So it can be explicitly specified by setting the min & max to the same value if you want. Another thing to note is that the fraction is relative to flink memory, not process memory(8gb in your example), and there is a difference of jvmMetaspaceAndOverload memory part between them.
Best regards, Weijie Hangxiang Yu <master...@gmail.com> 于2023年6月7日周三 10:44写道: > Hi, Pritam. > IIUC, the number is TM scope and just calculated by "available network > memory / buffer size". > For example, if the fraction is 0.1, the number may be about ( 8 * 0.1 * > 1024 * 1024 * 1024 / 32768). > > > On Tue, Jun 6, 2023 at 3:14 PM Pritam Agarwala < > pritamagarwala...@gmail.com> wrote: > >> Thanks for answering Hangxiang! >> >> Still confused. How did Flink get this number 22773 ? Couldn't find the >> default value of "taskmanager.network.numberOfBuffers" config. >> and According to the formula it should be around 1000. ( #slots-per-TM^2 >> * #TMs * 4 = 4^2 * 16 * 4 = 1024) >> >> I have a total of 6000 tasks with 16 TM , 4 cores each with >> jobmanger/taskmanger.momry.process.size = 8 gb . >> >> >> Thanks & Regards, >> Pritam >> >> >> >> On Tue, Jun 6, 2023 at 9:02 AM Hangxiang Yu <master...@gmail.com> wrote: >> >>> Hi, Pritam. >>> This error message indicates that the current configuration of the >>> network buffer is not enough to handle the current workload. >>> >>>> What is the meaning of this exception (The total number of network >>>> buffers is currently set to 22773 of 32768 bytes each)? >>>> >>> This just provides some information about the current status of network >>> buffers (22773 * 32768 bytes ~= 711MB). >>> >>> How to figure out a good combination of >>>> ('taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', >>>> and 'taskmanager.memory.network.max'.) >>>> for this issue ? >>>> >>> IIUC, There is no absolute standard for setting these parameters. >>> These parameters may be affected by many factors, such as the data flow >>> rate, computational complexity, and memory usage of your job. >>> >>> Some steps to setup and adjust these parameters: >>> 1. Check the available memory on your job. >>> 2. Evaluate the network usage and consider how much memory could be >>> used for network buffers. >>> 3. Monitor the system and collect metrics such as network throughput, >>> memory usage. >>> 4. Adjust these parameters if the job has a high network usage or >>> memory-intensive. >>> >>> Just a personal and immature suggestion about how to adjust when it's >>> not enough: 1. Increase taskmanager.memory.network.fraction from 0.1 >>> to 0.2, or just increase taskmanager.memory.network.max slightly. >>> 2. If the buffer size is too large, it may affect checkpoints. So it's >>> recommended to combine with buffer debloating. >>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism> >>> >>> >>> On Tue, Jun 6, 2023 at 2:44 AM Pritam Agarwala < >>> pritamagarwala...@gmail.com> wrote: >>> >>>> Hi All, >>>> >>>> >>>> java.io.IOException: Insufficient number of network buffers: required >>>> 2, but only 0 available. The total number of network buffers is currently >>>> set to 22773 of 32768 bytes each. >>>> >>>> What is the meaning of this exception (The total number of network >>>> buffers is currently set to 22773 of 32768 bytes each)? >>>> >>>> How to figure out a good combination of >>>> ('taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', >>>> and 'taskmanager.memory.network.max'.) >>>> for this issue ? >>>> >>>> >>>> Thanks & Regards, >>>> Pritam >>>> >>> >>> >>> -- >>> Best, >>> Hangxiang. >>> >> > > -- > Best, > Hangxiang. >