[ https://issues.apache.org/jira/browse/FLINK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524271#comment-17524271 ]
Anton Kalashnikov commented on FLINK-26762: ------------------------------------------- First of all, no rush is needed to implement this idea with overdraft-memory-size. Let's check your current implementation and then we can discuss whether it makes sense or not to use overdraft-memory-size rather than overdraft-buffers. ??Assuming that flatmap can produce 1Mb data at one time, they belong to different channels, do we apply for overdraft-buffer according to the actual data size??? I suppose we should consider the worst-case when all records will be sent to one channel(so for the user this property will be something like `max possible value at one time for one channel`) ??If the data is 100 bytes, do we apply for a memory segment of 100 bytes??? ??If the same Subpartition has 5 records, we will apply for 5 100byte memory segments. Maybe they don't perform well and take up more credit downstream??? We should not change our internal logic of memory segment size or buffers number. I suppose the implementation should just take the closest possible buffers number. For example, if the user request 100 Kb as overdraft but our memory segment size is 32Kb then we take 4 buffers (128 bytes) as overdraft. The second implementation idea we always request to set this value to a multiple of the buffer size(so 100 Kb is not possible but 128 or 96 is possible). It is one more reason why I don't still sure that this idea is good since on the one hand, it is transparent for the user(the user should not think about buffers number and etc.) but on the other hand, Flink should not execute this request blindly and it should adapt this value to internal implementation(which is not so transparent for user). But if we merge this idea with buffer debloating I think it will work pretty great. ??If = -1, flink automatically infers and will use the number of Subpartitions as overdraft-buffers. Not only can cover flatmap, window, join scenes, but also cover scenes broadcast by Watermark. It will make users more convenient to use.?? I don't really get what it means `the number of Subpartitions as overdraft-buffers`. `the number of Subpartitions ` is roughly equal to parallelism(depends on exchange) and I don't really understand how it correlates to overdraft? > Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being > blocked > ----------------------------------------------------------------------------------- > > Key: FLINK-26762 > URL: https://issues.apache.org/jira/browse/FLINK-26762 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Network > Affects Versions: 1.13.0, 1.14.0, 1.15.0 > Reporter: fanrui > Assignee: fanrui > Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > Attachments: image-2022-04-18-11-45-14-700.png, > image-2022-04-18-11-46-03-895.png > > > In some past JIRAs of Unaligned Checkpoint, the community has added the > recordWriter.isAvaliable() to reduce block for single record write. But for > large record, flatmap or broadcast watermark, they may need more buffer. > Can we add the overdraft buffer in BufferPool to reduce unaligned checkpoint > being blocked? > h2. Overdraft Buffer mechanism > Add the configuration of > 'taskmanager.network.memory.overdraft-buffers-per-gate=5'. > When requestMemory is called and the bufferPool is insufficient, the > bufferPool will allow the Task to overdraw up to 5 MemorySegments. And > bufferPool will be unavailable until all overdrawn buffers are consumed by > downstream tasks. Then the task will wait for bufferPool being available. > From the above, we have the following benefits: > * For scenarios that require multiple buffers, the Task releases the > Checkpoint lock, so the Unaligned Checkpoint can be completed quickly. > * We can control the memory usage to prevent memory leak. > * It just needs a litter memory, and can improve the stability of the Task > under back pressure. > * Users can increase the overdraft-buffers to adapt the scenarios that > require more buffers. > > Masters, please correct me if I'm wrong, thanks a lot. -- This message was sent by Atlassian Jira (v8.20.1#820001)