[
https://issues.apache.org/jira/browse/FLINK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524922#comment-17524922
]
Anton Kalashnikov commented on FLINK-26762:
-------------------------------------------
??Does this mean that the effective size is only 400 bytes???
??Can we still apply for an overdraft buffer after the data of the fifth
channel is generated???
Actually it should work the same as if we use overdraft-buffers. In your
example, let's suppose we configure overdraft-buffers = 4. How should it work?.
So in my opinion it can be two possible implementations. if this configuration
determines the total size/buffers for all subpartitions then the answer is the
`yes` and `no` correspondingly. But if this configuration will be for each
subpartition, then every subpartition has 128Kb or 4 buffers.
> Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being
> blocked
> -----------------------------------------------------------------------------------
>
> Key: FLINK-26762
> URL: https://issues.apache.org/jira/browse/FLINK-26762
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Affects Versions: 1.13.0, 1.14.0, 1.15.0
> Reporter: fanrui
> Assignee: fanrui
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
> Attachments: image-2022-04-18-11-45-14-700.png,
> image-2022-04-18-11-46-03-895.png
>
>
> In some past JIRAs of Unaligned Checkpoint, the community has added the
> recordWriter.isAvaliable() to reduce block for single record write. But for
> large record, flatmap or broadcast watermark, they may need more buffer.
> Can we add the overdraft buffer in BufferPool to reduce unaligned checkpoint
> being blocked?
> h2. Overdraft Buffer mechanism
> Add the configuration of
> 'taskmanager.network.memory.overdraft-buffers-per-gate=5'.
> When requestMemory is called and the bufferPool is insufficient, the
> bufferPool will allow the Task to overdraw up to 5 MemorySegments. And
> bufferPool will be unavailable until all overdrawn buffers are consumed by
> downstream tasks. Then the task will wait for bufferPool being available.
> From the above, we have the following benefits:
> * For scenarios that require multiple buffers, the Task releases the
> Checkpoint lock, so the Unaligned Checkpoint can be completed quickly.
> * We can control the memory usage to prevent memory leak.
> * It just needs a litter memory, and can improve the stability of the Task
> under back pressure.
> * Users can increase the overdraft-buffers to adapt the scenarios that
> require more buffers.
>
> Masters, please correct me if I'm wrong, thanks a lot.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
