Hi devs: I would like to start a discussion of PIP-1: Improve Shared Writer Buffer Pool For Sink [1]. Currently Paimon sink task creates a heap memory pool which is shared by writers. When there are multiple tasks in an Executor, it may cause FullGC, performance issues and even OOM.
This PIP aims to improve the buffer pool for writers in Paimon tasks. Paimon tasks can create memory pools based on Executor Memory which will be managed by Executor, such as Managed Memory in Flink TaskManager. It will improve the stability and performance of sinks by managing writer buffers for multiple tasks through Executor. Looking forward to your feedback, thanks. [1] https://cwiki.apache.org/confluence/display/PAIMON/PIP-1%3A+Improve+Shared+Writer+Buffer+Pool+For+Sink Best, Shammon FY
