datariver created FLINK-31914:
---------------------------------
Summary: Failing to close FlinkKafkaInternalProducer created in
KafkaWriter with exactly-once semantic results in memory leak
Key: FLINK-31914
URL: https://issues.apache.org/jira/browse/FLINK-31914
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: datariver
Attachments: image-2023-04-24-16-11-22-251.png
Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing
will be used. KafkaWriter will create FlinkKafkaInternalProducer in the
initialization and snapshotState methods, but there is no place to close it. As
Checkpoints increase, Producers will continue to accumulate. Each Producer
maintains a Buffer, which will cause memory leaks and Job OOM.
By dumping an in-memory instance of Task Manager, you can see that there are a
lot of Producers:
!image-2023-04-24-16-11-22-251.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
