Katie Liu created SAMZA-2785:
--------------------------------
Summary: Reuse of KafkaCheckpointManager causes memory leak
Key: SAMZA-2785
URL: https://issues.apache.org/jira/browse/SAMZA-2785
Project: Samza
Issue Type: Bug
Reporter: Katie Liu
Calling start() and stop() multiple times on the same KafkaCheckpointManager,
while stopConsumerAfterFirstRead == true, causes the SystemConsumer left
unclosed. The unclosed SystemConsumer can cause memory leaks in some
implementations.
Evidence:
In production logs, SystemConsumer was started 1741 times, but only closed 14
times.
{code:java}
[katxxxx@xxxxxx xxxx]$ grep "Starting the checkpoint SystemConsumer from oldest
offset" xxxxxx.log|wc -l
1741
[katxxxxx@xxxxxx xxxx]$ grep "Stopping system consumer" xxxxxxxxxx.log |wc -l
14 {code}
We also have a heap dump of KafkaSystemConsumer taking up 8Gbs of memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
