[
https://issues.apache.org/jira/browse/KAFKA-19763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028315#comment-18028315
]
Kamal Chandraprakash edited comment on KAFKA-19763 at 10/8/25 9:23 AM:
-----------------------------------------------------------------------
[~showuon]
Able to reproduce the issue consistently. Uploaded the
RemoteReadMemoryLeakReproducer. The leak was due to that the
DelayedRemoteFetchPurgatory holding the references of previously completed
DelayedRemoteFetch objects. DelayedRemoteFetch contains the RemoteReadResult
internally.
> Have you figured it out where do we leak the memory?
In a given FETCH request, if 1 out of 5 partition, read the data from local
log, then the watcherKey for that partition holds the reference of the
DelayedRemoteFetch in the purgatory; if there are no other remote-read happens
for that partition, then it won't get removed until the reaper thread cleans it
up after the purgeInterval (entries) of 1000.
{code:java}
% sh kafka-topics.sh --create --topic apple --partitions 5 --replication-factor
1 --bootstrap-server localhost:9092 --config remote.storage.enable=true
--config local.retention.ms=60000 --config retention.ms=7200000 --config
segment.bytes=104857600 --config file.delete.delay.ms=1000
% for i in `seq 1 100`; do echo $i; sleep 1; sh kafka-producer-perf-test.sh
--topic apple --num-records 1200000000 --record-size 1024 --throughput 1000
--producer-props bootstrap.servers=localhost:9092; done {code}
was (Author: ckamal):
[~showuon]
Able to reproduce the issue consistently. Uploaded the
RemoteReadMemoryLeakReproducer. The leak was due to that the
DelayedRemoteFetchPurgatory holding the references of previously completed
DelayedRemoteFetch objects. DelayedRemoteFetch contains the RemoteReadResult
internally.
> Have you figured it out where do we leak the memory?
In a given FETCH request, if 1 out of 5 partition, read the data from local
log, then the watcherKey for that partition holds the reference of the
DelayedRemoteFetch in the purgatory; if there are no other remote-read happens
for that partition, then it won't get removed until the reaper thread cleans it
up after the purgeInterval (entries) of 1000.
{code:java}
% sh kafka-topics.sh --create --topic apple --partitions 5 --replication-factor
1 --bootstrap-server localhost:9092 --config remote.storage.enable=true
--config local.retention.ms=60000 --config retention.ms=720000 --config
segment.bytes=104857600 --config file.delete.delay.ms=1000
% for i in `seq 1 100`; do echo $i; sleep 1; sh kafka-producer-perf-test.sh
--topic apple --num-records 1200000000 --record-size 1024 --throughput 1000
--producer-props bootstrap.servers=localhost:9092; done {code}
> Parallel remote reads causes memory leak in broker
> --------------------------------------------------
>
> Key: KAFKA-19763
> URL: https://issues.apache.org/jira/browse/KAFKA-19763
> Project: Kafka
> Issue Type: Task
> Reporter: Kamal Chandraprakash
> Assignee: Kamal Chandraprakash
> Priority: Blocker
> Fix For: 4.2.0
>
> Attachments: RemoteReadMemoryLeakReproducer.java, Screenshot
> 2025-10-07 at 8.25.45 PM.png
>
>
> Broker heap memory gets filled up and throws OOM error when remote reads are
> triggered for multiple partitions within a FETCH request.
> Steps to reproduce:
> 1. Start a one node broker and configure LocalTieredStorage as remote
> storage.
> 2. Create a topic with 5 partitions.
> 3. Produce message and ensure that few segments are uploaded to remote.
> 4. Start a consumer to read from those 5 partitions. Seek the offset to
> beginning for 4 partitions and to end for 1 partition. This is to simulate
> that the FETCH request read from both remote-log and local-log.
> 5. The broker crashes with the OOM error.
> 6. The DelayedRemoteFetch / RemoteLogReadResult references are being held by
> the purgatory, so the broker crashes.
> cc [~showuon]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
