Sean Quah created KAFKA-19716:
---------------------------------
Summary: OOM when loading large uncompacted __consumer_offsets
partitions with transactional workload
Key: KAFKA-19716
URL: https://issues.apache.org/jira/browse/KAFKA-19716
Project: Kafka
Issue Type: Bug
Components: group-coordinator
Reporter: Sean Quah
Assignee: Sean Quah
When loading a large poorly compacted __consumer_offsets partition with a
transactional workload (alternating transaction offset commits and commit
markers), we create and discard lots of TimelineHashMaps. These accumulate in
the SnapshotRegistry's rollback mechanism. Overall, memory usage is linear in
the size of the partition.
We can commit the snapshot every N offsets in {{{}CoordinatorLoaderImpl{}}}, to
allow the memory to be reclaimed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
