[
https://issues.apache.org/jira/browse/KAFKA-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095697#comment-18095697
]
Karsten Stöckmann commented on KAFKA-19992:
-------------------------------------------
[~muralibasani] [~mjsax] Is there anything I could help with here? We're having
a painful backfill phase right now with frequent OOM crashes and the resulting
rebalances and degraded performance, which reminded me of this issue. Please
let me know if there is anything I could do to assist.
> Allow to configure max memory for restore path
> ----------------------------------------------
>
> Key: KAFKA-19992
> URL: https://issues.apache.org/jira/browse/KAFKA-19992
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Muralidhar Basani
> Priority: Major
> Labels: kip
>
> Kafka Streams allows users to configure many different memory limits, to
> pro-actively manage memory usage to avoid OutOfMemory errors.
> However, on the restore code path, there is no such memory management. The
> `StoreChangelogReader` fetches a batch of message, and buffers them inside
> `ChangelogMetadata`, and there is no limit enforced on this buffer.
> We could either re-use the existing config for input topic buffer (`
> buffered.records.per.partition`) or introduce a new config (this would
> require a KIP though). – Even if we re-use the existing config, not 100% sure
> if we would need a KIP or not?
> KIP-1325:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1325%3A+Add+restore.buffered.records.per.partition+to+cap+Kafka+Streams+state-restore+memory]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
