[
https://issues.apache.org/jira/browse/NIFI-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wojciech Trefon reassigned NIFI-16554:
--------------------------------------
Assignee: Wojciech Trefon
> Expose Kafka consumer fetch size properties in Kafka3ConnectionService
> ----------------------------------------------------------------------
>
> Key: NIFI-16554
> URL: https://issues.apache.org/jira/browse/NIFI-16554
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Wojciech Trefon
> Assignee: Wojciech Trefon
> Priority: Major
>
> The Kafka consumer settings {{max.partition.fetch.bytes}} and
> {{fetch.max.bytes}} control how much data a single poll can return, but
> Kafka3ConnectionService (and AmazonMSKConnectionService, which extends it)
> only supports them as dynamic properties, which users have to know about.
> These settings matter most for ConsumeKafka with Schema Conflict Resolution
> set to Continue with Merged Schema. In that mode each concurrent task holds
> all records from a poll in memory until they are written. When consumers fall
> behind, every poll is full, so heap usage grows with the number of tasks and
> the fetch size. When each consumer is assigned only a few partitions, the
> per-partition limit, not {{fetch.max.bytes}}, bounds the size of a poll. In a
> test with a large backlog and several ConsumeKafka processors, lowering
> {{max.partition.fetch.bytes}} from the default 1 MiB to 256 KiB eliminated
> repeated full garbage collections and reduced peak heap usage, while lowering
> {{fetch.max.bytes}} had no effect.
> Proposal:
> * Add {{max.partition.fetch.bytes}} and {{fetch.max.bytes}} as regular
> properties of Kafka3ConnectionService, following the existing
> {{max.poll.records}} property.
> * Keep the Kafka client defaults (1048576 and 52428800 bytes), so existing
> behavior does not change.
> * Use the Kafka property names, so flows that already set these as dynamic
> properties map onto the new properties without migration.
> * Describe in the ConsumeKafka additional details how Continue with Merged
> Schema relates to memory usage.
> Related to NIFI-14271.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
