[
https://issues.apache.org/jira/browse/KAFKA-21083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sanghyeok An updated KAFKA-21083:
---------------------------------
Description:
*Description*
When a producer sends a record batch that exceeds the effective broker-side
batch size limit, the broker rejects it with MESSAGE_TOO_LARGE. This limit
applies to the record batch, after compression if enabled.
The producer currently splits the rejected batch using the configured
batch.size. If an already-split batch is rejected again, it reduces the split
target to half the estimated batch size, bounded below by the largest record
size.
This approach allows progressively smaller batches, but the producer does not
know the limit enforced by the broker. Consequently, split batches may still
exceed that limit, resulting in additional rejected requests, decompression,
batch rebuilding, and retransmission.
*Proposed improvement*
Include the effective maximum allowed record batch size in the
MESSAGE_TOO_LARGE response for the affected partition. The value should reflect
the applicable topic-level max.message.bytes, including the broker default when
no topic override is configured.
The producer could use this value to choose a split target that respects both
its configured batching target and the reported limit, reducing the need to
discover an acceptable size through repeated failures.
For example, with a producer batch.size of 8 MiB and an effective broker limit
of 1 MiB, the producer could use the reported limit immediately when splitting,
instead of requiring further rejections to progressively reduce the target.
{*}Considerations{*}{*}{{*}}
* Preserve the existing splitting behavior when the response does not provide
a limit. (for compatiability for old version broker and producer)
* Treat the reported limit as a hint: configuration changes and compression
estimation errors can still cause subsequent rejections.
* Preserve failure handling for a single-record batch that exceeds the limit;
batch splitting does not divide an individual record.
*Related work*
KIP-126 identified obtaining the broker’s maximum message size as a future
direction.
(https://cwiki.apache.org/confluence/spaces/KAFKA/pages/68715855/KIP-126+-+Allow+KafkaProducer+to+split+and+resend+oversized+batches.)
was:
*Description*
When a producer sends a record batch that exceeds the effective broker-side
batch size limit, the broker rejects it with MESSAGE_TOO_LARGE. This limit
applies to the record batch, after compression if enabled.
The producer currently splits the rejected batch using the configured
batch.size. If an already-split batch is rejected again, it reduces the split
target to half the estimated batch size, bounded below by the largest record
size.
This approach allows progressively smaller batches, but the producer does not
know the limit enforced by the broker. Consequently, split batches may still
exceed that limit, resulting in additional rejected requests, decompression,
batch rebuilding, and retransmission.
*Proposed improvement*
Include the effective maximum allowed record batch size in the
MESSAGE_TOO_LARGE response for the affected partition. The value should reflect
the applicable topic-level max.message.bytes, including the broker default when
no topic override is configured.
The producer could use this value to choose a split target that respects both
its configured batching target and the reported limit, reducing the need to
discover an acceptable size through repeated failures.
For example, with a producer batch.size of 8 MiB and an effective broker limit
of 1 MiB, the producer could use the reported limit immediately when splitting,
instead of requiring further rejections to progressively reduce the target.
{*}Considerations{*}{*}{*}
* Preserve the existing splitting behavior when the response does not provide
a limit. (for compatiability for old version broker and producer)
* Treat the reported limit as a hint: configuration changes and compression
estimation errors can still cause subsequent rejections.
* Preserve failure handling for a single-record batch that exceeds the limit;
batch splitting does not divide an individual record.
*Related work*
KIP-126 identified obtaining the broker’s maximum message size as a future
direction.
> Return the effective batch size limit in MESSAGE_TOO_LARGE responses to
> improve producer batch splitting
> ----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21083
> URL: https://issues.apache.org/jira/browse/KAFKA-21083
> Project: Kafka
> Issue Type: Improvement
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: broker, needs-kip, producer
>
> *Description*
> When a producer sends a record batch that exceeds the effective broker-side
> batch size limit, the broker rejects it with MESSAGE_TOO_LARGE. This limit
> applies to the record batch, after compression if enabled.
>
> The producer currently splits the rejected batch using the configured
> batch.size. If an already-split batch is rejected again, it reduces the split
> target to half the estimated batch size, bounded below by the largest record
> size.
>
> This approach allows progressively smaller batches, but the producer does not
> know the limit enforced by the broker. Consequently, split batches may still
> exceed that limit, resulting in additional rejected requests, decompression,
> batch rebuilding, and retransmission.
>
> *Proposed improvement*
> Include the effective maximum allowed record batch size in the
> MESSAGE_TOO_LARGE response for the affected partition. The value should
> reflect the applicable topic-level max.message.bytes, including the broker
> default when no topic override is configured.
>
> The producer could use this value to choose a split target that respects both
> its configured batching target and the reported limit, reducing the need to
> discover an acceptable size through repeated failures.
>
> For example, with a producer batch.size of 8 MiB and an effective broker
> limit of 1 MiB, the producer could use the reported limit immediately when
> splitting, instead of requiring further rejections to progressively reduce
> the target.
>
> {*}Considerations{*}{*}{{*}}
> * Preserve the existing splitting behavior when the response does not
> provide a limit. (for compatiability for old version broker and producer)
> * Treat the reported limit as a hint: configuration changes and compression
> estimation errors can still cause subsequent rejections.
> * Preserve failure handling for a single-record batch that exceeds the
> limit; batch splitting does not divide an individual record.
>
> *Related work*
> KIP-126 identified obtaining the broker’s maximum message size as a future
> direction.
> (https://cwiki.apache.org/confluence/spaces/KAFKA/pages/68715855/KIP-126+-+Allow+KafkaProducer+to+split+and+resend+oversized+batches.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
