[
https://issues.apache.org/jira/browse/KAFKA-20917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Wang updated KAFKA-20917:
----------------------------
Description:
After upgrading producer client from 2.9 to 3.9, we noticed produce latency
increase, especially for ack-all producer, e.g. P50 latency increase from 8ms
to 13ms, ~60% increase.
The profiling shows ready call is hot. The 3.9 client include more checks and
calculations in RecordAccumulator.ready -> partitionReady method.
*2.9 client profiling*
!Screenshot 2026-08-19 at 2.29.20 PM.png|width=859,height=225!
low cluster.leaderFor call.
!Screenshot 2026-08-19 at 2.34.29 PM.png|width=863,height=194!
no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
*3.9 client profiling*
heavy metadataSnapshot.leaderEpochFor and cluster.leaderFor calls
!Screenshot 2026-08-19 at 2.31.36 PM.png|width=863,height=260!
!Screenshot 2026-08-19 at 2.38.46 PM.png|width=866,height=218!
!Screenshot 2026-08-19 at 2.40.28 PM.png|width=864,height=260!
3.9 with fix
was:
After upgrading producer client from 2.9 to 3.9, we noticed produce latency
increase, especially for ack-all producer, e.g. P50 latency increase from 8ms
to 13ms, ~60% increase.
The profiling shows ready call is hot. The 3.9 client include more checks and
calculations in RecordAccumulator.ready -> partitionReady method.
*2.9 client profiling*
!Screenshot 2026-08-19 at 2.29.20 PM.png|width=859,height=225!
low cluster.leaderFor call.
!Screenshot 2026-08-19 at 2.34.29 PM.png|width=863,height=194!
no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
*3.9 client profiling* (heavy metadataSnapshot.leaderEpochFor and
cluster.leaderFor calls)
!Screenshot 2026-08-19 at 2.31.36 PM.png|width=863,height=260!
We found out the linked PRs address performance issue and after deploying it to
our ack-all cluster (with adaptive partitioning disabled), the latency became
similar as 2.9 client.
> [Java-client] Improve RecordAccumulator.ready performance
> ---------------------------------------------------------
>
> Key: KAFKA-20917
> URL: https://issues.apache.org/jira/browse/KAFKA-20917
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Reporter: Xi Wang
> Priority: Major
> Attachments: Screenshot 2026-08-19 at 2.29.20 PM.png, Screenshot
> 2026-08-19 at 2.31.36 PM.png, Screenshot 2026-08-19 at 2.34.29 PM.png,
> Screenshot 2026-08-19 at 2.38.46 PM.png, Screenshot 2026-08-19 at 2.40.28
> PM.png
>
>
> After upgrading producer client from 2.9 to 3.9, we noticed produce latency
> increase, especially for ack-all producer, e.g. P50 latency increase from 8ms
> to 13ms, ~60% increase.
> The profiling shows ready call is hot. The 3.9 client include more checks and
> calculations in RecordAccumulator.ready -> partitionReady method.
> *2.9 client profiling*
> !Screenshot 2026-08-19 at 2.29.20 PM.png|width=859,height=225!
> low cluster.leaderFor call.
> !Screenshot 2026-08-19 at 2.34.29 PM.png|width=863,height=194!
> no metadataSnapshot.leaderEpochFor call (introduced in 3.9)
> *3.9 client profiling*
> heavy metadataSnapshot.leaderEpochFor and cluster.leaderFor calls
> !Screenshot 2026-08-19 at 2.31.36 PM.png|width=863,height=260!
> !Screenshot 2026-08-19 at 2.38.46 PM.png|width=866,height=218!
> !Screenshot 2026-08-19 at 2.40.28 PM.png|width=864,height=260!
> 3.9 with fix
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
