[
https://issues.apache.org/jira/browse/KAFKA-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hejie updated KAFKA-17918:
--------------------------
Description:
In Kafka 3.5.0, producers connect to the Kafka server and use the callback
method to report messages. The process memory overflows. According to the
stack, a large number of callback methods are stacked in the RecordAccumulator
object, occupying 500 MB memory.
The service logic is as follows: When each message is reported, the message is
placed in the callback method for log printing. The message usage of each
producerBatch is 0.5 MB. (The value of batch.size is 500 KB.) However, the
corresponding callback method in each producbatch occupies 10 MB.
The data processing capability of the current Kafka server is poor. The Kafka
server can process only about 10 messages within 8 seconds.
Do you have the problem that the buffer pool is released but the callback is
not released after the scenario message is reported successfully?
!image-2024-11-01-14-51-23-273.png!
!image-2024-11-01-14-53-27-291.png!
was:
In Kafka 3.5.0, producers connect to the Kafka server and use the callback
method to report messages. The process memory overflows. According to the
stack, a large number of callback methods are stacked in the RecordAccumulator
object, occupying 500 MB memory.
The service logic is as follows: When each message is reported, the message is
placed in the callback method for log printing. The message usage of each
producerBatch is 0.5 MB. (The value of batch.size is 500 KB.) However, the
corresponding callback method in each producbatch occupies 10 MB.
Check whether the scenario message is reported successfully, the buffer pool is
released, but the callback is not released.
!image-2024-11-01-14-51-23-273.png!
!image-2024-11-01-14-53-27-291.png!
> The Kafka producer's RecordAccumulator object stores more callback methods
> than messages.
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-17918
> URL: https://issues.apache.org/jira/browse/KAFKA-17918
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 3.5.0
> Reporter: hejie
> Priority: Major
> Attachments: image-2024-11-01-14-51-23-273.png,
> image-2024-11-01-14-53-27-291.png
>
>
> In Kafka 3.5.0, producers connect to the Kafka server and use the callback
> method to report messages. The process memory overflows. According to the
> stack, a large number of callback methods are stacked in the
> RecordAccumulator object, occupying 500 MB memory.
> The service logic is as follows: When each message is reported, the message
> is placed in the callback method for log printing. The message usage of each
> producerBatch is 0.5 MB. (The value of batch.size is 500 KB.) However, the
> corresponding callback method in each producbatch occupies 10 MB.
>
> The data processing capability of the current Kafka server is poor. The Kafka
> server can process only about 10 messages within 8 seconds.
>
> Do you have the problem that the buffer pool is released but the callback is
> not released after the scenario message is reported successfully?
>
>
> !image-2024-11-01-14-51-23-273.png!
>
> !image-2024-11-01-14-53-27-291.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
