[
https://issues.apache.org/jira/browse/KAFKA-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram resolved KAFKA-7639.
-----------------------------------
Fix Version/s: 2.5.0
Reviewer: Jun Rao
Resolution: Fixed
> Read one request at a time from socket to reduce broker memory usage
> --------------------------------------------------------------------
>
> Key: KAFKA-7639
> URL: https://issues.apache.org/jira/browse/KAFKA-7639
> Project: Kafka
> Issue Type: Improvement
> Components: network
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
> Priority: Major
> Fix For: 2.5.0
>
>
> Broker's Selector currently reads all requests available on the socket when
> the socket is ready for read. These are queued up as staged receives. We mute
> the channel and stop reading any more data until all the staged requests are
> processed. This behaviour is slightly inconsistent since for the initial read
> we drain the socket buffer, allowing it to get filled up again, but if data
> arrives slighly after the initial read, then we dont read from the socket
> buffer until pending requests are processed.
> To avoid holding onto requests for longer than required, we should read one
> request at a time even if more data is available in the socket buffer. This
> is especially useful for produce requests which may be large and may take
> long to process.
> Note that with the default socket read buffer size of 100K, this is not a
> critical issue. But with larger socket buffers, this could result in
> excessive memory usage if a lot of produce requests are buffered in the
> broker and the producer times out, reconnects and sends more data before
> broker has cleared older requests. By reading one-at-a-time, we reduce the
> amount of time the broker holds onto memory for each request.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
