Rajini Sivaram created KAFKA-7639:
-------------------------------------
Summary: Read one request at a time from socket to reduce broker
memory usage
Key: KAFKA-7639
URL: https://issues.apache.org/jira/browse/KAFKA-7639
Project: Kafka
Issue Type: Improvement
Components: network
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
Fix For: 2.2.0
Broker's Selector currently reads all requests available on the socket when the
socket is ready for read. These are queued up as staged receives. We mute the
channel and stop reading any more data until all the staged requests are
processed. This behaviour is slightly inconsistent since for the initial read
we drain the socket buffer, allowing it to get filled up again, but if data
arrives slighly after the initial read, then we dont read from the socket
buffer until pending requests are processed.
To avoid holding onto requests for longer than required, we should read one
request at a time even if more data is available in the socket buffer. This is
especially useful for produce requests which may be large and may take long to
process.
Note that with the default socket read buffer size of 100K, this is not a
critical issue. But with larger socket buffers, this could result in excessive
memory usage if a lot of produce requests are buffered in the broker and the
producer times out, reconnects and sends more data before broker has cleared
older requests. By reading one-at-a-time, we reduce the amount of time the
broker holds onto memory for each request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
