[
https://issues.apache.org/jira/browse/KAFKA-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014161#comment-17014161
]
ASF GitHub Bot commented on KAFKA-7639:
---------------------------------------
rajinisivaram commented on pull request #5920: KAFKA-7639: Read one request at
a time from socket to avoid OOM
URL: https://github.com/apache/kafka/pull/5920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Read one request at a time from socket to reduce broker memory usage
> --------------------------------------------------------------------
>
> Key: KAFKA-7639
> URL: https://issues.apache.org/jira/browse/KAFKA-7639
> Project: Kafka
> Issue Type: Improvement
> Components: network
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
> Priority: Major
>
> Broker's Selector currently reads all requests available on the socket when
> the socket is ready for read. These are queued up as staged receives. We mute
> the channel and stop reading any more data until all the staged requests are
> processed. This behaviour is slightly inconsistent since for the initial read
> we drain the socket buffer, allowing it to get filled up again, but if data
> arrives slighly after the initial read, then we dont read from the socket
> buffer until pending requests are processed.
> To avoid holding onto requests for longer than required, we should read one
> request at a time even if more data is available in the socket buffer. This
> is especially useful for produce requests which may be large and may take
> long to process.
> Note that with the default socket read buffer size of 100K, this is not a
> critical issue. But with larger socket buffers, this could result in
> excessive memory usage if a lot of produce requests are buffered in the
> broker and the producer times out, reconnects and sends more data before
> broker has cleared older requests. By reading one-at-a-time, we reduce the
> amount of time the broker holds onto memory for each request.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
