Eric Chang created KAFKA-21587:
----------------------------------
Summary: Change the default socket buffer configs on broker and
clients to -1 (OS default)
Key: KAFKA-21587
URL: https://issues.apache.org/jira/browse/KAFKA-21587
Project: Kafka
Issue Type: Improvement
Components: clients, core, network
Reporter: Eric Chang
h3. Summary
Kafka sets fixed socket buffer sizes by default. On Linux, the kernel normally
adjusts the buffer size of each TCP connection by itself (this is called
auto-tuning). A fixed size turns this off for every Kafka connection. In my
test, one consumer reads about 2 times faster when both the broker and the
consumer use -1 (about 430 to 600 MB/s with the defaults, about 1000 to 1200
MB/s with -1). With 1 ms of added delay it reads 14 times faster, and with 10
ms 50 times faster. The cost is more kernel memory for connections where the
consumer stops reading. I propose to make -1 the default.
h3. Proposal
# Change the defaults in the table in Background to -1.
# Remove {{socket.send.buffer.bytes=102400}} and
{{socket.receive.buffer.bytes=102400}} from config/server.properties,
config/broker.properties, config/controller.properties and
docker/server.properties. These files set the values explicitly, so a change in
the code alone does not reach users who start from them.
# Add a note to the upgrade docs.
h3. Background
||Config||Component||Default||
|socket.send.buffer.bytes|broker|102400|
|socket.receive.buffer.bytes|broker|102400|
|replica.socket.receive.buffer.bytes|broker|65536|
|send.buffer.bytes|clients|131072|
|receive.buffer.bytes|consumer, admin client|65536|
|receive.buffer.bytes|producer, Connect, Streams|32768|
Kafka passes these values to {{Socket.setSendBufferSize}} and
{{Socket.setReceiveBufferSize}} (SO_SNDBUF and SO_RCVBUF). Without these
options, Linux sizes the buffers of each connection by itself, between the
limits in the {{tcp_rmem}} and {{tcp_wmem}} sysctls. It grows a buffer when the
connection needs it. When an application sets these options, Linux keeps the
buffer at that size and does not grow it. (Linux doubles the requested value,
so 102400 becomes 204800.) The buffer size limits how much data can be in
flight on the connection, so it limits throughput.
All these configs already accept -1 (KAFKA-724, KAFKA-3840 and follow-ups).
With -1, Kafka does not call the setters and the kernel sizes the buffers. Some
of Kafka's internal clients already use -1: the node-to-controller channel for
both buffers, and the KRaft client and the replica fetcher for the send buffer.
The current values do not come from a measurement, as far as I can find. In the
example config files, KAFKA-1531 (2014) changed both broker values from 1 MB to
102400 (send) and 65536 (receive), and a later commit (49d7f8ee1cd, KAFKA-1723,
2015) changed the receive value to 102400. I did not find the origin of the 100
KB default in the code. The 64 KB consumer default comes from KAFKA-3463
(2016), a mitigation for KAFKA-3135, which is still open.
h3. Measurement
Kafka trunk (commit e90d6f42c2), one broker (n2-standard-16) and one client
(c4-standard-8) in one Google Cloud zone. Linux kernel 7.0.0-1017-gcp (Ubuntu
24.04), with tcp_wmem max 4 MB and tcp_rmem max 32 MB (not changed). One
consumer reads a topic with 12 partitions with {{kafka-consumer-perf-test}}. I
added delay on the client with {{tc netem}}. Each case has 3 runs. The numbers
are the median in MB/s. The first comment has the setup, the commands and more
numbers.
||Added delay||Measured round-trip time||Both default||Only consumer -1||Only
broker -1||Both -1||
|none|0.28 ms|427|731|436|1002|
|1 ms|1.3 to 1.5 ms|58.6|144|58.2|829|
|10 ms|10.3 ms|3.2|17.7|3.2|162|
* -1 on both sides is about 2.3 times faster with no added delay, 14 times with
1 ms, and 50 times with 10 ms.
* Changing only the broker does nothing, because the consumer's receive buffer
is the smaller one and stays the limit.
* Changing only the consumer gives about half of the gain with no added delay
and about a tenth with 1 ms or 10 ms, because the limit moves to the broker's
send buffer. Both must change.
* With the defaults, throughput falls faster than the round-trip time grows
(about 7 and 133 times lower for a round-trip time about 5 and 37 times
higher). I did not investigate why.
* The default case differs between days (427 to about 600 MB/s with no added
delay) and I do not know why. All comparisons in this ticket are between runs
on the same day.
The docs already say that high-latency connections may need larger socket
buffers (docs/operations/datacenters.md).
h3. The kernel version matters with a fixed buffer
The Linux kernel commit
[697a6c8cec03|https://github.com/torvalds/linux/commit/697a6c8cec03] ("tcp:
increase the default TCP scaling ratio") names Kafka's default SO_RCVBUF=64k.
It reports an application timeout after a kernel change made the window of a
fixed buffer smaller. With a fixed buffer, Kafka depends on how each kernel
version turns a buffer size into a window. With -1 the kernel can grow the
buffer, so this matters much less.
h3. Cost
With -1 a connection can use more kernel memory, up to the tcp_wmem or tcp_rmem
maximum. I measured the kernel TCP memory on the broker host ({{TCP: mem}} in
/proc/net/sockstat) with many consumers in three states. "Stalled" means that
each consumer read one fetch response, sent the next fetch, and then stopped
reading. I used {{max.partition.fetch.bytes}} = 8 MB for this state, so that
one response is larger than the largest send buffer. This is the worst case per
connection.
||State||Consumers||Defaults on both sides||-1 on both sides||
|Idle (connected, no fetch)|2000|0 MB|0 MB|
|All reading|2000|307 MB|241 MB|
|Stalled|500|99 MB|1469 MB|
The idle row is the change from before the consumers connected (with -1 the
counter was 3 MB before and after). The other rows are the reading itself.
* Idle connections cost the same. 2000 connections that all read also cost
about the same, but each one was slow in this test (about 1.6 MB/s, because the
client CPU was the limit). I did not measure many fast connections.
* A stalled fetch connection holds about 2.9 MB on the broker with -1 and about
0.2 MB with the defaults. So a broker with many slow or stalled consumers uses
much more kernel memory with -1. At this rate, about 1400 stalled fetch
connections would reach the TCP memory pressure threshold of this broker host
(the second value of the {{net.ipv4.tcp_mem}} sysctl, a threshold for all TCP
sockets of the host together; about 4 GB on this host with 62 GB of memory). I
did not measure that.
* The client side has a cost too, and its limit per connection is higher
(tcp_rmem maximum 32 MB, against 4 MB for tcp_wmem). The client host with 500
stalled consumers used 1962 MB with -1 (66 MB with the defaults). It reached
the TCP memory pressure threshold of that host (about 1.9 GB; the host has 29
GB of memory).
* I have no stalled number for 2000 consumers, because my test tool ran out of
heap. See the first comment.
Users who want the old behaviour can set the old values explicitly.
h3. Not measured
* Produce direction (producer send buffer, broker receive buffer) and follower
fetch. The same mechanism applies, but I have no numbers.
* Operating systems other than Linux.
* Only one broker, one consumer and one record size were tested.
* KAFKA-3135. I did not test if the pause described there can happen with -1.
On Linux with -1, the receive buffer starts at the tcp_rmem default (131072 on
the test VMs), which is the same size as today's 65536 after the kernel doubles
it.
* Brokers in containers with a memory limit. Socket buffer memory counts toward
the memory limit of the cgroup (cgroup v2).
h3. Questions
* Is there a reason to keep the fixed defaults that I have missed?
* Does this change need a KIP? I am happy to write one.
Related: KAFKA-16496 proposes -1 for the client receive.buffer.bytes only.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)