Hi Divij, Thanks for raising this. The valid minimum value 1 for `segment.ms` is completely unreasonable. Similarly for `segment.bytes`, `metadata.log.segment.ms`, `metadata.log.segment.bytes`.
In addition to that, there are also some config default values we'd like to propose to change in v4.0. We can collect more comments from the community, and come out with a KIP for them. 1. num.recovery.threads.per.data.dir: The current default value is 1. But the log recovery is happening before brokers are in ready state, which means, we should use all the available resource to speed up the log recovery to bring the broker to ready state soon. Default value should be... maybe 4 (to be decided)? 2. Other configs might be able to consider to change the default, but open for comments: 2.1. num.replica.fetchers: default is 1, but that's not enough when there are multiple partitions in the cluster 2.2. `socket.send.buffer.bytes`/`socket.receive.buffer.bytes`: Currently, we set 100kb as default value, but that's not enough for high-speed network. Thank you. Luke On Tue, Mar 12, 2024 at 1:32 AM Divij Vaidya <divijvaidy...@gmail.com> wrote: > Hey folks > > Before I file a KIP to change this in 4.0, I wanted to understand the > historical context for the value of the following setting. > > Currently, segment.ms minimum threshold is set to 1ms [1]. > > Segments are expensive. Every segment uses multiple file descriptors and > it's easy to run out of OS limits when creating a large number of segments. > Large number of segments also delays log loading on startup because of > expensive operations such as iterating through all directories & > conditionally loading all producer state. > > I am currently not aware of a reason as to why someone might want to work > with a segment.ms of less than ~10s (number chosen arbitrary that looks > sane) > > What was the historical context of setting the minimum threshold to 1ms for > this setting? > > [1] https://kafka.apache.org/documentation.html#topicconfigs_segment.ms > > -- > Divij Vaidya >
