Thanks for the discussion folks. I have started a KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations to keep track of the changes that we are discussion. Please consider this as a collaborative work-in-progress KIP and once it is ready to be published, we can start a discussion thread on it.
I am also going to start a thread to solicit feedback from users@ mailing list as well. -- Divij Vaidya On Wed, Mar 13, 2024 at 12:55 PM Christopher Shannon < christopher.l.shan...@gmail.com> wrote: > I think it's a great idea to raise a KIP to look at adjusting defaults and > minimum/maximum config values for version 4.0. > > As pointed out, the minimum values for segment.ms and segment.bytes don't > make sense and would probably bring down a cluster pretty quickly if set > that low, so version 4.0 is a good time to fix it and to also look at the > other configs as well for adjustments. > > On Wed, Mar 13, 2024 at 4:39 AM Sergio Daniel Troiano > <sergio.troi...@adevinta.com.invalid> wrote: > > > hey guys, > > > > Regarding to num.recovery.threads.per.data.dir: I agree, in our company > we > > use the number of vCPUs to do so as this is not competing with ready > > cluster traffic. > > > > > > On Wed, 13 Mar 2024 at 09:29, Luke Chen <show...@gmail.com> wrote: > > > > > Hi Divij, > > > > > > Thanks for raising this. > > > The valid minimum value 1 for `segment.ms` is completely unreasonable. > > > Similarly for `segment.bytes`, `metadata.log.segment.ms`, > > > `metadata.log.segment.bytes`. > > > > > > In addition to that, there are also some config default values we'd > like > > to > > > propose to change in v4.0. > > > We can collect more comments from the community, and come out with a > KIP > > > for them. > > > > > > 1. num.recovery.threads.per.data.dir: > > > The current default value is 1. But the log recovery is happening > before > > > brokers are in ready state, which means, we should use all the > available > > > resource to speed up the log recovery to bring the broker to ready > state > > > soon. Default value should be... maybe 4 (to be decided)? > > > > > > 2. Other configs might be able to consider to change the default, but > > open > > > for comments: > > > 2.1. num.replica.fetchers: default is 1, but that's not enough when > > > there are multiple partitions in the cluster > > > 2.2. `socket.send.buffer.bytes`/`socket.receive.buffer.bytes`: > > > Currently, we set 100kb as default value, but that's not enough for > > > high-speed network. > > > > > > Thank you. > > > Luke > > > > > > > > > On Tue, Mar 12, 2024 at 1:32 AM Divij Vaidya <divijvaidy...@gmail.com> > > > wrote: > > > > > > > Hey folks > > > > > > > > Before I file a KIP to change this in 4.0, I wanted to understand the > > > > historical context for the value of the following setting. > > > > > > > > Currently, segment.ms minimum threshold is set to 1ms [1]. > > > > > > > > Segments are expensive. Every segment uses multiple file descriptors > > and > > > > it's easy to run out of OS limits when creating a large number of > > > segments. > > > > Large number of segments also delays log loading on startup because > of > > > > expensive operations such as iterating through all directories & > > > > conditionally loading all producer state. > > > > > > > > I am currently not aware of a reason as to why someone might want to > > work > > > > with a segment.ms of less than ~10s (number chosen arbitrary that > > looks > > > > sane) > > > > > > > > What was the historical context of setting the minimum threshold to > 1ms > > > for > > > > this setting? > > > > > > > > [1] > > https://kafka.apache.org/documentation.html#topicconfigs_segment.ms > > > > > > > > -- > > > > Divij Vaidya > > > > > > > > > >
