RE: broker properties explanations
It's really help me, thank Thomas ! Just a jot of another question, I don't understand correctly this property: leader.imbalance.per.broker.percentage The ratio of leader imbalance allowed per broker. The controller would trigger a leader balance if it goes above this value per broker. The value is specified in percentage. int 10 How the ratio is calculated ? The imbalance broker compare to what ? Moreover the 10% by default value is very small, no ? Thank again Thomas, Adrien De : Thomas Aley Envoyé : mercredi 21 février 2018 09:24:13 À : users@kafka.apache.org Objet : Re: broker properties explanations Hi Adrien, log.dirs exists to facilitate multiple data directories which allows more than one disk to be used without the need for RAID. This increases throughput but beware of naive load balancing that may fill up one disk way before another. When log.flush.interval.ms is null the log.flush.interval.messages property is not used. With default settings, messages are written to disk immediately. Hope this helps. Tom Aley thomas.a...@ibm.com From: adrien ruffie To: "users@kafka.apache.org" Date: 20/02/2018 20:46 Subject: broker properties explanations Hello all, after reading several properties in Kafka documentations, I asked mysleft some questions ... these 2 following options are available: log.dir The directory in which the log data is kept (supplemental for log.dirs property)string /tmp/kafka-logs high log.dirsThe directories in which the log data is kept. If not set, the value in log.dir is used But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's difference in usage ? Also I noticed that the "data" of logs partition and also the "application" logs of Kafka are written in the same directory. Is a mistake of my part ? Because it's very strange for me to log error, debug, warn application message in the same location of my data ... After that I don't very understand why the log.flush.interval.messages have a so big default value ??? log.flush.interval.messages The number of messages accumulated on a log partition before messages are flushed to disk long 9223372036854775807 And the log.flush.interval.ms is by default null ... ? It means that until there are so many messages (9223372036854775807) in my topics, they will not be flushed to disk ? It can be very long for a default value Best regards, Adrien Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: broker properties explanations
Hi Adrien, log.dirs exists to facilitate multiple data directories which allows more than one disk to be used without the need for RAID. This increases throughput but beware of naive load balancing that may fill up one disk way before another. When log.flush.interval.ms is null the log.flush.interval.messages property is not used. With default settings, messages are written to disk immediately. Hope this helps. Tom Aley thomas.a...@ibm.com From: adrien ruffie To: "users@kafka.apache.org" Date: 20/02/2018 20:46 Subject: broker properties explanations Hello all, after reading several properties in Kafka documentations, I asked mysleft some questions ... these 2 following options are available: log.dir The directory in which the log data is kept (supplemental for log.dirs property)string /tmp/kafka-logs high log.dirsThe directories in which the log data is kept. If not set, the value in log.dir is used But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's difference in usage ? Also I noticed that the "data" of logs partition and also the "application" logs of Kafka are written in the same directory. Is a mistake of my part ? Because it's very strange for me to log error, debug, warn application message in the same location of my data ... After that I don't very understand why the log.flush.interval.messages have a so big default value ??? log.flush.interval.messages The number of messages accumulated on a log partition before messages are flushed to disk long 9223372036854775807 And the log.flush.interval.ms is by default null ... ? It means that until there are so many messages (9223372036854775807) in my topics, they will not be flushed to disk ? It can be very long for a default value Best regards, Adrien Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
broker properties explanations
Hello all, after reading several properties in Kafka documentations, I asked mysleft some questions ... these 2 following options are available: log.dir The directory in which the log data is kept (supplemental for log.dirs property)string /tmp/kafka-logs high log.dirsThe directories in which the log data is kept. If not set, the value in log.dir is used But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's difference in usage ? Also I noticed that the "data" of logs partition and also the "application" logs of Kafka are written in the same directory. Is a mistake of my part ? Because it's very strange for me to log error, debug, warn application message in the same location of my data ... After that I don't very understand why the log.flush.interval.messages have a so big default value ??? log.flush.interval.messages The number of messages accumulated on a log partition before messages are flushed to disk long9223372036854775807 And the log.flush.interval.ms is by default null ... ? It means that until there are so many messages (9223372036854775807) in my topics, they will not be flushed to disk ? It can be very long for a default value Best regards, Adrien
