Rather than forcing writes to disk after each message, the usual method of ensuring durability is to configure the topic with a Replication Factor of 3, min.insync.replicas=2 and have producers configure acks=all.
This ensures that the record has been replicated to at least 2 brokers before an acknowledgement is sent back to the producer. If you really want to force an fsync after reach record, you could configure the brokers with log.flush.interval.messages=1 https://kafka.apache.org/documentation/#brokerconfigs_log.flush.interval.messages If you only want that behavior for certain topics, you could configure the topic with flush.messages=1 https://kafka.apache.org/documentation/#topicconfigs_flush.messages Do note that this is explicitly not recommended in the documentation. On Fri, Aug 20, 2021 at 8:01 AM sunil chaudhari <sunilmchaudhar...@gmail.com> wrote: > Hi Kunal, > This article may help you. > > https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e > > > Cheers, > Sunil. > > On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <kunal.go...@cohesity.com> > wrote: > > > Hello, > > > > We are exploring using Kafka for our application. Our requirement is that > > once we write some messages to Kafka, it should be guaranteed that the > > messages are persisted to disk. > > We found this > > < > > > https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does > > > > > article which says that a Kafka broker acknowledges a record after it has > > written the record to the buffer of the I/O device; it does not issue an > > explicit fsync operation nor does it wait for the OS to confirm that the > > data has been written. Is this statement true for the current > > implementation? If so, is there any way in which we can ensure fsync is > > called before acknowledgement of messages? > > Any help would be appreciated. > > > > -- > > > > Thanks & Regards > > > > Kunal Goyal > > >
