Kunal, I recommend looking at the broker and topic parameters that include the term “flush” , such as https://kafka.apache.org/documentation/#topicconfigs_flush.messages <https://kafka.apache.org/documentation/#topicconfigs_flush.messages>
Kafka lets you configure how often log messages are flushed to disk, either per topic or globally. The default settings leave the flushing completely to the OS. Kafka was designed to take full advantage of the OS page cache because it significantly improves performance for both producers and consumers, allowing them to write to and read from memory. If your application requires absolute disk persistence and you are willing to take a significant performance hit, you can set the topic property flush.messages to 1 for any topic that requires this guarantee. — Peter > On Aug 24, 2021, at 10:31 PM, Kunal Goyal <kunal.go...@cohesity.com> wrote: > > Hi Sunil > > The article that you shared talks about acks. But even if the message is > received by all in-sync replicas and kafka sends response back to the > producer, it is possible that none of the replicas did not flush the > messages to disk. So, if all the replicas crash for some reason, the > messages would be lost. For our application, we require some way to > guarantee that the messages are persisted to disk. > > Regards, > Kunal > > On Tue, Aug 24, 2021 at 8:40 PM Vairavanathan Emalayan < > vairavanathan.emala...@cohesity.com> wrote: > >> >> >> ---------- Forwarded message --------- >> From: sunil chaudhari <sunilmchaudhar...@gmail.com> >> Date: Fri, Aug 20, 2021 at 8:00 AM >> Subject: Re: Ensuring that the message is persisted after acknowledgement >> To: <users@kafka.apache.org> >> Cc: Vairavanathan Emalayan <vairavanathan.emala...@cohesity.com> >> >> >> Hi Kunal, >> This article may help you. >> >> https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e >> >> >> Cheers, >> Sunil. >> >> On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal <kunal.go...@cohesity.com> >> wrote: >> >>> Hello, >>> >>> We are exploring using Kafka for our application. Our requirement is that >>> once we write some messages to Kafka, it should be guaranteed that the >>> messages are persisted to disk. >>> We found this >>> < >>> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does >>>> >>> article which says that a Kafka broker acknowledges a record after it has >>> written the record to the buffer of the I/O device; it does not issue an >>> explicit fsync operation nor does it wait for the OS to confirm that the >>> data has been written. Is this statement true for the current >>> implementation? If so, is there any way in which we can ensure fsync is >>> called before acknowledgement of messages? >>> Any help would be appreciated. >>> >>> -- >>> >>> Thanks & Regards >>> >>> Kunal Goyal >>> >>
