Hi Guang, Thanks so much for the quick response.
However I have one question, a lot of material on the web says Kafka writes to the pagecache. Then does that mean java.nio.GatheringByteChannel.write(buffer) actually writes to the pagecache ? I only ask because this is not mentioned in the docs: https://docs.oracle.com/javase/8/docs/api/java/nio/channels/GatheringByteChannel.html#write-java.nio.ByteBuffer:A- https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#write-java.nio.ByteBuffer- Hence not guaranteed, am I correct? So if Kafka depends on just writing to the pagecache for performance shouldn't it use some sort of API or mechanism to do just that ? Or are the docs on the web wrong about how Kafka operates ? https://kafka.apache.org/documentation/#design_filesystem Just confused about what is really happening. On Thu, Mar 6, 2025 at 5:31 AM Zhao, Guang <[email protected]> wrote: > (Sorry, missing links) -- the appending operation in FileRecords: > https://github.com/apache/kafka/blob/56fc8e9d2a04172fd014b89428f1c97da93f4894/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java#L184 > > Cheers, > Guang > -- > Guang Zhao, NetApp > [email protected]<mailto:[email protected]> > > > From: Zhao, Guang <[email protected]> > Date: Thursday, 6 March 2025 at 10:57 am > To: [email protected] <[email protected]> > Subject: Re: How does Kafka write to the pagecache/pagetable ? > Hi Sreyan, > > Thanks for the question. The page cache is managed by the OS, so I think > it is transparent to Kafka writes, which are to files. Just at a higher > abstraction level. > > Some pointers in code: > > * LogSegment: > https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java > * FileRecrods: > https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java > > You can see appending records actually write to a file through Java’s > FileChannel. > > Hope this helps! > > Cheers, > Guang > -- > Guang Zhao, NetApp > [email protected]<mailto:[email protected]> > > > From: Sreyan Chakravarty <[email protected]> > Date: Thursday, 6 March 2025 at 2:01 am > To: [email protected] <[email protected]> > Subject: How does Kafka write to the pagecache/pagetable ? > [You don't often get email from [email protected]. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hi devs, > > I am curious in trying to understand the internals of how Kafka actually > writes to disk. I am wondering as to how it writes to the > pagecache/pagetable ? > > From what I understand, Kafka never writes to the files directly instead it > writes to the pagecache and lets the OS do the flush to disk. Please > correct me if I am wrong. > > Where in the source code can I find code that does this ? > > Any help would be greatly appreciated. > > -- > Regards, > Sreyan Chakravarty > -- Regards, Sreyan Chakravarty
