[
https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782179#comment-17782179
]
Divij Vaidya commented on KAFKA-15609:
--------------------------------------
[~ijuma] No, my understanding (which might be incorrect, I am a bit naive in
this area) is as follows.
RAM/Physical memory is logically divided into pages by the OS. Each process is
allocated a range of these pages called virtual address space. This VAS logical
abstract away the fragmented nature of physical memory from the process.
mmap() reserves a range of this virtual address space and maps an underlying
file to it. "Maps an underlying file" means that a portion of the file is read
into physical memory by the OS and the pointer is returned by Mmap (this
pointer can be shared across process for IPC using same physical memory). OS
now manages reading the underlying storage and loading portions of file as
needed into the physical memory. Any changes done by Mmap is actually performed
on the physical memory space reserved by Mmap.
Page cache is a range of pages in the physcial memory which can be used by any
application. File I/O is one of the application using page cache such that any
(non-direct) writes/reads to OS go via page cache.
I think that when OS reads part of file into the physical memory reserved by
Mmap, it read it into a different space than occupied by page cache. Hence,
writes to mmap are writes to physical memory but not to page cache.
It can be validated by creating two different mmap for the same file and
verifying whether they access the same entry in physical memory or not. As per
my assumption they should not since they should access their own reserved space
in physical memory.
> Corrupted index uploaded to remote tier
> ---------------------------------------
>
> Key: KAFKA-15609
> URL: https://issues.apache.org/jira/browse/KAFKA-15609
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.6.0
> Reporter: Divij Vaidya
> Priority: Minor
>
> While testing Tiered Storage, we have observed corrupt indexes being present
> in remote tier. One such situation is covered here at
> https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another
> such possible case of corruption.
> Potential cause of index corruption:
> We want to ensure that the file we are passing to RSM plugin contains all the
> data which is present in MemoryByteBuffer i.e. we should have flushed the
> MemoryByteBuffer to the file using force(). In Kafka, when we close a
> segment, indexes are flushed asynchronously [1]. Hence, it might be possible
> that when we are passing the file to RSM, the file doesn't contain flushed
> data. Hence, we may end up uploading indexes which haven't been flushed yet.
> Ideally, the contract should enforce that we force flush the content of
> MemoryByteBuffer before we give the file for RSM. This will ensure that
> indexes are not corrupted/incomplete.
> [1]
> [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
