[
https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782222#comment-17782222
]
Haruki Okada commented on KAFKA-15609:
--------------------------------------
I added
[MmapTest3.java|https://gist.github.com/ocadaruma/fc26fc122829c63cb61e14d7fc96896d]
and confirmed the reads by read() and writes via mmap are consistent.
> Would you happen to have some reference that I can read about this?
I failed to find good web reference but the book "The Linux Programming
Interface 2nd edition" mentions about this.
excerpt:
{quote}Like many other modern UNIX implementations, Linux provides a so-called
unified virtual memory system. This means that, where possible, memory mappings
and blocks of the buffer cache share the same pages of physical memory. Thus,
the views of a file obtained via a mapping and via I/O system calls (read(),
write(), and so on) are always consistent, and the only use of msync() is to
force the contents of a mapped region to be flushed to disk.{quote}
> Corrupted index uploaded to remote tier
> ---------------------------------------
>
> Key: KAFKA-15609
> URL: https://issues.apache.org/jira/browse/KAFKA-15609
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.6.0
> Reporter: Divij Vaidya
> Priority: Minor
>
> While testing Tiered Storage, we have observed corrupt indexes being present
> in remote tier. One such situation is covered here at
> https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another
> such possible case of corruption.
> Potential cause of index corruption:
> We want to ensure that the file we are passing to RSM plugin contains all the
> data which is present in MemoryByteBuffer i.e. we should have flushed the
> MemoryByteBuffer to the file using force(). In Kafka, when we close a
> segment, indexes are flushed asynchronously [1]. Hence, it might be possible
> that when we are passing the file to RSM, the file doesn't contain flushed
> data. Hence, we may end up uploading indexes which haven't been flushed yet.
> Ideally, the contract should enforce that we force flush the content of
> MemoryByteBuffer before we give the file for RSM. This will ensure that
> indexes are not corrupted/incomplete.
> [1]
> [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
