[
https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775737#comment-17775737
]
Divij Vaidya commented on KAFKA-15609:
--------------------------------------
> So you mean, when a segment rolled, before the data flushed, the RSM is
> copying the incomplete segment/index into remote tier, right?
Yes.
For a log segment, this might not be a problem because the RLM plugin will read
the log segment from the page cache even if it hasn't been fsync()'ed. I am
assuming OS is intelligent enough to ensure that multiple threads read the same
dirty copy of data from page cache.
But for index files which are stored in MappedByteBuffer (backed by Mmapped
files), we are reading from two different threads. One thread (request handler
thread) accesses the file via Mmap interface which will provide consistent
view. But the other thread, RLM plugin thread, will read the file via OS I/O
without going through Mmap interfaces. In such cases, is the OS intelligent
enough to understand that it should provide a "dirty" / non-flushed view of the
file to the second thread as well?
I don't have an answer and we may need some research to establish if this could
be a problem or not. I will wait for [~ocadaruma] to comment here since they
seem like an expert on impact of fsync, mmaps and page cache optimizations.
> Corrupted index uploaded to remote tier
> ---------------------------------------
>
> Key: KAFKA-15609
> URL: https://issues.apache.org/jira/browse/KAFKA-15609
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.6.0
> Reporter: Divij Vaidya
> Priority: Minor
>
> While testing Tiered Storage, we have observed corrupt indexes being present
> in remote tier. One such situation is covered here at
> https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another
> such possible case of corruption.
> Potential cause of index corruption:
> We want to ensure that the file we are passing to RSM plugin contains all the
> data which is present in MemoryByteBuffer i.e. we should have flushed the
> MemoryByteBuffer to the file using force(). In Kafka, when we close a
> segment, indexes are flushed asynchronously [1]. Hence, it might be possible
> that when we are passing the file to RSM, the file doesn't contain flushed
> data. Hence, we may end up uploading indexes which haven't been flushed yet.
> Ideally, the contract should enforce that we force flush the content of
> MemoryByteBuffer before we give the file for RSM. This will ensure that
> indexes are not corrupted/incomplete.
> [1]
> [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
