Divij Vaidya created KAFKA-15609:
------------------------------------
Summary: Corrupted index uploaded to remote tier
Key: KAFKA-15609
URL: https://issues.apache.org/jira/browse/KAFKA-15609
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Affects Versions: 3.6.0
Reporter: Divij Vaidya
While testing Tiered Storage, we have observed corrupt indexes being present in
remote tier. One such situation is covered here at
https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another
such possible case of corruption.
Potential cause of index corruption:
We want to ensure that the file we are passing to RSM plugin contains all the
data which is present in MemoryByteBuffer i.e. we should have flushed the
MemoryByteBuffer to the file using force(). In Kafka, when we close a segment,
indexes are flushed asynchronously. Hence, it might be possible that when we
are passing the file to RSM, the file doesn't contain flushed data. Hence, we
may end up uploading indexes which haven't been flushed yet. Ideally, the
contract should enforce that we force flush the content of MemoryByteBuffer
before we give the file for RSM. This will ensure that indexes are not
corrupted/incomplete.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
