[
https://issues.apache.org/jira/browse/KAFKA-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778807#comment-17778807
]
Jun Rao commented on KAFKA-15401:
---------------------------------
[~nickstery] : Thanks for reporting this issue. I am wondering how a disk full
event will lead to corrupted indexes. When a disk is full, the flushing of the
index will fail with an IOException. This will prevent the recovery point from
advancing. When the broker is restarted, it seems that we should rebuild the
index by recovering from the recovery point.
> Segment with corrupted index should not be uploaded to remote storage
> ---------------------------------------------------------------------
>
> Key: KAFKA-15401
> URL: https://issues.apache.org/jira/browse/KAFKA-15401
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.6.0
> Reporter: Viktor Nikitash
> Assignee: Viktor Nikitash
> Priority: Minor
> Labels: KIP-405
> Fix For: 3.7.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> During data disk full event, there could be a situation with index
> corruption. There are existing functions which perform sanity checks to
> TimeIndex [1], TxnIndex [2], OffsetIndex [2]. The idea is performing same
> checks before in RemoteLogManager before we upload segment to remote storage
> [4].
> Resources:
> [1][TimeIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L73]
> [2][TransationIndex::sanityCheck()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/storage/src/main/java/org/apache/kafka/storage/internals/log/TransactionIndex.java#L187]
> [3][OffsetIndex::sanityCheck()|#L78]]
> [4][RemoteLogManager::copyLogSegmentsToRemote()|https://github.com/apache/kafka/blob/88d2c4460a1c8c8cf5dbcc9edb43f42fe898ca00/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L649]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
