kamalcph commented on code in PR #15634: URL: https://github.com/apache/kafka/pull/15634#discussion_r1569328393
########## core/src/main/scala/kafka/log/UnifiedLog.scala: ########## @@ -282,15 +282,15 @@ class UnifiedLog(@volatile var logStartOffset: Long, /** * Update high watermark with offset metadata. The new high watermark will be lower - * bounded by the log start offset and upper bounded by the log end offset. + * bounded by the local-log-start-offset and upper bounded by the log-end-offset. * * @param highWatermarkMetadata the suggested high watermark with offset metadata * @return the updated high watermark offset */ def updateHighWatermark(highWatermarkMetadata: LogOffsetMetadata): Long = { val endOffsetMetadata = localLog.logEndOffsetMetadata - val newHighWatermarkMetadata = if (highWatermarkMetadata.messageOffset < logStartOffset) { - new LogOffsetMetadata(logStartOffset) + val newHighWatermarkMetadata = if (highWatermarkMetadata.messageOffset < _localLogStartOffset) { Review Comment: > For the makeLeaders path, it will call UnifiedLog.convertToOffsetMetadataOrThrow. Within it, checkLogStartOffset(offset) shouldn't throw OFFSET_OUT_OF_RANGE since we are comparing the offset with logStartOffset. Do you know which part throws OFFSET_OUT_OF_RANGE error? The next line `localLog.convertToOffsetMetadataOrThrow` in [convertToOffsetMetadataOrThrow](https://sourcegraph.com/github.com/apache/kafka@a3dcbd4e28a35f79f75ec1bf316ef0b39c0df164/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L1429) method reads the segment from disk, there it throws the error. The call [segments.floorSegment(startOffset)](https://sourcegraph.com/github.com/apache/kafka@a3dcbd4e28a35f79f75ec1bf316ef0b39c0df164/-/blob/core/src/main/scala/kafka/log/LocalLog.scala?L366) in LocalLog fails to find the segment with log-start-offset, then OffsetOutOfRangeException is thrown. > For the follower fetch path, it's bounded by LogEndOffset. So it shouldn't need to call UnifiedLog.fetchHighWatermarkMetadata, right? The regular consumer will call UnifiedLog.fetchHighWatermarkMetadata. yes, you're right. I attached the wrong call stack for handling the follower request. Please find the updated call stack below: Leader with invalid high-watermark handles the FETCH requests from follower and throws OFFSET_OUT_OF_RANGE error: ``` KafkaApis.handleFetchRequest ReplicaManager.fetchMessages ReplicaManager.readFromLog Partition.fetchRecords # readFromLocalLog Partition.updateFollowerFetchState Partition.maybeIncrementLeaderHW LeaderLog.maybeIncrementHighWatermark UnifiedLog.fetchHighWatermarkMetadata UnifiedLog.convertToOffsetMetadataOrThrow LocalLog.convertToOffsetMetadataOrThrow LocalLog.read # OffsetOutOfRangeException exception ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org