horkyada opened a new pull request, #22661: URL: https://github.com/apache/kafka/pull/22661
JIRA: https://issues.apache.org/jira/browse/KAFKA-20732 **Problem** On a freshly elected leader of a tiered partition, `RemoteLogManager` size-retention can run while `highestOffsetInRemoteStorage` is still `-1` — it is seeded only by the copy and follower tasks, which race the expiration task (both are scheduled with `initialDelay=0` on separate thread pools). `UnifiedLog.onlyLocalLogSegmentsSize()` then filters `baseOffset >= -1` and counts the entire local log, which is also counted in `remoteLogSizeBytes` for the same offsets. The apparent size roughly doubles, a false `retention.bytes` breach fires, and in-retention data is deleted from the remote tier and — via the resulting `logStartOffset` advance, replicated to followers — the local tier on all replicas. The follower path already guards this: `RLMFollowerTask.execute()` seeds the offset with the comment *"so that the local log segments are not deleted before they are copied to remote storage."* The leader/expiration path, which runs size-retention, had no equivalent step. **Fix** In `cleanupExpiredRemoteLogSegments`, when `highestOffsetInRemoteStorage() == -1`, seed it from remote metadata (`findHighestRemoteOffset`) before computing size-retention. It is a no-op once the offset is already seeded. **Testing** Added `RemoteLogManagerTest.testSizeRetentionDoesNotOverDeleteOnFreshLeaderUntilHighestRemoteOffsetSeeded` — fails without the fix (deletes both copied segments), passes with it. Also reproduced and fix-validated on a live 3.9.0 cluster (details in the JIRA). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
