imavishek opened a new pull request, #23471: URL: https://github.com/apache/kafka/pull/23471
### Summary `RemoteDeleteLagSegments` / `RemoteDeleteLagBytes` can get latched at a non-zero value that never drains back to 0 after a partition leader moves to another broker, even though remote deletion is healthy and no segment is orphaned in remote storage. It is purely a metrics artifact, but under frequent leadership movement the broker/cluster aggregate steps up over time as each affected partition contributes a stuck residual, producing false-positive "delete lag stuck" alerts. JIRA: https://issues.apache.org/jira/browse/KAFKA-21098 ### Root cause The copy path was hardened for this exact race in KAFKA-16948 (reset lag metrics on becoming follower + guard `recordLagStats` with `!isCancelled()`), but the delete path never got the equivalent treatment. On a leader→follower transition, `onLeadershipChange` runs `doHandleFollowerPartition` (cancels the `RLMExpirationTask`) and then `removeRemoteTopicPartitionMetrics` (removes the delete-lag gauges). But an expiration-pool thread already mid-`cleanupExpiredRemoteLogSegments` for that partition can call `updateRemoteDeleteLagWith(...)` **after** the gauge was removed, re-registering it at a non-zero value. `updateRemoteDeleteLagWith` emits unconditionally — it has no `!isCancelled()` guard. Because the same pass then reaches `deleteRemoteLogSegment(..., ignored -> !isCancelled())`, which returns `false` for the now-cancelled task, the segment is not deleted by this old leader and the counter is never decremented back down. The gauge stays pinned until the broker re-leads the partition and completes a fresh pass, the partition is stopped, or the broker restarts. The underlying segment is not orphaned — the new leader independently re-evaluates retention and deletes it. ### Fix Guard `updateRemoteDeleteLagWith` with `!isCancelled()`, mirroring the copy path's `recordLagStats`. A cancelled expiration task can no longer re-register the gauge after `removeRemoteTopicPartitionMetrics` has cleared it. Also adds a package-private `rlmExpirationTask` test accessor (like the existing `rlmCopyTask`) so the behavior can be tested directly. ### Testing Added `testRemoteDeleteLagResetsToZeroOnBecomingFollower` (mirrors `testTierLagResetsToZeroOnBecomingFollower` on the copy path). It fails without the guard (the cancelled task re-registers a non-zero lag) and passes with it. Existing tiered-storage delete-lag tests remain green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
