fhan688 commented on PR #19041:
URL: https://github.com/apache/hudi/pull/19041#issuecomment-4965210418
Based on Teacher Danny’s feedback, this PR refines the clean/archive
interaction for KEEP_LATEST_BY_HOURS by separating cleaner-side retention
semantics from archiver-side safety checks.
The cleaner now computes earliestCommitToRetain (ECTR) for
KEEP_LATEST_BY_HOURS from completed instants using completion time, with
requested time as a fallback for older timeline entries. It no longer clips
this cleaner-side ECTR by pending instants, because archival is already blocked
by the existing pending-instant protection in the archiver.
On the archival side, the safety boundary is derived from the ECTR
recorded in the latest completed clean metadata. Both V1 and V2 archivers use
that clean-recorded ECTR as the archive boundary, so commit metadata is
archived only after cleaner has completed and recorded that it is safe to move
past that boundary. ECTR and later instants are retained.
This aligns the implementation with two points from Danny’s review:
1. Pending instant protection belongs to archiver pending-block logic, not
KEEP_LATEST_BY_HOURS ECTR calculation.
2. KEEP_LATEST_BY_HOURS ECTR should be based on completion time, not only
requested time.
The PR also adds tests for requestedTime/completionTime mismatch, pending
instants before ECTR, empty ECTR, V1/V2 archive boundary behavior, empty clean
metadata updates, and delete-partition replace commit completion times.
please help review again, thanks! @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
