deepakpanda93 commented on issue #17274: URL: https://github.com/apache/hudi/issues/17274#issuecomment-5240013342
Closing — this was documented in #19552 (merge commit e313ed2), now live on the site. **What was added.** The issue asked for instant vs. completion time to be spelled out on the Cleaning and Tech Spec pages, so both were updated: - **`/docs/cleaning`** — a new `### Instant Times in Clean Metadata` subsection, tabulating the four timestamp fields the cleaner writes and what each one holds. - **`/learn/tech-specs`** — a matching field table in the `Cleaning` section, prefaced with the general rule: cleaning is planned and tracked in terms of instant (start) times, even though actions on the timeline are ordered by completion time. **The answer, for anyone arriving from a search:** every timestamp the cleaner records is a start time. | Field | Written to | Value | |---|---|---| | `earliestInstantToRetain.timestamp` | `HoodieCleanerPlan` (the `clean.requested` instant) | Instant time of the oldest commit this clean run retains | | `earliestCommitToRetain` | `HoodieCleanMetadata` (the completed `clean` instant) | Copied from the plan, so also an instant time | | `lastCompletedCommitTimestamp` | both | Instant time of the last write to complete before the clean was planned — despite the name, a start time | | `startCleanTime` | `HoodieCleanMetadata` | Instant time of the clean action itself | The docs also note that `earliestCommitToRetain` is absent under `KEEP_LATEST_FILE_VERSIONS`, and that incremental clean planning follows the same convention, ranging over the requested instant times of completed commits. **Where this comes from in the code** (identical at `master` and the `release-1.2.0` tag): - `CleanPlanActionExecutor.java:111` and `:177` build the plan's `earliestInstantToRetain` from `hoodieInstant.requestedTime()`. - `CleanPlanner.java:655-657` — `getLastCompletedCommitTimestamp()` returns `getCommitTimeline().lastInstant().requestedTime()`. The source already carried the observation inline at `CleanPlanActionExecutor.java:178`: *"Note: This is the start time of the last completed ingestion before this clean."* - `CleanActionExecutor.java:172-179` and `:256` copy the plan values into `HoodieCleanMetadata`. - `CleanPlanner.java:241-245` filters completed commits by `instant.requestedTime()` against `earliestCommitToRetain`. **Verified against a real table**, not source reading alone — the read path runs `CleanMetadataMigrator.upgradeToLatest`, so the values were read back through `CleanerUtils.getCleanerMetadata`, the same call a debugging user makes. On a COW table with `hoodie.clean.commits.retained=3` and requested/completion times distinct on every instant, all four recorded values matched a requested time and **none** appeared anywhere in the completion-time column. **On scope.** The parent ticket HUDI-8077, which would have moved clean metadata onto completion time, is resolved *Won't Do* and its PR #11972 was closed unmerged. So this documents the behaviour as it stands — which matches this issue's own wording that "the start/instant time is **still** used in the clean metadata". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
