deepakpanda93 opened a new pull request, #19552:
URL: https://github.com/apache/hudi/pull/19552
### Describe the issue this Pull Request addresses
Closes #17274. (JIRA: HUDI-8258, a subtask of HUDI-8894.)
Hudi 1.x stamps every action with both a requested instant time and a
completion time, and orders the timeline by
completion time. Nothing in the docs said which of the two the **cleaner**
records, so someone reading clean metadata to
debug a retention problem has no way to tell whether
`earliestCommitToRetain` is a start time or a completion time — and
picking the wrong interpretation sends them looking at the wrong commit.
It is a start time, on every field. The issue asks for exactly this to be
stated on the Cleaning and Tech Spec pages.
### Summary and Changelog
Added an `### Instant Times in Clean Metadata` subsection to the Cleaning
page tabulating the four timestamp fields, and
a matching field table to the `Cleaning` section of the 1.0 tech spec:
| Field | Written to | Value |
|---|---|---|
| `earliestInstantToRetain.timestamp` | `HoodieCleanerPlan` (the
`clean.requested` instant) | Instant time of the oldest commit this clean run
retains |
| `earliestCommitToRetain` | `HoodieCleanMetadata` (the completed `clean`
instant) | Copied from the plan, so also an instant time |
| `lastCompletedCommitTimestamp` | both | Instant time of the last completed
write before the clean was planned — despite the name, a start time |
| `startCleanTime` | `HoodieCleanMetadata` | Instant time of the clean
action itself |
Plus a note that incremental clean planning follows the same convention,
ranging over the **requested** instant times of
completed commits.
Files: `website/docs/cleaning.md` (next) and
`website/versioned_docs/version-1.2.0/cleaning.md` (current released docs),
per the next-plus-current convention used in #19473 and #19551.
`website/learn/tech-specs.md` has no versioned copies —
the `learn` plugin is configured without versioning — so it has a single
edit.
### Where this comes from in the code
Checked at both `master` and the `release-1.2.0` tag; the relevant lines are
identical in each.
- `CleanPlanActionExecutor.java:111` and `:177` build the plan's
`earliestInstantToRetain` from `hoodieInstant.requestedTime()`.
- `CleanPlanner.java:655-657` — `getLastCompletedCommitTimestamp()` returns
`getCommitTimeline().lastInstant().requestedTime()`. The source carries the
same observation inline at `CleanPlanActionExecutor.java:178`: *"Note: This is
the start time of the last completed ingestion before this clean."*
- `CleanActionExecutor.java:172-179` and `:256` copy the plan values into
`HoodieCleanMetadata`.
- `CleanPlanner.java:241-245` filters completed commits by
`instant.requestedTime()` against `earliestCommitToRetain`.
### Reproduction
Source reading alone felt insufficient here, because the page promises what
a user will *see* — and the read path runs
`CleanMetadataMigrator.upgradeToLatest`, which I had not verified leaves
these timestamps alone. So the values below were
read back through `CleanerUtils.getCleanerMetadata`, the same call a
debugging user would make.
Setup: Spark 3.5.7 with `hudi-spark3.5-bundle_2.12:1.2.0`, a
local-filesystem COW table, `hoodie.clean.commits.retained=3`,
`hoodie.clean.incremental.enabled=true`, 8 write batches. Requested and
completion times differ on every instant, so no
value can match by coincidence.
Timeline (tail):
| Action | requested | completion |
|---|---|---|
| commit | `20260807112136674` | `20260807112136958` |
| commit | `20260807112137041` | `20260807112137362` |
| commit | `20260807112137452` | `20260807112137746` |
| clean | `20260807112137760` | `20260807112137796` |
Metadata read back from that last clean:
| Field | Value | Is the... |
|---|---|---|
| `startCleanTime` | `20260807112137760` | clean's **requested** time (not
`…137796`) |
| `earliestCommitToRetain` | `20260807112136674` | that commit's
**requested** time (not `…136958`) |
| `lastCompletedCommitTimestamp` | `20260807112137452` | last commit's
**requested** time (not `…137746`) |
| `HoodieCleanerPlan.earliestInstantToRetain` | `20260807112136674`
(action=commit, state=COMPLETED) | identical to the metadata, confirming it is
copied from the plan |
| `HoodieCleanerPlan.lastCompletedCommitTimestamp` | `20260807112137452` |
same |
**No recorded value appears anywhere in the completion-time column.** Every
completion time on that timeline
(`134434, 135308, 135737, 136087, 136511, 136630, 136958, 137008, 137362,
137415, 137746, 137796`) is absent from the
metadata.
Two notes on method. I first ran with `retained=1`, where
`earliestCommitToRetain` and `lastCompletedCommitTimestamp`
collapse to the same value and only one field would have been proven; the
`retained=3` run above separates them into two
distinct values, both still requested times. And for the
incremental-planning sentence, which is not observable in
metadata, I enabled `CleanPlanner` INFO logging:
```
Incremental Cleaning mode is enabled. Looking up partition-paths that have
changed since last clean at 20260807112242500.
New Instant to retain
Option{val=[20260807112242946__20260807112243250__commit__COMPLETED]}.
```
`20260807112242500` is the **requested** time of
`20260807112242500_20260807112242806.commit`, not its completion.
### A note on scope
The parent ticket HUDI-8077, which would have moved clean metadata onto
completion time, is resolved *Won't Do* and its
PR #11972 was closed unmerged. This PR therefore documents the behaviour as
it actually stands, which matches the
issue's own wording that "the start/instant time is **still** used in the
clean metadata".
### Impact
Documentation only. No code, config, or behaviour change.
### Risk Level
none
### Documentation Update
This PR is the documentation update — the Cleaning page (`/docs/cleaning`,
`/docs/next/cleaning`) and the technical
specification (`/learn/tech-specs`).
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
