ravjot28 opened a new issue, #20582:
URL: https://github.com/apache/hudi/issues/20582
### Bug Description
**What happened:**
With `hoodie.clean.policy=KEEP_LATEST_BY_HOURS`, no clean is planned when
every completed commit of the table is older
than `hoodie.clean.hours.retained`. Every older file slice stays until a
commit lands inside the window again: a table
written less often than its window never cleans, and a window of 0 hours
never cleans at all.
**What you expected:**
The file slices older than the window are removed and the latest file slice
of each file group is kept, as
`CleanPlanner#getFilesToCleanKeepingLatestHours` documents ("all the files
with commit time earlier than N hours will
be removed. Also the latest file for any file group is retained."). Every
commit being outside the window is the case
in which the documented behaviour removes the most. A file group written by
the latest commit may keep the slice before it, as at the
edge of the window.
**Steps to reproduce:**
1. Create a copy-on-write table with
`hoodie.clean.policy=KEEP_LATEST_BY_HOURS` and `hoodie.clean.hours.retained=2`.
2. Write three commits that update the same file group, all more than 2
hours ago (or set the window to 0 hours, so
every commit is older than the window).
3. Run a clean: nothing is planned and all three file slices remain.
**Root cause (master d5230b69e22):**
-
`hudi-common/src/main/java/org/apache/hudi/common/util/CleanerUtils.java:151-155`:
for `KEEP_LATEST_BY_HOURS`,
`getEarliestCommitToRetain` returns the first completed commit at or after
`now - hoursRetained`; when every
completed commit is older there is none and it returns empty.
-
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:157-160`
sends
`KEEP_LATEST_BY_HOURS` to `getPartitionPathsForCleanByCommits`, which
returns no partition when there is no commit to
retain (`CleanPlanner.java:200-204`, "No earliest commit to retain. No
need to scan partitions !!").
- For `KEEP_LATEST_COMMITS` an empty result correctly means "fewer commits
than retained, nothing to clean". For the
hours policy it means "every commit is outside the window", the case in
which the documented behaviour cleans the
most.
Related, not the same defect: #16014 (which time the hours window compares),
#8209, #7600.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
**Query engine:** any (the clean planner is engine independent); seen with
Spark 3.5. The metadata table derives
the hours policy from the data table and is affected the same way.
**Relevant configs:** `hoodie.clean.policy=KEEP_LATEST_BY_HOURS`,
`hoodie.clean.hours.retained`
### Logs and Stack Trace
```
INFO CleanPlanner: No earliest commit to retain. No need to scan partitions
!!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]