mahsoodebrahim opened a new pull request, #19121:
URL: https://github.com/apache/hudi/pull/19121
### Describe the issue this Pull Request addresses
Hudi can accumulate "orphan" data/log files — files left on the filesystem
that are not referenced by any commit in the active or archived timeline (e.g.
from failed/partial writes or interrupted operations). Today the only way to
find and remove them is the `HoodieRepairTool` spark-submit utility, which
requires cluster-submit access and operational knowledge of the tool.
This PR exposes the same capability as a Spark SQL stored procedure
(`repair_orphan_files`), so orphan-file detection (dry-run/view) and cleanup
become accessible from any Spark SQL session — consistent with the other
`repair_*`/maintenance procedures already registered in `HoodieProcedures`. It
reuses the existing `org.apache.hudi.table.repair.RepairUtils` detection logic
so results match `HoodieRepairTool`, and adds safety guards (MDT cross-check,
`max_orphans` cap, backup-before-remove) suited to interactive use.
### Summary and Changelog
Adds a `repair_orphan_files` Spark SQL stored procedure that finds and
optionally removes orphan data files — files present on the filesystem but not
referenced by any commit (active or archived). It exposes orphan-file
detection/cleanup to any Spark SQL session, reusing the same detection logic
(`org.apache.hudi.table.repair.RepairUtils`) that backs the `HoodieRepairTool`
spark-submit utility.
Parameters: `table`/`path`, `partition` (scope to one partition), `dry_run`
(default`true`), `backup_path` (required when `dry_run=false`),
`archived_start_ts`/`archived_end_ts` (optional instant range), `max_orphans`
(default `100000`, fail-fast safety cap).
Output: `partition, file_name, instant_time, backup_path, status` where
`status ∈ {IDENTIFIED, BACKED_UP, BACKUP_FAILED, SKIPPED_PRESENT_IN_MDT}`.
- Handles COW (base files) and MOR (base + log files) and all commit action
types (COMMIT, DELTA_COMMIT, REPLACE_COMMIT).
- One Spark task per partition (listing + classification) so only orphan
candidates reach the driver; `max_orphans` guards against driver OOM.
- MDT safety cross-check: candidates still visible in the metadata table are
surfaced as `SKIPPED_PRESENT_IN_MDT` rather than removed.
- Cleanup mode moves orphans to `backup_path` (no destructive delete) and
logs `BACKUP_FAILED` root causes.
- Registered in `HoodieProcedures`; adds `TestRepairOrphanFilesProcedure` (7
cases).
### Impact
New, additive Spark SQL stored procedure. No changes to existing procedures
or behavior.
### Risk Level: low
New procedure only, covered by unit tests. Cleanup is opt-in
(`dry_run=false`), moves files to a backup location, has an MDT cross-check,
and a `max_orphans` cap.
### Documentation Update
Usage documented in the class Scaladoc; the SQL procedures reference page
can list `repair_orphan_files` in a follow-up.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
