laserninja opened a new pull request, #11700: URL: https://github.com/apache/gravitino/pull/11700
### What changes were proposed in this pull request? Add a design document for the built-in Iceberg maintenance job `builtin-iceberg-remove-orphan-files` that identifies and removes orphaned data and metadata files from Iceberg table storage locations via Spark's `remove_orphan_files` procedure. The design doc covers: - Full end-to-end architecture: Policy → Strategy → Adapter → Job - Policy content class (`IcebergOrphanFileRemovalContent`) with configurable parameters: `older_than` (timestamp), `location` (custom path), `dry_run` (preview mode) - Strategy handler for time-based trigger evaluation - Job adapter for context-to-config conversion - Safety considerations (3-day default, dry-run mode, policy-gated execution) - Comparison with existing compaction and snapshot expiration flows - Proposed PR plan (3 incremental PRs) ### Why are the changes needed? Orphan files accumulate from failed writes, incomplete transactions, schema evolution, or concurrent operations. Without periodic cleanup, these files waste significant storage. The existing built-in jobs cover data compaction, statistics, and snapshot expiration but do not address orphan file cleanup. A design doc is needed before implementation to align on the approach, safety mechanisms, and PR structure. Fix: #11195 ### Does this PR introduce any user-facing change? No. This is a design document only. ### How was this patch tested? N/A — design doc only, no code changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
