ibrahimErbilen opened a new pull request, #11216: URL: https://github.com/apache/gravitino/pull/11216
### What changes were proposed in this pull request? - Added a new built-in Iceberg maintenance job named `builtin-iceberg-rewrite-manifests`. - Implemented `IcebergRewriteManifestsJob` to execute Iceberg's `rewrite_manifests` action via Spark's SQL procedure (`CALL catalog.system.rewrite_manifests`). - Supported configurable parameters: `--catalog`, `--table`, and an optional `--use-caching` argument. ### Why are the changes needed? Over time, Iceberg tables accumulate many small manifest files from frequent commits. This slows down scan planning because the engine must open and read each manifest to determine which data files match a query filter. Rewriting manifests into fewer, larger files with better-aligned partition specs significantly improves query planning performance. While `IcebergRewriteDataFilesJob` addresses data file layout, this new job specifically addresses manifest file bloat, enabling automatic policy-driven manifest optimization on the server side. Fix: #11196 ### Does this PR introduce _any_ user-facing change? Yes. Users can now run the `builtin-iceberg-rewrite-manifests` job via the Gravitino API to optimize Iceberg table manifest files. ### How was this patch tested? - Added unit tests for `IcebergRewriteManifestsJob` to verify argument parsing and Spark SQL procedure string generation. - Manually tested the job execution and verified that manifest files are successfully rewritten and consolidated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
