laserninja opened a new pull request, #13329: URL: https://github.com/apache/gravitino/pull/13329
### What changes were proposed in this pull request? Implements the first phase of the merged design in #12942: the `builtin-iceberg-rewrite-manifests` Spark job, registration, validated arguments, spec selection, and operator documentation. Manifest statistics and automatic policy triggering remain separate follow-ups in the design. Continues #12937 on current main and preserves the original commits by @ibrahimErbilen from #11216, with co-author attribution on follow-up work. Adds Spark-backed tests invoking the job entry point through resolved template arguments. They verify default and explicit partition-spec selection, unchanged table records and other-spec manifests, repeat-run no-op behavior, and cleanup without metadata changes after an unknown-spec failure. The entry point checks the Iceberg runtime, closes Spark on success or failure, and uses structured logging without dumping Spark configuration values. ### Why are the changes needed? Manifest rewriting maintains manifest layout independently of data-file compaction. The previous implementation lacked Spark-backed verification of partition-spec selection and failure cleanup. Fix: #11196 ### Does this PR introduce _any_ user-facing change? Adds `builtin-iceberg-rewrite-manifests` with required `catalog_name` and `table_identifier`, plus optional `use_caching`, `spec_id`, and `spark_conf`. Existing jobConf/CLI naming conventions are retained. Invalid, duplicate, or missing argument values fail before session creation; omitted optional values retain Iceberg defaults. `spec_id` selects an existing spec, without migrating files between specs. ### How was this patch tested? - 179 jobs-module tests passed, including 32 manifest unit tests and 3 Spark-backed tests on Spark 3.5 / Iceberg 1.11.0. - `./gradlew :maintenance:jobs:test :maintenance:jobs:spotlessCheck :maintenance:jobs:javadoc :docs:build rat -PskipITs` passed with JDK 17. - `git diff --check` passed. Spark tests use a local Hadoop catalog; remote catalog deployment and the server REST submission path were not exercised end to end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
