andygrove opened a new issue, #7493: URL: https://github.com/apache/datafusion-comet/issues/7493
### What is the problem the feature request solves? Iceberg 1.12.0 (released 2026-09-30) is the first Iceberg release with a Spark 4.2 module. Comet does not use it yet: - The `spark-4.2` profile in `spark/pom.xml` reuses `iceberg-spark-runtime-4.0_2.13:1.10.0`, because no 4.2 runtime existed. That jar is binary-incompatible with Spark 4.2 (`connector.catalog.View` became a class), so `icebergAvailable` is false on 4.2 and `CometIcebergTestBase` and `CometFuzzIcebergBase` skip every Iceberg suite there. The native Iceberg REST catalog scan test is also skipped (#4969). - No CI profile covers Iceberg 1.12. There is no `dev/diffs/iceberg/1.12.0.diff`, and Iceberg's own suites do not run on Spark 4.2 at all. The existing jobs are `iceberg_1_8` through `iceberg_1_11`, and the newest pairs with Spark 4.1.3. - Iceberg 1.12 removed the public 2-arg `ManifestFiles.read(ManifestFile, FileIO)`, so native Iceberg writes silently fall back (#7240). A profile on 1.12 would have caught this. Related: #4871, #4969, #7240, #5649. ### Describe the potential solution One PR, one commit per part: 1. Point the `spark-4.2` profile at `iceberg-spark-runtime-4.2_2.13:1.12.0`. Drop the `!isSpark42Plus` guards in the Iceberg test bases and re-enable the REST catalog scan test (#4969) if it passes. 2. Fix #7240 by resolving the 3-arg `ManifestFiles.read(manifest, io, specsById)`, which is public on every Iceberg version Comet supports (1.5.2 through 1.12). 3. Add `dev/diffs/iceberg/1.12.0.diff`, ported from the 1.11.0 diff and regenerated against the `apache-iceberg-1.12.0` tag, plus an `iceberg_1_12` CI job (Iceberg 1.12.0 on Spark 4.2.0, nightly tier and the `run-iceberg-tests` label). That touches `ci.yml`, `dev/ci/compute-changes.py`, `dev/ci/check-ci-config.py` and the workflows README. 4. Update `iceberg-spark-tests.md`, `iceberg-writes.md` (which says the Iceberg suites skip on 4.2) and the Iceberg compatibility notes. Open question for #4871: whether 1.12 should join the queue tier or stay nightly. Spark 4.2 is still experimental, so nightly plus the label seems right for now. ### Additional context I checked the release tags in `apache/iceberg`: `knownSparkVersions` is `3.4,3.5,4.0` on 1.10.2, `3.4,3.5,4.0,4.1` on 1.11.0, and `3.5,4.0,4.1,4.2` on 1.12.0 (Spark 4.2.0). I could not confirm the `iceberg-spark-runtime-4.2_2.13:1.12.0` artifact on Maven Central, so the first step is to check that it resolves. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
