andygrove opened a new issue, #5882:
URL: https://github.com/apache/datafusion-comet/issues/5882
## Context
#5365 adds the Delta Lake scan contrib as a separate Maven module
(`contrib/delta-spark`) that only enters the reactor under the `delta` profile
and produces its own jar, `comet-contrib-delta-spark<version>_<scala>`. The
user guide and the module README tell users to put that jar on the classpath
alongside `delta-spark`. This issue is for deciding how that jar reaches users,
and more generally how table-format integrations are packaged and released,
since Delta and Iceberg currently take different shapes.
## How the release tooling behaves today
- `dev/release/build-release-comet.sh` runs `./mvnw install` into a
temporary local repository for six profile combinations (Spark 3.4 and 3.5 with
Scala 2.12 and 2.13, then Spark 4.0 and 4.1). None of those invocations pass
`-Pdelta`, so the contrib module is never built during a release.
- `dev/release/publish-to-maven.sh` does not run `mvn deploy`. It signs and
uploads every `.jar` and `.pom` found under `org/apache/datafusion` in that
local repository. Two things follow:
- `maven.deploy.skip` has no effect on what gets published. Whether the
contrib jar ships depends only on whether `-Pdelta` is on the `install` lines.
- Anything `install` produces is published. #5365 adds an unconditional
`test-jar` execution to `spark/pom.xml` (about 6.7 MB per profile) so that the
contrib suites can reuse `CometTestBase`. As written it would ship six
`-tests.jar` artifacts per release.
- The native library needs no release change. The release containers build
with default Cargo features, and the `delta` feature (deletion-vector decoding
plus the planner arm) is in the default set, so the released `libcomet` already
carries everything the contrib jar needs.
## Questions to settle
1. **Should the contrib jar be published at all?** If yes, `-Pdelta` goes on
the Spark 3.5, 4.0 and 4.1 `install` lines (the 3.4 pairing is `delta-core` 2.4
and does not compile). If no, the docs should say build-from-source is the only
route and the `maven.deploy.skip=false` override in the contrib pom should go.
An experimental feature that only exists if users build from source is unlikely
to get real-workload testing.
2. **What Delta versions does each published jar support?** Each jar
compiles against one Delta release per Spark line (3.3.2, 4.0.1, 4.3.1).
Nothing verifies binary compatibility across Delta minors, and `pom.xml`
already records that Delta 4.1.0 and 4.2.0 are incompatible with Spark 4.1.3.
Publishing implies a support statement in the user guide.
3. **Should the spark test-jar be published?** Options: bind the `test-jar`
execution inside the `delta` profile so it only exists when the contrib is
built, or accept publishing it (there is partial precedent in the
`-test-sources.jar`).
4. **Why is Delta packaged differently from Iceberg, and should it stay that
way?** Iceberg support lives inside `comet-spark`: `iceberg-spark-runtime` is a
test-scope dependency only and `IcebergReflection` addresses every Iceberg
class by name, so there is no profile, no separate jar, and no dependency on a
specific Iceberg release. The Delta contrib compiles against `delta-spark`
internals (`DeltaParquetFileFormat`, `DeletionVectorDescriptor`,
`DeltaColumnMapping`), which forces a `provided` dependency, one artifact per
Spark-and-Delta pairing, and rules out Spark 3.4. That structural difference
explains the module, the profile, the separate CI workflow and the opt-in
config. The two consistent end states are (a) fold Delta into `comet-spark`
behind reflection like Iceberg, or (b) keep the contrib shape and finish it
(publish the jar, document the Delta pairing, gate the test-jar). Deciding this
on purpose also settles part of the core-versus-contrib boundary question
raised in
the #5365 review and in #5411.
## Related
- #5365 (the contrib PR)
- #174 (Delta Lake support)
- #5411 (convergence of the two Delta read paths)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
