schenksj opened a new issue, #5390: URL: https://github.com/apache/datafusion-comet/issues/5390
Follow-up from review of #4952 ([tracking comment](https://github.com/apache/datafusion-comet/pull/4366#issuecomment-5320446796), item 3). ### What is pinned today The `contrib-delta` Maven profile pins a `delta-spark` version per Spark profile, in `spark/pom.xml`: | Spark profile | `delta.version` | |---|---| | spark-3.5 | 3.3.2 | | spark-4.0 | 4.0.0 | | spark-4.1 | 4.1.0 | with a top-level default of `4.1.0` in the root `pom.xml`. `dev/verify-contrib-delta-gate.sh` asserts the mapping holds (spark-4.1 -> delta-spark 4.1.x, spark-3.5 -> 3.x, spark-4.0 -> 4.0.x) and fails the build gate otherwise. ### The gap The pin is a real compatibility decision — the contrib reflects into delta-spark internals (`TahoeLogFileIndex`, `PreparedDeltaFileIndex`, `TahoeBatchFileIndex`, DV plumbing), so the supported range is narrower than "any delta-spark". But that decision is currently only implicit in the poms plus an assertion in the gate script. There is no issue recording: - **which** delta-spark versions the contrib claims to support, and why those; - **what a user should expect** if they run against a different delta-spark than the pinned one; - **who updates the pin** when a new delta-spark ships, and what has to be re-verified (the reflection surface is the risky part, not the wire format); - whether the per-Spark mapping should ever become a **range** rather than an exact pin. ### Ask Settle the policy and write it down — in the contrib user guide and/or the profile comments — and keep the gate script assertion as the enforcement. This is a documentation/decision issue rather than a code change; the current pins work. --- 🤖 Filed with [Claude Code](https://claude.com/claude-code). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
