GitHub user voonhous added a comment to the discussion: Proposal: drop Spark 4.0 support in Hudi 1.3.0
Thanks @wombatu-kun -- all three land, and I've rewritten the top post around them rather than patching it. **Shredding reachability.** Confirmed: `HoodieSparkSchemaConverters` only ever builds `createVariant(...)`, and the only production route to `typed_value` is a schema that already carries it. Removed that argument entirely. **Self-describing.** Confirmed, and the most useful catch. `AvroSchemaConverterWithTimestampNTZ` emits a bare `GroupType`, and it sits in `hudi-hadoop-common`, which must keep compiling under `spark3.5` at parquet 1.13.1 -- so the annotation is out of reach regardless of 4.0. The top post now has an explicit "what this does not fix" section, and the checklist says keep `isVariantPhysicalSchema`. Bucket B item 2 of #18935 is dead; I'll file the avro-path annotation follow-up separately, blocked on 3.5. **CI.** Right, `hudi_trino_e2e.yml` runs `-Dspark4.0` on push/PR. Narrowed to the accurate claim: no *unit* job exercises `hudi-spark4.0.x`. The compose move is mechanical -- we already have higher Spark composes to switch onto, it just needs `trinocoordinator` on the target pair. It's a checklist item, not a blocker. I've also taken your framing as the TL;DR, since it's sharper than what I had: `hudi-spark4-common` compiles once per profile, so while `spark4.0` is in the build the source cannot reference `VariantLogicalTypeAnnotation` at all. Two things added: **Sequencing against #18961.** Auto-inference is what makes shredded files common in the wild for the first time, which flips the 4.0 gap from latent to live. Writes are guarded and the AVRO read path is fine on 4.0; the exposure is a 4.0 *native* read of a 4.1/4.2-written shredded file. Keep 4.0 and that needs a test plus probably a read guard before #18937 flips; drop it and neither is needed. **On DBR 17.3 LTS** -- I chased the vendor picture and I think it argues the other way. Every runtime shipping Spark 4.0 ships a Hudi far behind this change: EMR's Spark 4.0 (`emr-spark-8.0`) bundles Hudi **1.0.2**, EMR 7.10.0 bundles 0.15.0-amzn-7 on Spark 3.5.5, Dataproc 2.3 is Spark 3.5.3 with Hudi 0.15.0. DBR doesn't bundle Hudi at all. So the only population 1.3.0 could strand is self-managed on Spark 4.0, who control both versions. Table's in the top post -- happy to be corrected if I've missed a population. GitHub link: https://github.com/apache/hudi/discussions/19585#discussioncomment-17987441 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
