GitHub user voonhous edited a discussion: Proposal: drop Spark 4.0 in Hudi 1.3.0 -- the only Spark 4 profile below the parquet 1.16 VARIANT floor
**Proposal: drop the `spark4.0` profile and `hudi-spark-datasource/hudi-spark4.0.x` in 1.3.0.** Supported Spark matrix becomes 3.5, 4.1, 4.2. We want to push hard on VARIANT. Spark 4.0 is not "one minor behind" on variant -- its native variant support sits on the other side of a format boundary. It also reaches [EOL on 2026-11-23](https://lists.apache.org/thread/cxyfz1zlqhltzd4dsk7wn9v4jdmljcd5). Spark 4.0 ships parquet-java 1.15.2; 4.1 ships 1.16.0. Concretely, on 4.0: - **No `VariantLogicalTypeAnnotation`** -- added in parquet-java 1.16.0 ([apache/parquet-java#3070](https://github.com/apache/parquet-java/issues/3070)). Variant groups are written as bare, unannotated `metadata`/`value` structs, so nothing in the file metadata says "this is a VARIANT". Hudi falls back to a name/arity shape heuristic to recognise them. - **Variant fields read by position, not by name** -- 4.0's `ParquetUnshreddedVariantConverter` hardcodes its converter array as `[value, metadata]` and then indexes by schema position, so spec-ordered `[metadata, value]` files fail with `MALFORMED_VARIANT` (#18334). Hudi carries a forked `ReadSupport` purely to reorder around it. SPARK-54410 fixed this in 4.1 by reading by name. - **No shredded-variant reconstruction** -- `buildFullVariantReadSchema` is implemented for 4.1/4.2 only; 4.0 falls through to the base `None`. 4.1 and 4.2 have none of these. This is a capability boundary, not a version gap. ## The core problem: a compile floor, not a runtime gate The annotation gap is not something a runtime check can paper over. `hudi-spark4-common` is compiled once per Spark profile, and under `-Dspark4.0` `parquet.version` resolves to 1.15.2 -- so while `spark4.0` is in the build, **shared Spark 4 code cannot reference `VariantLogicalTypeAnnotation` at all.** No config flag, version check, or reflection shim buys that back. | Profile | Spark | parquet-java | `VariantLogicalTypeAnnotation` | |---|---|---|---| | `spark3.5` | 3.5.3 | 1.13.1 | n/a (no VariantType) | | `spark4.0` | 4.0.2 | **1.15.2** | **no** | | `spark4.1` | 4.1.1 | 1.16.0 | yes | | `spark4.2` | 4.2.0 | 1.17.0 | yes | This is why `BaseSpark4Adapter` carries a no-op that 4.1 and 4.2 each override identically: ```scala // TODO(#18935) drop-spark4.0: when all remaining 4.x adapters are parquet 1.16+, // apply variantType() in this base and delete the no-op default plus the // Spark4_1Adapter override. protected def applyVariantLogicalType(builder: Types.GroupBuilder[GroupType]) = builder ``` `hudi-spark4-common` is exactly where cross-version variant logic belongs. The floor evicts that logic into per-version modules -- the duplication the version-module layout exists to prevent. ## What it costs today **Variant features get pushed out of shared code.** In #18961, `Spark41VariantShreddingSchemaInferrer` had to go in `hudi-spark4.1.x` rather than `hudi-spark4-common`, because it needs parquet 1.16-era APIs. Compare `Spark4VariantShreddingProvider`, which *is* shared across all three minors because it only needs Spark's variant library. That split is the compile floor drawn in code. **Every variant PR pays a per-profile compile tax.** #18961 records compile checks under "the spark3.5, spark4.0 and spark4.1 profiles" -- three passes, soon four with 4.2. **~5,000 LOC.** `hudi-spark4.0.x` is 21 files, eight with no 4.1 counterpart -- the forked `ReadSupport` above, a forked legacy parquet format and reader, and a version-specific extended SQL parser and AST builder. **Refactor multiplication.** 34 commits touched the module since 2025-01-01, mostly cross-cutting refactors (#19193, #19195, #19147-#19150, #19132, #19455, #19460) applied N times because N version modules exist. ## Why now: sequencing against #18961 #18961 (auto-infer per-file shredding schemas) is what makes shredded files common in the wild for the first time. That flips a latent Spark 4.0 gap into a live one. - **Writes are safe.** The inferrer lives in `hudi-spark4.1.x`, so a 4.0 writer has none on the classpath and keeps writing unshredded. - **The AVRO read path is safe on 4.0.** `HoodieVariantReconstruction` resolves `Spark4VariantShreddingProvider` from `hudi-spark4-common`, and fails loudly rather than dropping `typed_value`. - **The gap is a 4.0 *native* read of a shredded file written by a 4.1/4.2 writer** in a mixed fleet. There `buildFullVariantReadSchema` returns the base `None`, plus SPARK-54410. I have not verified whether that last case throws `MALFORMED_VARIANT` or returns wrong results -- and it is awkward to test today because no unit job runs `hudi-spark4.0.x` (all `spark4.0` rows in `bot.yml` are commented out under `[CI-TRIM]`; only `spark3.5` and `spark4.2` run unit tests). **If we keep 4.0**, that test plus likely an explicit read guard become prerequisites for flipping the inference default (#18937). **If we drop it**, neither is needed. ## Scope: what this does not fix Worth stating plainly so expectations are right. This does **not** make Hudi's variant files self-describing to other engines. AVRO is the default write-path record type, and it converts variants via `AvroSchemaConverterWithTimestampNTZ`, which emits a bare `GroupType` with no annotation. That converter lives in `hudi-hadoop-common`, which must keep compiling under `spark3.5` at parquet 1.13.1 -- so it cannot reference the annotation regardless of what happens to 4.0. Only `HoodieRowParquetWriteSupport` reaches `applyVariantLogicalType`. Consequences: 1. The floor rises for `hudi-spark4-common` only. 2. `isVariantPhysicalSchema` (the name/arity shape heuristic) **stays** as the fallback for unannotated files. It cannot be deleted by this change. Annotating the AVRO path is a follow-up blocked on Spark 3.5, not on 4.0. I will file it separately. ## Who is actually affected Vendors shipping Spark 4.0 do not ship a Hudi version this change would reach: | Runtime | Spark | Hudi | |---|---|---| | EMR `emr-spark-8.0` | 4.0 | 1.0.2 | | EMR 7.10.0 | 3.5.5-amzn-1 | 0.15.0-amzn-7 | | Dataproc 2.3 | 3.5.3 | 0.15.0 (optional component) | Databricks Runtime 17.3 LTS ships Spark 4.0.0 into Oct 2028, but does not bundle Hudi. So the only population this could strand is self-managed deployments on Hudi 1.3.0+ with Spark 4.0 -- who control both versions and can move to 4.1/4.2. Existing Spark 4.0 users also keep working 1.2.x bundles; this removes a build profile going forward, it does not break deployments. Happy to be corrected if there is a population I have missed. ## Proposed motion Drop `spark4.0` in 1.3.0: - [ ] Remove the profile and module; update the README build matrix. - [ ] Move `hudi_trino_e2e.yml` (currently hardcoded to `-Dspark4.0` / the `spark402` compose) onto a higher Spark compose -- needs `trinocoordinator` added to the target pair. - [ ] Collapse `applyVariantLogicalType` into `BaseSpark4Adapter`, delete both overrides. - [ ] Delete the bundle-validation `spark4.0.0` rows and `spark400` base images. - [ ] Keep `isVariantPhysicalSchema` -- still needed for unannotated files. **Open question:** does Hudi require a deprecation-notice release before removal? If so, deprecate in 1.3.0 and remove in 1.4.0. ## References - [Parquet 1.16.0 release notes](https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0) | [parquet-java#3070 "Add Variant Logical Type"](https://github.com/apache/parquet-java/issues/3070) | [Parquet variant spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) - [Spark versioning policy](https://spark.apache.org/versioning-policy.html) | [Spark EOL dates](https://endoflife.date/apache-spark) - [Spark 4.0 on Amazon EMR](https://aws.amazon.com/blogs/big-data/announcing-general-availability-of-apache-spark-4-0-on-amazon-emr/) | [EMR 7.10.0](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-7100-release.html) | [Dataproc 2.3](https://docs.cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.3) - #18935 (cleanup umbrella) | #18334 (4.0 field-order workaround) | #18961 (auto-inference) | #18937 (default flip) - SPARK-54410 (read variant fields by name, fixed in 4.1) | SPARK-53659 (per-file shredding inference) GitHub link: https://github.com/apache/hudi/discussions/19585 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
