GitHub user voonhous edited a discussion: Proposal: drop Spark 4.0 support in Hudi 1.3.0
## TL;DR We want to push for VARIANT support. `hudi-spark4-common` is compiled once per Spark profile. While the `spark4.0` profile is in the build, that module's source cannot reference `VariantLogicalTypeAnnotation` at all -- the class does not exist in parquet 1.15.2. No config flag, runtime gate, or reflection shim buys that back: a compile floor is not a runtime problem. The consequence is that every variant feature needing parquet >= 1.16 must live in a per-version module instead of the shared one, or not exist at all. #18961 is the current proof: `Spark41VariantShreddingSchemaInferrer` had to be placed in `hudi-spark4.1.x` rather than `hudi-spark4-common`. Every future variant feature pays the same tax. Spark 4.0 reaches EOL on [2026-11-23](https://lists.apache.org/thread/cxyfz1zlqhltzd4dsk7wn9v4jdmljcd5). Dropping it in 1.3.0 raises `hudi-spark4-common` to parquet >= 1.16, removes ~5,000 LOC, and lets shared variant code be written once instead of per-minor. > **Edit:** revised after @wombatu-kun's review below. Three claims in the > original version were wrong or overstated -- the shredded-write correctness > claim, the "makes files self-describing" claim, and "zero active CI > coverage". All three are corrected inline and marked. The proposal stands on > the compile floor, which none of them touch. # Parquet-java Version | Profile | Spark | parquet-java | avro | jackson | `VariantLogicalTypeAnnotation` available | |---|---|---|---|---|---| | `spark3.5` | 3.5.3 | 1.13.1 | 1.11.x | 2.15.x | n/a (no VariantType) | | `spark4.0` | 4.0.2 | **1.15.2** | 1.12.0 | 2.18.2 | **no** | | `spark4.1` | 4.1.1 | 1.16.0 | 1.12.1 | 2.20.0 | yes | | `spark4.2` | 4.2.0 | 1.17.0 | 1.12.1 | 2.21.2 | yes | The argument is not "we need to move off parquet 1.15" -- `spark3.5` pins 1.13.1 and stays. It is narrower: **dropping 4.0 raises `hudi-spark4-common`'s compile floor to parquet >= 1.16**, so shared Spark 4 code can finally reference the variant APIs directly. ## Why the 1.15 / 1.16 boundary specifically matters parquet-java 1.16.0 added `LogicalTypeAnnotation.VariantLogicalTypeAnnotation` (GH-3070). Under `-Dspark4.0`, `parquet.version` resolves to 1.15.2, so any source file in `hudi-spark4-common` referencing that class simply fails to compile. That is why `BaseSpark4Adapter` carries a no-op: ```scala // TODO(#18935) drop-spark4.0: when all remaining 4.x adapters are parquet 1.16+, // apply variantType() in this base and delete the no-op default plus the // Spark4_1Adapter override. protected def applyVariantLogicalType(builder: Types.GroupBuilder[GroupType]) = builder ``` The shared base cannot apply the annotation, so 4.1 and 4.2 each override it identically. One dead default plus two duplicate overrides, purely to keep 4.0 compiling. This is not the usual "support an old version at runtime" problem where a version check suffices. `hudi-spark4-common` is exactly where cross-version variant logic belongs; the floor evicts that logic into per-version modules, which is the duplication the version-module layout exists to prevent. ### The tax, measured on a live PR #18961 (auto-infer per-file variant shredding schemas) is the cleanest illustration: - The inferrer had to go in `hudi-spark4.1.x`, not `hudi-spark4-common`, because it needs Spark 4.1's `InferVariantShreddingSchema`. - Its own risk statement records compile checks under "the spark3.5, spark4.0 and spark4.1 profiles" -- three compile passes per change, soon four with 4.2. - Its documented behavior is "Spark 4.0/Flink/Java silently keep writing unshredded." Contrast `Spark4VariantShreddingProvider`, which *is* in `hudi-spark4-common` and serves all three minors -- because it only needs Spark's variant library, not parquet 1.16. The split between those two classes is the compile floor, drawn in code. ## ~~Correctness, not just tidiness~~ (retracted) > **Correction, per @wombatu-kun.** The original version claimed Spark 4.0 > writes shredded files its reader cannot rebuild. That is **not reachable > today**. `HoodieSparkSchemaConverters` always builds unshredded variants: > > ```scala > case other if sparkAdapter.isVariantType(other) => > HoodieSchema.createVariant(recordName, nameSpace, null) // never > createVariantShredded > ``` > > Shredding engages only when the schema already carries `typed_value`, or via > the test-only `hoodie.parquet.variant.force.shredding.schema.for.test`. > `hoodie.parquet.variant.write.shredding.enabled` defaults true but sits > downstream of a schema that never carries `typed_value` in production. The > 4.0 read gap is **latent**, not live. The annotation gap, by contrast, is > unconditional -- that is the one to lead with. ## Why #18961 changes the sequencing This is the part that matters for ordering, and the reason I would rather drop 4.0 *before* auto-inference lands than after. Auto-inference is what makes shredded files common in the wild for the first time. It converts the latent 4.0 read gap into a live one: - **Write side is guarded.** `Spark41VariantShreddingSchemaInferrer` lives in `hudi-spark4.1.x`, so a 4.0 writer has no inferrer on the classpath and keeps writing unshredded. 4.0 will not produce bad files. - **The AVRO read path is fine on 4.0.** `HoodieVariantReconstruction` resolves `Spark4VariantShreddingProvider` from `hudi-spark4-common`, which 4.0 has, and it fails loudly rather than silently dropping `typed_value`. - **The exposed surface is a 4.0 *native* read of a shredded file written by a 4.1/4.2 writer** in a mixed fleet. There `buildFullVariantReadSchema` returns the base `None`, and 4.0's `ParquetUnshreddedVariantConverter` indexes converters positionally (SPARK-54410, fixed in 4.1). I have **not** verified whether that last case throws `MALFORMED_VARIANT` or returns wrong results. It should be tested before inference defaults on (#18937) -- and it is awkward to test today precisely because no unit-test job runs `hudi-spark4.0.x`. If we keep 4.0, that test plus probably an explicit read guard are prerequisites for flipping the default. If we drop 4.0, neither is needed. Dropping first is strictly less work. ## Scope: this does not make variant files self-describing > **Correction, per @wombatu-kun.** AVRO is the default write-path record type, > and the AVRO path converts variants through > `AvroSchemaConverterWithTimestampNTZ`: > > ```java > case VARIANT: > // Variant is represented as a record with value and metadata binary fields > return new GroupType(repetition, fieldName, > convertFields(schema.getFields(), schemaPath)); > ``` > > A bare `GroupType`, no annotation. That converter lives in > `hudi-hadoop-common`, which must keep compiling under `spark3.5` at parquet > 1.13.1, so it cannot reference the annotation class regardless of what > happens to 4.0. Only `HoodieRowParquetWriteSupport` reaches > `applyVariantLogicalType`. Two consequences: 1. The floor rises for `hudi-spark4-common` only. Files written through the AVRO path on 4.1/4.2 stay exactly as opaque to other engines as 4.0's. 2. `isVariantPhysicalSchema` (the name/arity shape heuristic) **cannot** be deleted -- it stays as the fallback for unannotated files, both historical and current. This kills Bucket B item 2 of the #18935 cleanup. Annotating the AVRO path is a separate follow-up, blocked on Spark 3.5 rather than on 4.0. Will file it alongside this. ## CI coverage > **Correction.** The original version said Spark 4.0 has zero active CI > coverage. That is wrong: `.github/workflows/hudi_trino_e2e.yml` builds and > tests with `-Dspark4.0` on push and PR to master (path-filtered). The accurate statement is narrower: in `bot.yml`, every `spark4.0` matrix row is commented out, so **no unit or functional job exercises `hudi-spark4.0.x`**. The `spark4.1` and some `spark3.5` rows are commented out too, under a shared `[CI-TRIM] to revisit for CI improvement` marker (19 occurrences) -- so this was a cost trim, not a 4.0-specific decision. Only `spark3.5` and `spark4.2` run unit tests today. For the drop PR: `hudi_trino_e2e.yml` hardcodes `-Dspark4.0` and `COMPOSE_PREFIX: docker-compose_hadoop340_hive2310_spark402`, and the two `spark402` composes are the only ones carrying a `trinocoordinator` service. Mechanical to move -- we already have higher Spark composes to switch onto, it just needs `trinocoordinator` added to the target pair. ## Adoption: who is actually on Spark 4.0 with recent Hudi? Worth separating "vendor ships Spark 4.0" from "vendor ships a Hudi version that this change would reach". The lag is large: - **Amazon EMR** -- Spark 4.0 ships in `emr-spark-8.0`, which bundles **Hudi 1.0.2**. EMR 7.x (the releases carrying Flink/Trino/Presto) bundles Hudi 0.15.0-amzn-7. - **Google Dataproc** -- 2.3 is on Spark 3.5; Spark 4.0.0 appears in 3.0, in preview as of Sept 2025. - **Databricks** -- DBR 17.3 LTS ships Spark 4.0.0 with support into Oct 2028, but DBR does not bundle Hudi, and Hudi-on-DBR is a marginal population. So the users this change could actually strand are self-managed deployments running Hudi 1.3.0+ on Spark 4.0 -- and those control both versions, so they can move to 4.1/4.2. Vendor users on Spark 4.0 are pinned to Hudi 1.0.x/0.15.x and will not see 1.3.0 for a long time regardless. Happy to be corrected here if anyone has data on a population I have missed. ## Proposed motion Drop the `spark4.0` profile and delete `hudi-spark-datasource/hudi-spark4.0.x` in 1.3.0. Supported matrix becomes 3.5, 4.1, 4.2, with `hudi-spark4-common` on parquet >= 1.16 so shared variant code can be written once. Drop-PR checklist: - [ ] Remove the `spark4.0` profile and module; update the README build matrix. - [ ] Move `hudi_trino_e2e.yml` onto a higher Spark compose (add `trinocoordinator` to the target pair). - [ ] Collapse `applyVariantLogicalType` into `BaseSpark4Adapter`, delete both overrides. - [ ] Delete the bundle-validation `spark4.0.0` rows and `spark400` base images. - [ ] Do **not** delete `isVariantPhysicalSchema` -- still needed for unannotated files. Open question for the list: does Hudi require a deprecation-notice release before removal? If so, deprecate in 1.3.0 and remove in 1.4.0. ## References - Parquet 1.16.0 release notes (GH-3070, VARIANT logical type annotation): https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0 - Parquet variant spec: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md - Spark versioning policy: https://spark.apache.org/versioning-policy.html - Spark EOL dates: https://endoflife.date/apache-spark - Spark 4.0 on Amazon EMR (`emr-spark-8.0`, Hudi 1.0.2): https://aws.amazon.com/blogs/big-data/announcing-general-availability-of-apache-spark-4-0-on-amazon-emr/ - Dataproc 2.3 release versions: https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.3 - #18935 -- umbrella for Spark 4.0 workaround cleanup - #18334 -- Spark 4.0 variant field-order workaround - #18961 -- auto-infer per-file variant shredding schemas - #18937 -- flip shredding inference default - SPARK-54410 -- read variant fields by name (fixed in 4.1) - SPARK-53659 -- per-file shredding schema inference GitHub link: https://github.com/apache/hudi/discussions/19585 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
