voonhous opened a new pull request, #19691: URL: https://github.com/apache/hudi/pull/19691
### Describe the issue this Pull Request addresses Closes #18021. Test-only: verifies and pins how Spark 3.x behaves when `hoodie.schema.on.read.enable` meets a variant table. ### Summary and Changelog With schema-on-read enabled, Spark 3.x resolves the schema through the InternalSchema round trip, whose sentinel detection restores the VARIANT logical type - the exact input `HoodieSparkSchemaConverters` rejects on Spark 3.x. Verified empirically on the spark3.5 profile against a Spark 4.1-written COW variant table carrying a committed internal schema: | Read | Outcome | |---|---| | Auto-resolve, plain | `HoodieSchemaException` caused by `VARIANT type is only supported in Spark 4.0+` (the known, already-pinned rejection) | | Auto-resolve + `hoodie.schema.on.read.enable=true` | Same exception, same cause - the InternalSchema path fails identically, not worse | | Struct-DDL compat mode + `hoodie.schema.on.read.enable=true` | Same exception - internal-schema resolution overrides the user DDL, so the documented Spark 3.x compat mode breaks when the conf is on | So the path throws, but always loudly with the actionable message: no silent wrong data and no obscure secondary failure under `toSqlTypeHelper`. The compat-mode interaction is the one real caveat; real schema-on-read support for variant belongs to #18285. Changes: one Spark 3-gated test in `TestVariantDataType` (runs in the spark3.5 CI lane) pinning all three legs, plus the fixture `variant_backward_compat/variant_schema_on_read_cow.zip` (a Spark 4.1-written COW variant table with a committed internal schema) and its README entry. ### Impact None - test-only. Documents that Spark 3.x compat-mode readers must keep `hoodie.schema.on.read.enable` off for variant tables that carry an internal schema. ### Risk Level: none Test-only change; verified green on the spark3.5 profile (3 Spark 3 tests pass, Spark 4 tests gated-canceled). ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
