voonhous opened a new issue, #19690: URL: https://github.com/apache/hudi/issues/19690
### Goal Flip `hoodie.parquet.variant.shredding.schema.inference.enabled` (#18961) to default `true`, so variant columns shred automatically on Spark 4.1+ writers with no configuration. ### Ordering Deliberately last in the shredding sequence, each step its own PR: 1. #18961 - per-file shredding-schema inference (config lands default off). 2. #19687 - mixed-layout test matrix + fail-fast guards (Hive, Spark 4.0, schema-on-read) and shredded-read fixes, so no engine silently drops payloads once shredded files are everywhere. 3. #19689 - nested shredding parity between `HoodieRowParquetWriteSupport` and `HoodieAvroWriteSupport`, plus nested awareness in the guards. 4. This issue - the flip. ### Work items - Flip the default in `HoodieStorageConfig.PARQUET_VARIANT_SHREDDING_SCHEMA_INFERENCE_ENABLED` and update its `withDocumentation` text (default true, Spark 4.1+ writers only, per-file layouts including native parquet log files on table version 10+, buffering of up to 4096 records / 64MB per open writer, force DDL wins, `hoodie.internal.schema` disables it). - Default-on sweep: run the variant suites (`TestVariantDataType`, `TestVariantShreddingMixedLayouts`, `TestStreamingSource` variant tests, adapter suites) on spark4.1 and spark4.2 with the new default; update tests whose unshredded-layout assertions now see inferred `typed_value` - those deltas ARE the behavior change, enumerate them in the PR. - E2E with the default on: `ITTestCustomTypeHiveSync` (count/DESCRIBE stay green; a variant projection must hit the #19687 fail-fast), Trino behavior recorded, Flink fail-fast pin. - Release note: variant columns shred per file by default on Spark 4.1+; readers need Spark 4.1+; opt out via `hoodie.parquet.variant.shredding.schema.inference.enabled=false` or `hoodie.parquet.variant.write.shredding.enabled=false`; existing files unaffected; rewrite back to unshredded via clustering with inference off. - Cross-engine caveats documented: Spark 4.0, Hive, Trino cannot read shredded files (fail fast per #19687); Flink throws. - Website/docs PR on asf-site (separate PR, per convention): variant page section on shredding and inference, cross-engine matrix, how to un-shred. ### Blocked by #18937 (inference), #19688 (guards/fixes), #19689 (nested parity). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
