voonhous opened a new issue, #19689: URL: https://github.com/apache/hudi/issues/19689
### Context Nested variant shredding is asymmetric between the two write supports: - `HoodieRowParquetWriteSupport` (bulk-insert row writer and the SPARK record-type file writers): `processNestedDataType` recurses, so a shredding schema reaches a variant nested inside a struct/array/map. Pinned end to end by the nested test in #19687. - `HoodieAvroWriteSupport` (AVRO record type, Java client, future Flink): `applyForcedShreddingSchema` walks top-level fields only, so a nested variant silently declines to the unshredded layout. Data is correct, but the layouts diverge per record type for the same table and config. Unshredded nested variants are fully supported everywhere. The read side is already nested-capable: `HoodieVariantReconstruction` and `VariantSchemaUtils.alignShreddedVariants` recurse into records/arrays/maps (#19582/#19620), and Spark 4.1+ reads nested shredded groups natively. Ordering: do this AFTER #18961 lands (it touches the same write-support and `VariantSchemaUtils` code; rebasing the other way is churn). ### Scope **Write side (the actual parity change)** - `HoodieAvroWriteSupport.applyForcedShreddingSchema`: recurse into records, arrays and maps, splicing the shredded variant schema at any depth, mirroring `processNestedDataType`. `Spark4VariantShreddingProvider.shredVariantRecord` needs no change: it operates per variant value and already builds nested `typed_value` trees. - The strip path (`generateEffectiveSchema` with shredding disabled) already strips `typed_value` at every depth (#19620); keep its pin green. - Inference stays top-level-only on all paths (its documented scope). Growing inference into nested columns is a separate decision, out of scope here. **Guards and detection (must grow nested awareness in the same PR)** Once production paths can write nested-shredded files, the top-level-only sites become silent holes: | Site | Today | |---|---| | `HiveHoodieReaderContext` fail-fast guard | Top-level requested columns only; a nested-shredded file reads as silent nulls on Hive | | `ParquetSchemaEvolutionUtils` schema-on-read guard | Top-level only | | `TableSchemaResolver` footer fallback strip (`stripVariantShreddingByShape`, #18961) | Top-level only; a nested typed_value would leak into the resolved table schema | | `HoodieFileGroupReaderBasedFileFormat.supportBatch` | Disables vectorization for top-level `VariantType` only; nested shredded + the nested-column vectorized reader is unverified | Already nested-capable, no change needed: `alignShreddedVariants` / `isShreddedVariantTarget` rebuild-plan recursion, `HoodieVariantReconstruction`, `stripVariantShredding`. ### Tests - Extend `TestHoodieAvroWriteSupportShredding`: forced nested shredding on the Avro path (variant in struct/array/map, plus decline cases), asserting the parquet layout matches the row writer's for the same schema. - Extend the nested leg of `TestVariantShreddingMixedLayouts`: both record types write nested-shredded, then merge, compaction and clustering over the files. - Nested legs for the Hive and schema-on-read guards (fail fast naming the nested column). - Watch #19442 (FIXED typed_value) as more nested shapes start flowing through rebuild. ### Out of scope - Inference recursing into nested columns (separate follow-up if wanted). - Off-Spark reconstruction of shredded values (Hive/Trino/Flink still fail fast). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
