sdf-jkl commented on issue #519: URL: https://github.com/apache/parquet-format/issues/519#issuecomment-5626492364
Picking this back up with a concrete implementation bug: [apache/arrow-rs#11050](https://github.com/apache/arrow-rs/issues/11050), found while working on apache/arrow-rs#10635. In arrow-rs, looking up `[0]` in `[]` currently returns SQL NULL for unshredded input but a valid Variant null for shredded input. Looking up `[0]` in `[null]` returns Variant null in both cases. The same logical input therefore produces different null semantics depending on its representation. @scovich’s #520 already proposes clearer terminology for missing/SQL NULL versus Variant null. It would be useful to revisit that work and also make the empty-array case explicit in the [Parquet Variant array rules](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#arrays). Suggested wording: > Arrays may be empty. All elements of a nonempty Variant array must be present (not missing), because the Variant array encoding does not allow missing elements. The existing `["horror", null]` example shows an explicit Variant null element. Adding an empty array would clarify the zero-element case: | Array | `value` | `typed_value` | `typed_value...value` | `typed_value...typed_value` | |---|---|---|---|---| | `[]` | NULL | non-NULL | `[]` | `[]` | Here, the trailing `[]` entries mean there are zero element-child values. The capitalization follows #520’s proposed terminology. The distinction from [required-value recovery](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) also seems worth spelling out: an empty array has zero element groups. An existing element group with both children absent occupies a position but has invalid contents, for which the reader must return Variant null. Could we also specify how missing-path results should be represented while preserving their distinction from Variant null? For `{"a":[]}`, `$.a` resolves to an empty array, while `$.a[0]` finds no element. This distinction should survive Variant-returning extraction regardless of shredding. Spark preserves it by returning SQL NULL for a missing path and Variant null for an existing null element. The arrow-rs issue provides a concrete example of the inconsistency that can arise when these cases are conflated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
