sdf-jkl opened a new issue, #50622: URL: https://github.com/apache/arrow/issues/50622
### Describe the enhancement requested The Primitive Type Mappings table in the Variant canonical extension is the normative list of Arrow types allowed for `typed_value`. The only place the table is referenced from is the `typed_value` storage rule, so its whole role is defining legal `typed_value` types: https://github.com/apache/arrow/blob/f10c93c000775c5fb30ca230f540358611091589/docs/source/format/CanonicalExtensions.rst?plain=1#L453 The table appears to be derived from the Variant *encoding* types table (https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types), but its role corresponds to the shredding spec's Shredded Value Types table (https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types), which defines what a `typed_value` column may be. The two tables differ, so the Arrow table currently allows storage with no valid Parquet shredded representation: - `Uint8 -> Int16`, `Uint16 -> Int32`, `Uint32 -> Int64`: the shredding spec only permits signed integer `typed_value` columns, so unsigned Arrow storage can never round trip (`Arrow Uint -> Parquet INT -> Arrow Int`). A `Uint8` column also cannot represent the full value domain of its claimed variant type (int16), so it can only act as a partial storage that forces fallback to `value`. See apache/arrow-rs#10416 for discussion. - `Null -> Null`: the shredding table has no null row; a variant null must be encoded in the `value` column, so a `Null` typed_value column has no Parquet representation either. In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned `typed_value` columns to Parquet with `INT(8/16/32, false)` annotations, which the shredding spec does not allow. Proposal: re-derive the table from the shredding spec's Shredded Value Types table. Each row maps one variant type to its Parquet shredded type and the Arrow storage type(s) able to represent its full value domain. The table then defines reconstruction (a `typed_value` column of the listed Arrow type holds values of exactly the listed variant type) and the write-side lowering (the Parquet column is the only valid Parquet representation). | Variant type | Parquet type | Arrow `typed_value` type | |-----------------|-------------------------------------------------|-----------------------------------| | boolean | BOOLEAN | Boolean | | int8 | INT32, INT(8, true) | Int8 | | int16 | INT32, INT(16, true) | Int16 | | int32 | INT32 | Int32 | | int64 | INT64 | Int64 | | float | FLOAT | Float32 | | double | DOUBLE | Float64 | | decimal4 | INT32, DECIMAL(P, S) | Decimal32(P, S), 1 <= P <= 9 | | decimal8 | INT64, DECIMAL(P, S) | Decimal64(P, S), 10 <= P <= 18 | | decimal16 | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S)| Decimal128(P, S), 19 <= P <= 38 | | date | INT32, DATE | Date32 | | time | INT64, TIME(false, MICROS) | Time64(us) | | timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) | | timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) | | timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) | | timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) | | binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView | | string | BYTE_ARRAY, STRING | Utf8 / LargeUtf8 / Utf8View | | uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | UUID extension type | The decimal precision bands follow the decimal table in [VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types): precision alone selects the physical type (the bands are disjoint, so the narrowest type is required), and scale must satisfy `0 <= S <= P`. This also resolves three existing ambiguities: the current `Time64` row gains its required unit (variant time is micros only), the timestamp rows gain explicit units, and the decimal rows gain their precision bounds. Related: #50620, apache/parquet-format#591 ### Component(s) Format -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
