sdf-jkl commented on code in PR #50810: URL: https://github.com/apache/arrow/pull/50810#discussion_r3804929724
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -488,63 +488,82 @@ binary values look like. Primitive Type Mappings ----------------------- -+----------------------+------------------------+ -| Arrow Primitive Type | Variant Primitive Type | -+======================+========================+ -| Null | Null | -+----------------------+------------------------+ -| Boolean | Boolean (true/false) | -+----------------------+------------------------+ -| Int8 | Int8 | -+----------------------+------------------------+ -| Uint8 | Int16 | -+----------------------+------------------------+ -| Int16 | Int16 | -+----------------------+------------------------+ -| Uint16 | Int32 | -+----------------------+------------------------+ -| Int32 | Int32 | -+----------------------+------------------------+ -| Uint32 | Int64 | -+----------------------+------------------------+ -| Int64 | Int64 | -+----------------------+------------------------+ -| Float | Float | -+----------------------+------------------------+ -| Double | Double | -+----------------------+------------------------+ -| Decimal32 | decimal4 | -+----------------------+------------------------+ -| Decimal64 | decimal8 | -+----------------------+------------------------+ -| Decimal128 | decimal16 | -+----------------------+------------------------+ -| Date32 | Date | -+----------------------+------------------------+ -| Time64 | TimeNTZ | -+----------------------+------------------------+ -| Timestamp(us, UTC) | Timestamp (micro) | -+----------------------+------------------------+ -| Timestamp(us) | TimestampNTZ (micro) | -+----------------------+------------------------+ -| Timestamp(ns, UTC) | Timestamp (nano) | -+----------------------+------------------------+ -| Timestamp(ns) | TimestampNTZ (nano) | -+----------------------+------------------------+ -| Binary | Binary | -+----------------------+------------------------+ -| LargeBinary | Binary | -+----------------------+------------------------+ -| BinaryView | Binary | -+----------------------+------------------------+ -| String | String | -+----------------------+------------------------+ -| LargeString | String | -+----------------------+------------------------+ -| StringView | String | -+----------------------+------------------------+ -| UUID extension type | UUID | -+----------------------+------------------------+ +The following table defines the set of Arrow types that are valid as primitive +``typed_value`` storage. It is derived from the `Shredded Value Types +<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__ +table of the Parquet Variant Shredding specification: each row maps a Variant +primitive type to the Parquet type required for a shredded ``typed_value`` +column (physical type, followed by the logical type annotation if any) and to +the Arrow type(s) able to represent that Variant type's full value domain. +A ``typed_value`` field of one of the listed Arrow types holds values of +exactly the corresponding Variant type, and the listed Parquet type is its +only valid Parquet representation. + ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| Variant Type | Parquet Type | Arrow ``typed_value`` Type | ++========================================+==================================================+=============================================+ +| boolean | BOOLEAN | Boolean | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int8 | INT32, INT(8, true) | Int8 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int16 | INT32, INT(16, true) | Int16 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int32 | INT32 | Int32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int64 | INT64 | Int64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| float | FLOAT | Float32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| double | DOUBLE | Float64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) | Review Comment: There's a small separate table for decimals that shows what precision can fit in the underlying physical type. https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L440-L447 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
