alamb commented on code in PR #50810: URL: https://github.com/apache/arrow/pull/50810#discussion_r3798158456
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -488,63 +488,82 @@ binary values look like. Primitive Type Mappings ----------------------- -+----------------------+------------------------+ -| Arrow Primitive Type | Variant Primitive Type | -+======================+========================+ -| Null | Null | -+----------------------+------------------------+ -| Boolean | Boolean (true/false) | -+----------------------+------------------------+ -| Int8 | Int8 | -+----------------------+------------------------+ -| Uint8 | Int16 | -+----------------------+------------------------+ -| Int16 | Int16 | -+----------------------+------------------------+ -| Uint16 | Int32 | -+----------------------+------------------------+ -| Int32 | Int32 | -+----------------------+------------------------+ -| Uint32 | Int64 | -+----------------------+------------------------+ -| Int64 | Int64 | -+----------------------+------------------------+ -| Float | Float | -+----------------------+------------------------+ -| Double | Double | -+----------------------+------------------------+ -| Decimal32 | decimal4 | -+----------------------+------------------------+ -| Decimal64 | decimal8 | -+----------------------+------------------------+ -| Decimal128 | decimal16 | -+----------------------+------------------------+ -| Date32 | Date | -+----------------------+------------------------+ -| Time64 | TimeNTZ | -+----------------------+------------------------+ -| Timestamp(us, UTC) | Timestamp (micro) | -+----------------------+------------------------+ -| Timestamp(us) | TimestampNTZ (micro) | -+----------------------+------------------------+ -| Timestamp(ns, UTC) | Timestamp (nano) | -+----------------------+------------------------+ -| Timestamp(ns) | TimestampNTZ (nano) | -+----------------------+------------------------+ -| Binary | Binary | -+----------------------+------------------------+ -| LargeBinary | Binary | -+----------------------+------------------------+ -| BinaryView | Binary | -+----------------------+------------------------+ -| String | String | -+----------------------+------------------------+ -| LargeString | String | -+----------------------+------------------------+ -| StringView | String | -+----------------------+------------------------+ -| UUID extension type | UUID | -+----------------------+------------------------+ +The following table defines the set of Arrow types that are valid as primitive +``typed_value`` storage. It is derived from the `Shredded Value Types +<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__ +table of the Parquet Variant Shredding specification: each row maps a Variant +primitive type to the Parquet type required for a shredded ``typed_value`` +column (physical type, followed by the logical type annotation if any) and to +the Arrow type(s) able to represent that Variant type's full value domain. +A ``typed_value`` field of one of the listed Arrow types holds values of +exactly the corresponding Variant type, and the listed Parquet type is its +only valid Parquet representation. + ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| Variant Type | Parquet Type | Arrow ``typed_value`` Type | ++========================================+==================================================+=============================================+ +| boolean | BOOLEAN | Boolean | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int8 | INT32, INT(8, true) | Int8 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int16 | INT32, INT(16, true) | Int16 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int32 | INT32 | Int32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int64 | INT64 | Int64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| float | FLOAT | Float32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| double | DOUBLE | Float64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) | Review Comment: https://parquet.apache.org/docs/file-format/types/variantencoding/ says > The Decimal type contains a scale, but no precision. The implied precision of a decimal value is floor(log_10(val)) + 1. I don't unersatnd how that maps to `1 <= P <= 9, 0 <= S <= P` 🤔 ########## docs/source/format/CanonicalExtensions.rst: ########## @@ -488,63 +488,82 @@ binary values look like. Primitive Type Mappings ----------------------- -+----------------------+------------------------+ -| Arrow Primitive Type | Variant Primitive Type | -+======================+========================+ -| Null | Null | -+----------------------+------------------------+ -| Boolean | Boolean (true/false) | -+----------------------+------------------------+ -| Int8 | Int8 | -+----------------------+------------------------+ -| Uint8 | Int16 | -+----------------------+------------------------+ -| Int16 | Int16 | -+----------------------+------------------------+ -| Uint16 | Int32 | -+----------------------+------------------------+ -| Int32 | Int32 | -+----------------------+------------------------+ -| Uint32 | Int64 | -+----------------------+------------------------+ -| Int64 | Int64 | -+----------------------+------------------------+ -| Float | Float | -+----------------------+------------------------+ -| Double | Double | -+----------------------+------------------------+ -| Decimal32 | decimal4 | -+----------------------+------------------------+ -| Decimal64 | decimal8 | -+----------------------+------------------------+ -| Decimal128 | decimal16 | -+----------------------+------------------------+ -| Date32 | Date | -+----------------------+------------------------+ -| Time64 | TimeNTZ | -+----------------------+------------------------+ -| Timestamp(us, UTC) | Timestamp (micro) | -+----------------------+------------------------+ -| Timestamp(us) | TimestampNTZ (micro) | -+----------------------+------------------------+ -| Timestamp(ns, UTC) | Timestamp (nano) | -+----------------------+------------------------+ -| Timestamp(ns) | TimestampNTZ (nano) | -+----------------------+------------------------+ -| Binary | Binary | -+----------------------+------------------------+ -| LargeBinary | Binary | -+----------------------+------------------------+ -| BinaryView | Binary | -+----------------------+------------------------+ -| String | String | -+----------------------+------------------------+ -| LargeString | String | -+----------------------+------------------------+ -| StringView | String | -+----------------------+------------------------+ -| UUID extension type | UUID | -+----------------------+------------------------+ +The following table defines the set of Arrow types that are valid as primitive +``typed_value`` storage. It is derived from the `Shredded Value Types +<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__ +table of the Parquet Variant Shredding specification: each row maps a Variant +primitive type to the Parquet type required for a shredded ``typed_value`` +column (physical type, followed by the logical type annotation if any) and to +the Arrow type(s) able to represent that Variant type's full value domain. +A ``typed_value`` field of one of the listed Arrow types holds values of +exactly the corresponding Variant type, and the listed Parquet type is its +only valid Parquet representation. + ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| Variant Type | Parquet Type | Arrow ``typed_value`` Type | Review Comment: Perhaps we could also add a reference to the location of the parquet and arrow type lists I think the arrow ones are here: https://arrow.apache.org/docs/format/Columnar.html#data-types The parquet ones are here: https://parquet.apache.org/docs/file-format/types/ https://parquet.apache.org/docs/file-format/types/logicaltypes/ I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (`INT64`) , and some seem to have both logical and physical (`INT32, INT(8, true) `). It would help to be spcific about what the Parquet type column represents and use it consistently ########## docs/source/format/CanonicalExtensions.rst: ########## @@ -488,63 +488,82 @@ binary values look like. Primitive Type Mappings ----------------------- -+----------------------+------------------------+ -| Arrow Primitive Type | Variant Primitive Type | -+======================+========================+ -| Null | Null | -+----------------------+------------------------+ -| Boolean | Boolean (true/false) | -+----------------------+------------------------+ -| Int8 | Int8 | -+----------------------+------------------------+ -| Uint8 | Int16 | -+----------------------+------------------------+ -| Int16 | Int16 | -+----------------------+------------------------+ -| Uint16 | Int32 | -+----------------------+------------------------+ -| Int32 | Int32 | -+----------------------+------------------------+ -| Uint32 | Int64 | -+----------------------+------------------------+ -| Int64 | Int64 | -+----------------------+------------------------+ -| Float | Float | -+----------------------+------------------------+ -| Double | Double | -+----------------------+------------------------+ -| Decimal32 | decimal4 | -+----------------------+------------------------+ -| Decimal64 | decimal8 | -+----------------------+------------------------+ -| Decimal128 | decimal16 | -+----------------------+------------------------+ -| Date32 | Date | -+----------------------+------------------------+ -| Time64 | TimeNTZ | -+----------------------+------------------------+ -| Timestamp(us, UTC) | Timestamp (micro) | -+----------------------+------------------------+ -| Timestamp(us) | TimestampNTZ (micro) | -+----------------------+------------------------+ -| Timestamp(ns, UTC) | Timestamp (nano) | -+----------------------+------------------------+ -| Timestamp(ns) | TimestampNTZ (nano) | -+----------------------+------------------------+ -| Binary | Binary | -+----------------------+------------------------+ -| LargeBinary | Binary | -+----------------------+------------------------+ -| BinaryView | Binary | -+----------------------+------------------------+ -| String | String | -+----------------------+------------------------+ -| LargeString | String | -+----------------------+------------------------+ -| StringView | String | -+----------------------+------------------------+ -| UUID extension type | UUID | -+----------------------+------------------------+ +The following table defines the set of Arrow types that are valid as primitive +``typed_value`` storage. It is derived from the `Shredded Value Types +<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__ +table of the Parquet Variant Shredding specification: each row maps a Variant +primitive type to the Parquet type required for a shredded ``typed_value`` +column (physical type, followed by the logical type annotation if any) and to +the Arrow type(s) able to represent that Variant type's full value domain. +A ``typed_value`` field of one of the listed Arrow types holds values of +exactly the corresponding Variant type, and the listed Parquet type is its +only valid Parquet representation. + ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| Variant Type | Parquet Type | Arrow ``typed_value`` Type | ++========================================+==================================================+=============================================+ +| boolean | BOOLEAN | Boolean | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int8 | INT32, INT(8, true) | Int8 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int16 | INT32, INT(16, true) | Int16 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int32 | INT32 | Int32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| int64 | INT64 | Int64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| float | FLOAT | Float32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| double | DOUBLE | Float64 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64, DECIMAL(P, S) | Decimal64(P, S) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S) | Decimal128(P, S) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| date | INT32, DATE | Date32 | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| time | INT64, TIME(false, MICROS) | Time64(us) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| string | BYTE_ARRAY, STRING | String / LargeString / StringView | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ +| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | :ref:`UUID extension type <uuid_extension>` | ++----------------------------------------+--------------------------------------------------+---------------------------------------------+ + +The decimal precision bands follow the `Variant encoding types Review Comment: I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
