zeroshade commented on code in PR #47456: URL: https://github.com/apache/arrow/pull/47456#discussion_r2331266068
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -417,7 +421,133 @@ better zero-copy compatibility with various systems that also store booleans usi Metadata is an empty string. -========================= +.. _parquet_variant_extension: + +Parquet Variant +=============== + +Variant represents a value that may be one of: + +* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``) + +* Array: An ordered list of Variant values + +* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys + +Particularly, this provides a way to represent semi-structured data which is stored as a +`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in +a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__ +variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version +or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links +to the Parquet format specification for details on what the actual binary values should look like. Review Comment: updated with this text. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
