zeroshade commented on code in PR #47456: URL: https://github.com/apache/arrow/pull/47456#discussion_r2334743821
########## docs/source/format/CanonicalExtensions/Examples.rst: ########## @@ -0,0 +1,555 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _format_canonical_extension_examples: + +**************************** +Canonical Extension Examples +**************************** + +========================= +Parquet Variant Extension +========================= + + +Unshredded +'''''''''' + +The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of +the following storage types are valid (not an exhaustive list): + +* ``struct<metadata: binary non-nullable, value: binary nullable>`` +* ``struct<value: binary nullable, metadata: binary non-nullable>`` +* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>`` + +Simple Shredding +'''''''''''''''' + +Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency. +In Parquet, this could be represented as:: + + required group measurement (VARIANT) { + required binary metadata; + optional binary value; + optional int64 typed_value; + } + +Thus the corresponding storage type for the ``arrow.parquet.variant`` Arrow extension type would be:: + + struct< + metadata: binary non-nullable, + value: binary nullable, + typed_value: int64 nullable + > + +If we suppose a series of measurements consisting of:: + + 34, null, "n/a", 100 + +The data should be stored/represented in Arrow as:: + + * Length: 4, Null count: 1 + * Validity Bitmap buffer: + + | Byte 0 (validity bitmap) | Bytes 1-63 | + |--------------------------|---------------| + | 00001011 | 0 (padding) | + + * Children arrays: + * field-0 array (`VarBinary`) + * Length: 4, Null count: 0 + * Offsets buffer: + + | Bytes 0-19 | Bytes 20-63 | + |------------------|--------------------------| + | 0, 2, 4, 6, 8 | unspecified (padding) | + + * Value buffer: (01 00 -> indicates version 1 empty metadata) Review Comment: > 01(byte[0]) indicate the header with offset_size_minus_one = 0, and 00(byte[1]) means the dictionary_size = 0, so there would no offset list and bytes following. Correct > There are four 01 00 in the meta, because we have four variant/row(34, null, "n/a", 100) in all here, am I understand right? If it's yes, does this mean we'll have ${lenght_of_row} pair of <header,dictionay_size, offset, bytes> in the metadata VarBinary Correct. There should always be the same number of rows in the `metadata`, `value` and `typed_value` columns (if they exist) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
