Following up on this. Anyone interested in Variant, please take a look.
Kosta On Wednesday, August 19th, 2026 at 4:38 AM, Andrew Lamb <[email protected]> wrote: > Thanks Kosta -- > > For anyone else following along, my summary of the proposed change is to > explicitly disallow shreded columns in the Arrow representation > ("typed_value") that have no equivalent type in Variant (e.g. unsigned > integers, or Null). It makes sense to me, but I do think it is important to > have a wider discussion before making any changes. > > Andrew > > On Tue, Aug 18, 2026 at 2:02 PM Kosta Tarasov <[email protected]> > wrote: > > > Bumping up this thread. > > > > I submitted a PR to change the Arrow docs here - > > https://github.com/apache/arrow/pull/50810 > > > > @alamb suggested that it should go through the mailing list first, because > > the changes can be considered as changing the spec. > > > > Please take a look and let me know what you think. > > > > Thanks, > > Kosta > > > > > > > > Sent with Proton Mail secure email. > > > > On Thursday, July 23rd, 2026 at 6:32 PM, Kosta Tarasov < > > [email protected]> wrote: > > > > > Hi everyone, > > > > > > I was reviewing a PR (https://github.com/apache/arrow-rs/pull/10417) > > for an issue in arrow-rs - https://github.com/apache/arrow-rs/issues/10416 > > - and discovered an inconsistency between the Arrow spec and the Parquet > > spec. > > > > > > The issue proposed adding Unsigned integer typed_value support for > > Shredded Variant based on the Arrow spec - > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant > > > > > > It says - "A field named typed_value which can be a Primitive Type > > Mappings or a List, LargeList, ListView or Struct". Where Primitive Type > > Mapping is > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings > > > > > > This table is inconsistent with the Shredded Value Types table in the > > Parquet Shredding Spec - > > https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types > > > > > > Among other differences, the Arrow table allows Uint8/16/32 and Null > > typed_value columns, which have no valid Parquet shredded representation > > and can never round trip through Parquet. Implementations have already > > diverged over this: arrow-rs rejects unsigned typed_value, while Arrow Go > > accepts it and writes it to Parquet as INT(8/16/32, false), which the > > shredding spec does not allow. > > > > > > I propose refreshing the Arrow Parquet Variant spec table based on the > > one in the Parquet Shredding spec. > > > > > > I filed an issue in the Arrow repo with the proposed table - > > https://github.com/apache/arrow/issues/50622 > > > > > > Thank you, > > > Kosta > > >
