Note that you can already put such metadata into the footer by just putting it into the regular key-value metadata. Put a JSON array as value there with the same number of entries as the schema, then you have an implicit 1-to-1 mapping per column. We already use this to store per-column metadata and haven't encountered any problems with it so far.
IMHO, this is actually enough. I wouldn't see what practical advantage an extra key-value section within the schema element would give you (unless we're going back to the use case with super wide schemas and you want to only decode parts of the schema). Sure, it's conceptually nicer to have it within the schema, but do you see any concrete disadvantage besides conceptual clarity of just putting it into the existing key value metadata section? Cheers, Jan Am So., 2. Nov. 2025 um 04:18 Uhr schrieb Dewey Dunnington < [email protected]>: > Point cloud data in Parquet...cool! > > This sounds similar to the concept of Field metadata in Arrow, where we > also have extension types but people do use ad-hoc metadata to convey > non-extension type information like you are describing. If you are using an > Arrow implementation to write/read the Parquet file, the embedded Arrow > schema may already be able to roundtrip that information. > > If this is specific to a domain (e.g., LiDAR), you could also invent a > top-level key/value metadata standard (this is what GeoParquet did before > Parquet GEOMETRY/GEOGRAPHY). A wider variety of Parquet > implementations/versions would be able to access this information than if > the Parquet thrift were updated. > > Many of the techniques used to encode LiDAR data/make rows more compact can > also be achieved by expanding things like bitpacked fields into multiple > columns or resolving things like scaled integers into an existing Parquet > type. Parquet's encodings and compression may be able to accomplish > something similar to how this is frequently stored in point cloud native > formats (although would make it harder to roundtrip). > > Apologies if I'm missing context here! > > Cheers, > > -dewey > > On Fri, Oct 31, 2025 at 2:04 PM Andrew Bell <[email protected]> > wrote: > > > On Fri, Oct 31, 2025 at 1:14 PM Micah Kornfield <[email protected]> > > wrote: > > > > > Hi Andrew, > > > If this is to support new type (point cloud data), is there a reason to > > > choose a key value member to the schema over something like the > extension > > > type proposal [1] > > > > > > > In some ways it's no different -- you're providing some data to ride > along > > with a column. The extension type has the advantage of providing an > > indirection which *might* be useful for the case when you have many > columns > > of the same type, though this seems a pretty specific use case and adds > > additional complexity. However, extension types provide no hint of > meaning > > to be found in the "serialization" field (JSON is suggested, which could > > provide keys, but would also require an additional parsing step). > > > > Allowing the addition of data to the existing SchemaElement is trivially > > simple and more flexible. Users could add whatever data they like to > > annotate their schema element without introducing anything to the type > > system. For example, one could add a description to an integer element > > without creating an "Integer with Description" type or provide language > > information about a string without creating a type "String in French". > > > > The extension type proposal suggests that readers will be modified to > > support the extension types. Adding metadata directly to the > SchemaElement > > simply allows code *outside* of a Parquet reader to use the information > for > > its own purpose -- a reader only needs to provide an API to access the > > metadata to be useful. > > > > Some examples from point cloud data: > > > > - Integers to which a scale and offset are applied to create a nominal > > value (the current integer-based scale/offset are insufficient). > > - Units for many types. > > - GPS times are stored in several ways -- having metadata which may or > may > > not include an offset allows for proper interpretation. > > - Descriptions of bit fields packed into integers. > > - Indication that "return" numbers are synthetically generated. (A laser > > pulse can create multiple points, each known as a "return"). > > > > There's certainly nothing that precludes doing both extension types and > > adding metadata support for SchemaElements. > > > > -- > > Andrew Bell > > [email protected] > > >
