On Sat, 22 Jan 2022 at 20:27, Rok Mihevc <rok.mih...@gmail.com> wrote:
>
> Thanks for the input Weston!
>
> How about arrow/experimental/format/ExtensionTypes.fbs or
> arrow/format/ExtensionTypes.fbs for language independent schema and
> loosely arrow/<IMPLEMENTATION>/extensions for implementations?
>
> Having machine readable definitions could perhaps be useful for
> generating implementations in some cases.
Is it useful to put this in a flatbuffer file? Based on the list from
Weston just below, I think this will mostly contain a *description* of
those different aspect (a specification of the extension type), and
there is no data that actually fits in a flatbuffer table? In that
case a plain text (eg markdown) file seems more fitting?
>
> > * The name of the extension type (to go in ARROW:extension:name)
> > * A description of the extension type and how it should be used
> > * The storage type of the extension type
> > * The format and meaning of the content that will go into
> > ARROW:extension:metadata
>
> These sound pretty complete!
>
> I'll wait for a couple of days to see if there's more input and then
> draft a PR. Do we need a vote on this?
>
>
> Best,
> Rok
>
> On Fri, Jan 21, 2022 at 3:07 AM Weston Pace <weston.p...@gmail.com> wrote:
> >
> > Those all seem to be C++ locations. If we want to define
> > cross-implementation "Well Known Extension Types" then it seems we
> > would want to come up with some kind of language independent agreement
> > (could just be a markdown file but maybe there is some advantage to
> > having something programmatically consumable) describing:
> >
> > * The name of the extension type (to go in ARROW:extension:name)
> > * A description of the extension type and how it should be used
> > * The storage type of the extension type
> > * The format and meaning of the content that will go into
> > ARROW:extension:metadata
> >
> > I think (but am not sure) that, since these are metadata keys, we are
> > supposed to stick to printable ASCII for values (for backwards
> > compatibility).
> >
> > For example, in the docs, we currently have this little blurb about a
> > theoretical tensor extension type:
> >
> > > tensor (multidimensional array) stored as Binary values and
> > > having serialized metadata indicating the data type and shape
> > > of each value. This could be JSON like {'type': 'int8', 'shape':
> > > [4, 5]} for a 4x5 cell tensor.
> >
> > In my mind this file would be somewhat analogous to the way that
> > schema.fbs is the cross implementation "ground truth" for our logical
> > types.
> >
> > Then the C++ implementation would be free to put the implementation
> > (I'd vote for arrow/cpp/extensions but a separate repo is probably ok.
> > I'm -1 on arrow/extensions/...)
> >
> > On Thu, Jan 20, 2022 at 3:20 PM Rok Mihevc <rok.mih...@gmail.com> wrote:
> > >
> > > To continue the ExtensionType part of this thread - I would like to
> > > add TensorArray [1] as an ExtensionType to Arrow but we have not yet
> > > agreed on an "official" location for "Well Known Extension Types".
> > >
> > > Where could we put these? Some suggestions:
> > >
> > > * implementation folders (e.g. arrow/cpp/extensions/tensor_array.h)
> > > * extensions folder (e.g. arrow/extensions/cpp/tensor_array.h)
> > > * separate repo (e.g.
> > > github.com/apache/arrow_extensions/cpp/tensor_array.h)
> > >
> > > I'd be happy to also gather other Well Known Extension Types into one
> > > location if this moves forward.
> > >
> > > Rok
> > >
> > > [1] https://github.com/apache/arrow/pull/8510#issuecomment-991150389
> > >
> > > On Sat, May 1, 2021 at 12:12 PM Andrew Lamb <al...@influxdata.com> wrote:
> > > >
> > > > I agree with others on this thread. Thanks for writing this down Micah
> > > >
> > > > On Fri, Apr 30, 2021 at 11:16 AM Antoine Pitrou <anto...@python.org>
> > > > wrote:
> > > >
> > > > >
> > > > > I concur with both what Wes and Micah said.
> > > > >
> > > > > As for temporal types, they have wide-spread use and their semantics
> > > > > require dedicated treatment for arithmetic and conversion, so it's
> > > > > helpful to define dedicated types for them, as opposed to mere
> > > > > annotations.
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > > > >
> > > > > Le 30/04/2021 à 16:40, Wes McKinney a écrit :
> > > > > > I agree that the bar for adding new types to the Type union in
> > > > > > Schema.fbs
> > > > > > should be quite high going forward. Using extension types
> > > > > > increasingly
> > > > > for
> > > > > > adding specializations of built-in types will mean less burden for
> > > > > > implementations to simply "propagate forward" this data (by
> > > > > > preserving
> > > > > the
> > > > > > extra metadata) even if they don't understand what it does. It
> > > > > > would be
> > > > > > nice, therefore, to put us on a path to expanding our set of
> > > > > > "official"
> > > > > > extension types (which would include things like JSON or UUID)
> > > > > > since some
> > > > > > libraries may choose to implement convenience containers for these
> > > > > > for
> > > > > > usability.
> > > > > >
> > > > > > On Fri, Apr 30, 2021 at 9:22 AM Brian Hulette <bhule...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> +1 this looks good to me.
> > > > > >>
> > > > > >> My only concern is with criteria #3 " Is the underlying encoding
> > > > > >> of the
> > > > > >> type already semantically supported by a type?". I think this is a
> > > > > >> good
> > > > > >> criteria, but it's inconsistent with the current spec. By that
> > > > > >> criteria
> > > > > >> some existing types (Timestamp, Time, Duration, Date) should be
> > > > > >> well
> > > > > known
> > > > > >> extension types, right?
> > > > > >>
> > > > > >> Perhaps we should explicitly indicate these types are
> > > > > >> grandfathered in
> > > > > [1]
> > > > > >> because they existed before extension types, to avoid tension with
> > > > > >> this
> > > > > >> criteria.
> > > > > >>
> > > > > >> Brian
> > > > > >>
> > > > > >> [1] https://en.wikipedia.org/wiki/Grandfather_clause
> > > > > >>
> > > > > >> On Thu, Apr 29, 2021 at 9:13 PM Jorge Cardoso Leitão <
> > > > > >> jorgecarlei...@gmail.com> wrote:
> > > > > >>
> > > > > >>> Thanks for writing this.
> > > > > >>>
> > > > > >>> I agree. That is a good decision tree. +1
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Jorge
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Apr 29, 2021 at 6:08 PM Micah Kornfield
> > > > > >>> <emkornfi...@gmail.com
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> The discussion around adding another interval type to the
> > > > > >>>> Schema.fbs
> > > > > >>> raises
> > > > > >>>> the issue of when do we decide to add a new type to the
> > > > > >>>> Schema.fbs vs
> > > > > >>> using
> > > > > >>>> other means (primarily extension types [1]).
> > > > > >>>>
> > > > > >>>> A few criteria come to mind that could help decide (feedback
> > > > > >>>> welcome):
> > > > > >>>>
> > > > > >>>> 1. Is the type a new parameterization of an existing type?
> > > > > >>>> - If Yes, and we believe the parameterization is useful and
> > > > > >>>> can
> > > > > be
> > > > > >>> done
> > > > > >>>> in a forward/backward compatible manner then we would update
> > > > > >> Schema.fbs.
> > > > > >>>>
> > > > > >>>> 2. Does the type itself have its own specification for
> > > > > >>>> processing
> > > > > >> (e.g.
> > > > > >>>> JSON, BSON, Thrift, Avro, Protobuf)?
> > > > > >>>> - If yes, we would NOT add them to Schema.fbs. I think this
> > > > > >>>> would
> > > > > >>>> potentially yield too many new types.
> > > > > >>>>
> > > > > >>>> 3. Is the underlying encoding of the type already semantically
> > > > > >> supported
> > > > > >>>> by a type? (e.g. if we want to encode physical lengths like
> > > > > >>>> meters
> > > > > >> these
> > > > > >>>> can be represented by an integer).
> > > > > >>>> - If yes, we would NOT update the specification. This seems
> > > > > >>>> like
> > > > > >> the
> > > > > >>>> exact use-case that extension types are meant for.
> > > > > >>>>
> > > > > >>>> * How does this apply to Interval? *
> > > > > >>>> Interval extends an existing type in the specification and
> > > > > >>>> multiple
> > > > > >>> "packed
> > > > > >>>> fields" cannot be easily communicated with the current version
> > > > > >>>> of the
> > > > > >>>> specification. Hence, I feel comfortable making the addition to
> > > > > >>> Schema.fbs
> > > > > >>>>
> > > > > >>>> * What does this mean for other common types? *
> > > > > >>>>
> > > > > >>>> I think as types come up that are very common but we don't want
> > > > > >>>> to add
> > > > > >> to
> > > > > >>>> the Schema.fbs we should invest in formalizing them as "Well
> > > > > >>>> Known"
> > > > > >>>> Extension types. In this scenario, we would update the
> > > > > >>>> specification
> > > > > >> to
> > > > > >>>> include how to specify the extension type metadata (and still
> > > > > >>>> require
> > > > > >> at
> > > > > >>>> least two libraries support the Extension type before inclusion
> > > > > >>>> as
> > > > > >> "Well
> > > > > >>>> Known").
> > > > > >>>>
> > > > > >>>> * Practical implications *
> > > > > >>>>
> > > > > >>>> I think this means the type system in Schema.fbs is mostly closed
> > > > > (i.e.
> > > > > >>>> there is a high bar for adding new types). One potentially
> > > > > >>>> useful type
> > > > > >> to
> > > > > >>>> have would be a "packed struct" that supports something similar
> > > > > >>>> to
> > > > > >> python
> > > > > >>>> struct library [2]. I think this would likely cover many
> > > > > >>>> extension
> > > > > >> type
> > > > > >>>> use-cases.
> > > > > >>>>
> > > > > >>>> Thoughts?
> > > > > >>>>
> > > > > >>>> -Micah
> > > > > >>>>
> > > > > >>>> [1]
> > > > > https://arrow.apache.org/docs/format/Columnar.html#extension-types
> > > > > >>>> [2] https://docs.python.org/3/library/struct.html
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
