> > It might help this discussion and future discussions like it if we could > define how it is determined whether a type should be part of the Arrow > format, an extension type (and what does it mean to say there is a > "canonical" extension type), or just something that a language > implementation or downstream library builds for itself with metadata. I > feel like this has come up before but I don't recall a resolution.
There seemed to be consensus, but I guess we never formally voted on the decision points here: https://lists.apache.org/thread.html/r7ba08aed2809fa64537e6f44bce38b2cf740acbef0e91cfaa7c19767%40%3Cdev.arrow.apache.org%3E Applying the criteria to complex types: 1. Is the type a new parameterization of an existing type? No 2. Does the type itself have its own specification for processing (e.g. JSON, BSON, Thrift, Avro, Protobuf)? No 3. Is the underlying encoding of the type already semantically supported by a type? Yes. Two have been mentioned in this thread and I would also support adding a new packed struct type, but it appears isn't necessary for this. Note that FixedSizeLists have some limitations in regards to parquet compatibility around nullability, there might be a few other sharp edges. So if we use this criteria we would lean towards an extension type. We never converged on a standard for "canonical" extension types. I would propose it roughly be the same criteria as a first class type: 1. Specification/document update PR that describes the representation 2. Implementation showing working integration tests across two languages (for canonical types I think this can be any 2 languages instead of C++ and Java) 3. Formal vote accepting the canonical type. Thanks, Micah On Thu, Jun 10, 2021 at 9:34 PM Jorge Cardoso Leitão < jorgecarlei...@gmail.com> wrote: > Isn't an array of complexes represented by what arrow already supports? In > particular, I see at least two valid in-memory representations to use, that > depend on what we are going to do with it: > > * Struct[re, im] > * FixedList[2] > > In the first case, we have two buffers, [x0, x1, ...] and [y0, y1, ...], in > the second case we have 1 buffer, [x0, y0, x1, y1, ...]. > > The first representation is useful for column-based operations (e.g. taking > the real part in case 1 is trivial; requires a copy in the second case), > the second representation is useful for row-base operations (e.g. "take" > and "filter" require a single pass over buffer 1). Case 2 does not support > Re and Im of different physical types (arguably an issue). Both cases > support nullability of individual items or combined. > > What I conclude is that this does not seem to be a problem about a base > in-memory representation, but rather on whether we agree on a > representation that justifies adding associated metadata to the spec. > > The case for the complex interval type recently proposed [1] is more > compelling to me because a complex ops over intervals usually required all > parts of the interval (and thus the "FixedList" representation is more > compelling), but each part has a different type. I.e. it is like a > "FixedTypedList[int32, int32, int64]", which we do not natively support. > > [1] https://github.com/apache/arrow/pull/10177 > > Best, > Jorge > > > > On Fri, Jun 11, 2021 at 1:48 AM Neal Richardson < > neal.p.richard...@gmail.com> > wrote: > > > It might help this discussion and future discussions like it if we could > > define how it is determined whether a type should be part of the Arrow > > format, an extension type (and what does it mean to say there is a > > "canonical" extension type), or just something that a language > > implementation or downstream library builds for itself with metadata. I > > feel like this has come up before but I don't recall a resolution. > > > > Examples might also help: are there examples of "canonical extension > > types"? > > > > Neal > > > > On Thu, Jun 10, 2021 at 4:20 PM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > > > > > > > > > My understanding is that it means having COMPLEX as an entry in the > > > > arrow/type_fwd.h Type enum. I agree this would make implementation > > > > work in the C++ library much more straightforward. > > > > > > One idea I proposed would be to do that, and implement the > > > > serialization of the complex metadata using Extension types. > > > > > > > > > If this is a maintainable strategy for Canonical types it sounds good > to > > > me. > > > > > > On Thu, Jun 10, 2021 at 4:02 PM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > My understanding is that it means having COMPLEX as an entry in the > > > > arrow/type_fwd.h Type enum. I agree this would make implementation > > > > work in the C++ library much more straightforward. > > > > > > > > One idea I proposed would be to do that, and implement the > > > > serialization of the complex metadata using Extension types. > > > > > > > > On Thu, Jun 10, 2021 at 5:47 PM Weston Pace <weston.p...@gmail.com> > > > wrote: > > > > > > > > > > > While dedicated types are not strictly required, compute > functions > > > > would > > > > > > be much easier to add for a first-class dedicated complex > datatype > > > > > > rather than for an extension type. > > > > > @pitrou > > > > > > > > > > This is perhaps a naive question (and admittedly, I'm not up to > speed > > > > > on my compute kernels) but why is this the case? For example, if > > > > > adding a complex addition kernel it seems we would be talking > > about... > > > > > > > > > > dest_scalar.real = scalar1.real + scalar2.real; > > > > > dest_scalar.im = scalar1.im + scalar2.im; > > > > > > > > > > vs... > > > > > > > > > > dest_scalar[0] = scalar1[0] + scalar2[0]; > > > > > dest_scalar[1] = scalar1[1] + scalar2[1]; > > > > > > > > > > On Thu, Jun 10, 2021 at 11:27 AM Wes McKinney <wesmck...@gmail.com > > > > > > wrote: > > > > > > > > > > > > I'd be supportive of starting with this as a "canonical" > extension > > > > > > type so that all implementations are not expected to support > > complex > > > > > > types — this would encourage us to build sufficient integration > > e.g. > > > > > > with NumPy to get things working end-to-end with the on-wire > > > > > > representation being an extension type. We could certainly choose > > to > > > > > > treat the type as "first class" in the C++ library without it > being > > > > > > "top level" in the Type union in Flatbuffers. > > > > > > > > > > > > I agree that the use cases are more specialized, and the fact > that > > we > > > > > > haven't needed it until now (or at least, its absence suggests > > this) > > > > > > shows that this is the case. > > > > > > > > > > > > On Thu, Jun 10, 2021 at 4:17 PM Micah Kornfield < > > > emkornfi...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I'm convinced now that first-class types seem to be the way > to > > > go > > > > and I'm > > > > > > > > happy to take this approach. > > > > > > > > > > > > > > I agree from an implementation effort it is simpler, but I'm > > still > > > > not > > > > > > > convinced that we should be adding this as a first class type. > > As > > > > noted in > > > > > > > the survey below it appears Complex numbers are not a core > > concept > > > > in many > > > > > > > general purpose coding languages and it doesn't appear to be a > > > > common type > > > > > > > in SQL systems either. > > > > > > > > > > > > > > The reason why I am being nit-picky here is I think that > having a > > > > first > > > > > > > class type indicates that it should eventually be supported by > > all > > > > > > > reference implementations. An "well known" extension type I > > think > > > > offers > > > > > > > less guarantees which makes it seem more suitable for niche > > types. > > > > > > > > > > > > > > > I don't immediately see a Packed Struct type. Would this need > > to > > > be > > > > > > > > > implemented? > > > > > > > > Not necessarily (*). But before thinking about > implementation, > > > > this > > > > > > > > proposal must be accepted into the format. > > > > > > > > > > > > > > > > > > > > > Yes, this is a type that has been proposed in the past and I > > think > > > > handles > > > > > > > a lot of types not yet in Arrow but have been requested (e.g. > IP > > > > > > > Addresses, Geo coordinates), etc. > > > > > > > > > > > > > > On Thu, Jun 10, 2021 at 1:06 AM Simon Perkins < > > > > simon.perk...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > On Wed, Jun 9, 2021 at 7:56 PM Antoine Pitrou < > > > anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Le 09/06/2021 à 17:52, Micah Kornfield a écrit : > > > > > > > > > > > > > > > > > > > > Adding a new first-class type in Arrow requires working > > > > integration > > > > > > > > tests > > > > > > > > > > between C++ and Java libraries (once the idea is > informally > > > > agreed > > > > > > > > upon) > > > > > > > > > > and then a final vote for approval. We haven't > formalized > > > > extension > > > > > > > > > types > > > > > > > > > > but I imagine a similar cross language requirement would > be > > > > agreed > > > > > > > > upon. > > > > > > > > > > Implementation of computation wouldn't be required for > > adding > > > > a new > > > > > > > > type. > > > > > > > > > > Different language bindings have taken different > approaches > > > on > > > > how much > > > > > > > > > > additional computational elements are packaged in them. > > > > > > > > > > > > > > > > > > While dedicated types are not strictly required, compute > > > > functions would > > > > > > > > > be much easier to add for a first-class dedicated complex > > > > datatype > > > > > > > > > rather than for an extension type. > > > > > > > > > > > > > > > > > > Since complex numbers are quite common in some domains, and > > > > since they > > > > > > > > > are conceptually simply, IMHO it would make sense to add > them > > > to > > > > the > > > > > > > > > native Arrow datatypes (at least COMPLEX64 and COMPLEX128). > > > > > > > > > > > > > > > > > > > > > > > > > I'm convinced now that first-class types seem to be the way > to > > > go > > > > and I'm > > > > > > > > happy to take this approach. > > > > > > > > Regarding compute functions, it looks like the standard set > of > > > > scalar > > > > > > > > arithmetic and reduction functionality > > > > > > > > is desirable for complex numbers: > > > > > > > > https://arrow.apache.org/docs/cpp/compute.html# > > > > > > > > Perhaps it would be better to split the addition of the Types > > and > > > > addition > > > > > > > > Compute functionality into separate PRs? > > > > > > > > > > > > > > > > Regarding the process for managing this PR, it sounds like a > > > > proposal must > > > > > > > > be voted on? > > > > > > > > i.e. is this proposal still in this phase > > > > > > > > > > > > > > > > > > http://arrow.apache.org/docs/developers/contributing.html#before-starting > > > > > > > > Regards > > > > > > > > > > > > > > > > Simon > > > > > > > > > > > > > > > > > >
