Le 12/08/2021 à 15:05, Wes McKinney a écrit :
It seems that one adjacent problem here is how to make it simpler for
third parties (especially ones that act as front end interfaces) to
build and serialize/deserialize the IR structures with some kind of
ready-to-go middleware library, written in a language like C++.
A C++ library sounds a bit complicated to deal with for Java, Rust, Go,
etc. developers.
I'm not sure which design decision and set of compromises would make the
most sense. But this is why I'm asking the question "why not JSON?" (+
JSON-Schema if you want to ease validation by third parties).
(note I have already mentioned MsgPack, but only in the case a binary
encoding is really required; it doesn't have any other advantage that I
know of over JSON, and it's less ubiquitous)
Regards
Antoine.
To do that, one would need the equivalent of arrow/type.h and related
Flatbuffers schema serialization code that lives in arrow/ipc. If you
want to be able to completely and accurately serialize Schemas, you
need quite a bit of code now.
One possible approach (and not go crazy) would be to:
* Move arrow/types.h and its dependencies into a standalone C++
library that can be vendored into the main apache/arrow C++ library. I
don't know how onerous arrow/types.h's transitive dependencies /
interactions are at this point (there's a lot of stuff going on in
type.cc [1] now)
* Make the namespaces exported by this library configurable, so any
library can vendor the Arrow types / IR builder APIs privately into
their project
* Maintain this "Arrow types and ComputeIR library" as an always
zero-dependency library to facilitate vendoring
* Lightweight bindings in languages we care about (like Python or R or
GLib/Ruby) could be built to the IR builder middleware library
This seems like what is more at issue compared with rather projects
are copying the Flatbuffers files out of their project from
apache/arrow or apache/arrow-format.
[1]: https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.cc
On Thu, Aug 12, 2021 at 2:05 PM Andrew Lamb <al...@influxdata.com> wrote:
I support the idea of an independent repo that has the arrow flatbuffers
format definition files.
My rationale is that the Rust implementation has a copy of the `format`
directory [1] and potential drift worries me (a bit). Having a single
source of truth for the format that is not part of the large mono repo
would be a good thing.
Andrew
[1] https://github.com/apache/arrow-rs/tree/master/format
On Wed, Aug 11, 2021 at 2:40 PM Phillip Cloud <cpcl...@gmail.com> wrote:
Hi all,
I'd like to bring up an idea from a recent thread ([1]) about moving the
`format/` directory out of the primary apache/arrow repository.
I understand from that thread there are some concerns about using
submodules,
and I definitely sympathize with those concerns.
In talking with David Li (disclaimer: we work together at Voltron Data), he
has
a great idea that I think makes everyone happy: an `apache/arrow-format`
repository that is the official mirror for the flatbuffers IDL, that
library
authors should use as the source of truth.
It doesn't require a submodule, yet it also allows external projects the
ability to access the IDL without having to interact with the main arrow
repository and is backwards compatible to boot.
In this scenario, repositories that are currently copying in the
flatbuffers
IDL can migrate to this repository at their leisure.
My motivation for this was around sharing data structures for the compute
IR
proposal ([2]).
I can think of at least two ways for IR producers and consumers of all
languages to share the flatbuffers IDL:
1. A set of bindings built in some language that other languages can
integrate
with, likely C++, that allows library users to build IR using an API.
The primary downside to this is that we'd have to deal with
building another library while working out any kinks in the IR design and
I'd
rather avoid that in the initial phases of this project.
The benefit is that IR components don't interact much with `flatbuffers` or
`flatc` directly.
2. A single location where the format lives, that doesn't require depending
on
a large multi-language repository to access a handful of files.
I think the downside to this is that there's a bit of additional
infrastructure
to automate copying in `arrow-format`.
The benefit there is that producers and consumers can immediately start
getting
value from compute IR without having to wait for development of a new API.
One counter-proposal might be to just put the compute IR IDL in a separate
repo,
but that isn't tenable because the compute IR needs arrow's type
information
contained in `Schema.fbs`.
I was hoping to avoid conflating the discussion about bindings vs direct
flatbuffer usage (at least initially just supporting one, I predict we'll
need
both ultimately) with the decision about whether to split out the format
directory, but it's a good example of a choice for which splitting out the
format directory would be well-served.
I'll note that this doesn't block anything on the compute IR side, just
wanted
to surface this in a parallel thread and see what folks think.
[1]:
https://lists.apache.org/thread.html/rcebfcb4c5d0b7752fcdda6587871c2f94661b8c4e35119f0bcfb883b%40%3Cdev.arrow.apache.org%3E
[2]:
https://docs.google.com/document/d/1C_XVOG7iFkl6cgWWMyzUoIjfKt-X2UxqagPJrla0bAE/edit#heading=h.ie0ne0gm762l
