What do people think about using the C interface representation?

On Sun, Dec 29, 2019 at 12:42 PM Micah Kornfield <emkornfi...@gmail.com>
wrote:

> I opened https://github.com/google/flatbuffers/issues/5688 to try to get
> some clarity.
>
> On Tue, Dec 24, 2019 at 12:13 PM Wes McKinney <wesmck...@gmail.com> wrote:
>
> > On Tue, Dec 24, 2019 at 2:47 AM Micah Kornfield <emkornfi...@gmail.com>
> > wrote:
> > >>
> > >> If we were to make the same kinds of forward/backward compatibility
> > >> guarantees as with Flatbuffers it could create a lot of work for
> > >> maintainers.
> > >
> > > Does it pay to follow-up with the flatbuffer project to understand if
> > the forward/backward compatibility guarantees the flatbuffers provide
> > extend to their JSON format?
> >
> > I spent a few minutes looking at the Flatbuffers codebase and
> > documentation and did not find anything, so this seems like useful
> > information to have regardless.
> >
> > >
> > > On Sun, Dec 15, 2019 at 11:17 AM Wes McKinney <wesmck...@gmail.com>
> > wrote:
> > >>
> > >> I'd be open to looking at a proposal for a human-readable text
> > >> representation, but I'm definitely wary about making any kind of
> > >> cross-version compatibility guarantees (beyond "we will do our best").
> > >> If we were to make the same kinds of forward/backward compatibility
> > >> guarantees as with Flatbuffers it could create a lot of work for
> > >> maintainers.
> > >>
> > >> On Thu, Dec 12, 2019 at 12:43 AM Micah Kornfield <
> emkornfi...@gmail.com>
> > wrote:
> > >> >
> > >> > >
> > >> > > With these two together, it would seem not too difficult to create
> > a text
> > >> > > representation for Arrow schemas that (at some point) has some
> > >> > > compatibility guarantees, but maybe I'm missing something?
> > >> >
> > >> >
> > >> > I think the main risk is if somehow flatbuffers JSON parsing doesn't
> > handle
> > >> > backward compatible changes to the arrow schema message.  Given the
> > way the
> > >> > documentation is describing the JSON functionality I think this
> would
> > be
> > >> > considered a bug.
> > >> >
> > >> > The one downside to calling the "schema" canonical is the
> flatbuffers
> > JSON
> > >> > functionality only appears to be available in C++ and Java via JNI,
> > so it
> > >> > wouldn't have cross language support.  I think this issue is more
> one
> > of
> > >> > semantics though (i.e. does the JSON description become part of the
> > "Arrow
> > >> > spec" or does it live as a C++/Python only feature).
> > >> >
> > >> > -Micah
> > >> >
> > >> >
> > >> > On Tue, Dec 10, 2019 at 10:51 AM Christian Hudon <
> > chr...@elementai.com>
> > >> > wrote:
> > >> >
> > >> > > Micah: I didn't know that Flatbuffers supported serialization
> > to/from JSON,
> > >> > > thanks. That seems like a very good start, at least. I'll aim to
> > create a
> > >> > > draft pull request that at least wires everything up in Arrow so
> we
> > can
> > >> > > load/save a Schema.fbs instance from/to JSON. At least it'll make
> > it easier
> > >> > > for me to see how Arrow schemas would look in JSON with that.
> > >> > >
> > >> > > Otherwise, I'm still gathering requirements internally here. For
> > example,
> > >> > > one thing that would be nice would be to be able to output a JSON
> > Schema
> > >> > > from at least a subset of the Arrow schema. (That way our users
> > could start
> > >> > > by passing around JSON with a given schema, and transition pieces
> > of a
> > >> > > workflow to Arrow as they're ready.) But that part can also be
> done
> > outside
> > >> > > of the Arrow code, if deemed not relevant to have in the Arrow
> > codebase
> > >> > > itself.
> > >> > >
> > >> > > One core requirement for us, however, would be eventual
> > compatibility
> > >> > > between Arrow versions for a given text representation of a
> schema.
> > >> > > Meaning, if you have a text description of a given Arrow schema,
> > you can
> > >> > > load it into different versions of Arrow and it creates a valid
> > Schema
> > >> > > Flatbuffer description, that Arrow can use. Wes, were you thinking
> > of that,
> > >> > > or of something else, when you wrote "only makes sense if it is
> > offered
> > >> > > without any backward/forward compatibility guarantees"?
> > >> > >
> > >> > > For the now, or me, assuming the JSON serialization done by the
> > Flatbuffer
> > >> > > libraries is usable, it seems we have all the pieces to make this
> > happen:
> > >> > > 1) The binary Schema.fbs data structures has to be compatible
> > between
> > >> > > different versions of Arrow, otherwise two processes with
> different
> > Arrow
> > >> > > versions won't be able to interoperate, no?
> > >> > > 2) The Flatbuffer <-> JSON serialization supplied by the
> Flatbuffers
> > >> > > library also has to be compatible between different versions of
> the
> > >> > > Flatbuffers library, since the main use case seems to be storing
> > >> > > Flatbuffers assets into version control. Breaking changes there
> > will also
> > >> > > be painful to their users.
> > >> > >
> > >> > > With these two together, it would seem not too difficult to create
> > a text
> > >> > > representation for Arrow schemas that (at some point) has some
> > >> > > compatibility guarantees, but maybe I'm missing something?
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > >   Christian
> > >> > >
> > >> > > Le lun. 9 déc. 2019, à 07 h 00, Wes McKinney <wesmck...@gmail.com
> >
> > a
> > >> > > écrit :
> > >> > >
> > >> > > > The only "canonical" representation of schemas at the moment is
> > the
> > >> > > > Flatbuffers data structure [1]
> > >> > > >
> > >> > > > Having a human-readable/parseable text representation I think
> only
> > >> > > > makes sense if it is offered without any backward/forward
> > >> > > > compatibility guarantees.
> > >> > > >
> > >> > > > Note I had previously opened
> > >> > > > https://issues.apache.org/jira/browse/ARROW-3730 where I noted
> > that
> > >> > > > there's no way (aside from generating the Flatbuffers messages)
> to
> > >> > > > generate a schema representation that can be used later to
> > reconstruct
> > >> > > > a schema in a program. If such a representation were human
> > >> > > > readable/editable that seems beneficial.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > [1]:
> > https://github.com/apache/arrow/blob/master/format/Schema.fbs
> > >> > > >
> > >> > > > On Sat, Dec 7, 2019 at 11:56 AM Maarten Ballintijn <
> > maart...@xs4all.nl>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > >
> > >> > > > > Is there a syntax specified for schemas?
> > >> > > > >
> > >> > > > > Cheers,
> > >> > > > > Maarten.
> > >> > > > >
> > >> > > > >
> > >> > > > > > On Dec 6, 2019, at 5:01 PM, Micah Kornfield <
> > emkornfi...@gmail.com>
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > > Hi Christian,
> > >> > > > > > As far as I know no-one is working on a canonical text
> > representation
> > >> > > > for
> > >> > > > > > schemas.  A JSON serializer exists for integration test
> > purposes, but
> > >> > > > > > IMO it shouldn't be relied upon as canonical.
> > >> > > > > >
> > >> > > > > > It looks like Flatbuffers supports serialization to/from
> JSON
> > [1
> > >> > > > > > <
> > https://google.github.io/flatbuffers/flatbuffers_guide_use_cpp.html
> > >> > > > >],
> > >> > > > > > using that functionality might be a promising avenue to
> > pursue for a
> > >> > > > human
> > >> > > > > > readable schema. I could see adding a helper method
> someplace
> > under
> > >> > > > IPC for
> > >> > > > > > this.  Would that meet your needs?  I think if there are
> other
> > >> > > > > > requirements, then a proposal would be welcome.  Ideally, a
> > solution
> > >> > > > would
> > >> > > > > > not require additional build/runtime dependencies.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Micah
> > >> > > > > >
> > >> > > > > > [1] See Text & schema parsing
> > >> > > > > >
> > https://google.github.io/flatbuffers/flatbuffers_guide_use_cpp.html
> > >> > > > > >
> > >> > > > > > On Fri, Dec 6, 2019 at 1:26 PM Christian Hudon <
> > chr...@elementai.com
> > >> > > >
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > >> Hi,
> > >> > > > > >>
> > >> > > > > >> For the uses I would like to make of Arrow, I would need a
> > >> > > > human-readable
> > >> > > > > >> and -writable version of an Arrow Schema, that could be
> > converted to
> > >> > > > and
> > >> > > > > >> from the Arrow Schema C++ object. Going through the doc for
> > 0.15.1,
> > >> > > I
> > >> > > > don't
> > >> > > > > >> see anything to that effect, with the closest being the
> > ToString()
> > >> > > > method
> > >> > > > > >> on DataType instances, but which is meant for debugging
> > only. (I
> > >> > > need
> > >> > > > an
> > >> > > > > >> expression of an Arrow Schema that people can read, and
> that
> > can
> > >> > > live
> > >> > > > > >> outside of the code for a particular operation.)
> > >> > > > > >>
> > >> > > > > >> Is a text representation of an Arrow Schema something that
> > is being
> > >> > > > worked
> > >> > > > > >> on now? If not, would you folks be interested in me putting
> > up an
> > >> > > > initial
> > >> > > > > >> proposal for discussion? Any design constraints I should
> pay
> > >> > > > attention to,
> > >> > > > > >> then?
> > >> > > > > >>
> > >> > > > > >> Thanks,
> > >> > > > > >>
> > >> > > > > >>  Christian
> > >> > > > > >> --
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >> │ Christian Hudon
> > >> > > > > >>
> > >> > > > > >> │ Applied Research Scientist
> > >> > > > > >>
> > >> > > > > >>   Element AI, 6650 Saint-Urbain #500
> > >> > > > > >>
> > >> > > > > >>   Montréal, QC, H2S 3G9, Canada
> > >> > > > > >>   Elementai.com
> > >> > > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > >
> > >> > >
> > >> > > │ Christian Hudon
> > >> > >
> > >> > > │ Applied Research Scientist
> > >> > >
> > >> > >    Element AI, 6650 Saint-Urbain #500
> > >> > >
> > >> > >    Montréal, QC, H2S 3G9, Canada
> > >> > >    Elementai.com
> > >> > >
> >
>

Reply via email to