I'm supportive of having a bidirectional API for the reasons stated in
the document.
There seem to be some details to work out but probably nothing
insurmountable. It seems that how errors and metadata are handled are
one of the open questions
On Wed, Oct 16, 2019 at 7:13 PM David Li <li.david...@gmail.com> wrote:
>
> I was definitely considering having control messages without data, and
> I thought that could be encoded by a FlightData with only app_metadata
> set. I think I understand your position now: FlightData should always
> carry (some) data (with optional metadata)?
>
> That makes sense to me, and is consistent with the documentation on
> FlightData in the Protobuf file. I was worried about having a
> redundant metadata field, but oneof prevents that from happening, and
> overall having a clear separation between data and control messages is
> cleaner.
>
> As for using Protobuf's Any: so far, we've refrained from exposing
> Protobuf by using bytes, would we want to change that now?
>
> Best,
> David
>
> On 10/16/19, Jacques Nadeau <jacq...@apache.org> wrote:
> > Hey David,
> >
> > RE: Async: I was trying to match the pattern we use for doget/doput for
> > async. Yes, more thinking java given java grpc's async always pattern.
> >
> > On the comment around the FlightData, I think it is overloading the message
> > to use metadata for this. If I want to send a control message independently
> > of the data message, I would have to define something like an empty flight
> > data message that has custom metadata. Why not support a container object
> > with a oneof{FlightData, Any} in it instead so users can add more data as
> > desired. The default impl could be a noop for the Any messages.
> >
> > On Tue, Oct 15, 2019 at 6:50 PM David Li <li.david...@gmail.com> wrote:
> >
> >> Hi Jacques,
> >>
> >> Thanks for the comments.
> >>
> >> - I do agree DoExchange is a better name!
> >> - FlightData already has metadata fields as a result of prior
> >> proposals, so I don't think we need a new message to carry that kind
> >> of information.
> >> - I like the suggestion of an async handler to handle incoming
> >> messages as the fundamental API; it would actually be quite natural to
> >> implement in Flight/Java. I will note that it's not possible in
> >> C++/Python without spawning a thread, though. (In essence, gRPC-Java
> >> is async-always and gRPC-C++ is sync-always.) There are experimental
> >> C++ APIs that would let us do something similar to Java, but those are
> >> only in relatively recent gRPC versions and are still under
> >> development (contrary to the interceptor APIs which have been around
> >> for quite a while).
> >>
> >> Thanks,
> >> David
> >>
> >> On 10/15/19, Jacques Nadeau <jacq...@apache.org> wrote:
> >> > I like it. Added some comments to the doc. Might worth discussion here
> >> > depending on your thoughts.
> >> >
> >> > On Tue, Oct 15, 2019 at 7:11 AM David Li <li.david...@gmail.com> wrote:
> >> >
> >> >> Hey Ryan,
> >> >>
> >> >> Thanks for the comments.
> >> >>
> >> >> Concrete example: I've edited the doc to provide a Python strawman.
> >> >>
> >> >> Sync vs async: while I don't touch on it, you could interleave uploads
> >> >> and downloads if you were so inclined. Right now, synchronous APIs
> >> >> make this error-prone, e.g. if both client and server wait for each
> >> >> other due to an application logic bug. (gRPC doesn't give us the
> >> >> ability to have per-read timeouts, only an overall timeout.) As an
> >> >> example of this happening with DoPut, see ARROW-6063:
> >> >> https://issues.apache.org/jira/browse/ARROW-6063
> >> >>
> >> >> This is mostly tangential though, eventually we will want to design
> >> >> asynchronous APIs for Flight as a whole. A bidirectional stream like
> >> >> this (and like DoPut) just makes these pitfalls easier to run into.
> >> >>
> >> >> Using DoPut+DoGet: I discussed this in the proposal, but the main
> >> >> concern is that depending on how you deploy, two separate calls could
> >> >> get routed to different instances. Additionally, gRPC has some
> >> >> reconnection behaviors; if the server goes away in between the two
> >> >> calls, but it then restarts or there is another instance available,
> >> >> the client will happily reconnect to the new server without warning.
> >> >>
> >> >> Thanks,
> >> >> David
> >> >>
> >> >> On 10/15/19, Ryan Murray <rym...@dremio.com> wrote:
> >> >> > Hey David,
> >> >> >
> >> >> > I think this proposal makes a lot of sense. I like it and the
> >> >> > possibility
> >> >> > of remote compute via arrow buffers. One thing that would help me
> >> would
> >> >> be
> >> >> > a concrete example of the API in a real life use case. Also, what
> >> would
> >> >> the
> >> >> > client experience be in terms of sync vs asyc? Would the client
> >> >> > block
> >> >> till
> >> >> > the bidirectional call return ie c = flight.vector_mult(a, b) or
> >> >> > would
> >> >> the
> >> >> > client wait to be signaled that computation was done. If the later
> >> >> > how
> >> >> > is
> >> >> > that different from a DoPut then DoGet? I suppose that this could be
> >> >> > implemented without extending the RPC interface but rather by a
> >> >> > function/util?
> >> >> >
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > Ryan
> >> >> >
> >> >> > On Sun, Oct 13, 2019 at 9:24 PM David Li <li.david...@gmail.com>
> >> wrote:
> >> >> >
> >> >> >> Hi all,
> >> >> >>
> >> >> >> We've been using Flight quite successfully so far, but we have
> >> >> >> identified a new use case on the horizon: being able to both send
> >> >> >> and
> >> >> >> retrieve Arrow data within a single RPC call. To that end, I've
> >> >> >> written up a proposal for a new RPC method:
> >> >> >>
> >> >> >>
> >> >>
> >> https://docs.google.com/document/d/1Hh-3Z0hK5PxyEYFxwVxp77jens3yAgC_cpp0TGW-dcw/edit?usp=sharing
> >> >> >>
> >> >> >> Please let me know if you can't view or comment on the document.
> >> >> >> I'd
> >> >> >> appreciate any feedback; I think this is a relatively
> >> >> >> straightforward
> >> >> >> addition - it is essentially "DoPutThenGet".
> >> >> >>
> >> >> >> This is a format change and would require a vote. I've decided to
> >> >> >> table the other format change I had proposed (on DoPut), as it
> >> doesn't
> >> >> >> functionally change Flight, just the interpretation of the
> >> >> >> semantics.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> David
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Ryan Murray | Principal Consulting Engineer
> >> >> >
> >> >> > +447540852009 | rym...@dremio.com
> >> >> >
> >> >> > <https://www.dremio.com/>
> >> >> > Check out our GitHub <https://www.github.com/dremio>, join our
> >> >> > community
> >> >> > site <https://community.dremio.com/> & Download Dremio
> >> >> > <https://www.dremio.com/download>
> >> >> >
> >> >>
> >> >
> >>
> >
