Can one of you explain what is being proposed in non-protobuf terms? Knowledge of protobuf shouldn't be required to use Flight.
Regards Antoine. Le 21/10/2019 à 15:46, David Li a écrit : > Oneof doesn't actually change the wire encoding; it would just be > application-level logic. (The official guide doesn't even mention it > in the encoding docs; I found > https://stackoverflow.com/questions/52226409/how-protobuf-encodes-oneof-message-construct > as well.) > > If I follow you, Jacques, then you are proposing essentially inlining > the definition of Any, e.g. > > message FlightMessage { > oneof message { > FlightData data = 1; > FlightAny metadata = 2; > } > } > > message FlightAny { > string type = 1; > bytes data = 2; > } > > Is this correct? > > It might be nice to consider the wrapper message for DoGet/DoPut as > well, but at that point, I'd rather we be consistent with all of them, > rather than have one of the three methods do its own thing. > > Thanks, > David > > On 10/20/19, Jacques Nadeau <jacq...@apache.org> wrote: >> I think we could probably expose the oneof behavior without exposing the >> protobuf functions. On the any... hmm. I guess we could expose as two >> fields: type and data. Then users could use it for whatever but if people >> wanted to treat it as any, it would work. (Basically a user could use any >> with it easily but they could also use any other mechanism). At least in >> java, the any concepts are pretty simple/diy. Are other language bindings >> less diy? >> >> I'm *not* hardcore against the empty FlightData + metadata but it just >> seemed a bit janky. >> >> Thinking about the control message/wrapper object thing, I wonder if we >> should redefine DoPut and DoGet to have the same property if we think it is >> a good idea... >> >> On Wed, Oct 16, 2019 at 5:13 PM David Li <li.david...@gmail.com> wrote: >> >>> I was definitely considering having control messages without data, and >>> I thought that could be encoded by a FlightData with only app_metadata >>> set. I think I understand your position now: FlightData should always >>> carry (some) data (with optional metadata)? >>> >>> That makes sense to me, and is consistent with the documentation on >>> FlightData in the Protobuf file. I was worried about having a >>> redundant metadata field, but oneof prevents that from happening, and >>> overall having a clear separation between data and control messages is >>> cleaner. >>> >>> As for using Protobuf's Any: so far, we've refrained from exposing >>> Protobuf by using bytes, would we want to change that now? >>> >>> Best, >>> David >>> >>> On 10/16/19, Jacques Nadeau <jacq...@apache.org> wrote: >>>> Hey David, >>>> >>>> RE: Async: I was trying to match the pattern we use for doget/doput for >>>> async. Yes, more thinking java given java grpc's async always pattern. >>>> >>>> On the comment around the FlightData, I think it is overloading the >>> message >>>> to use metadata for this. If I want to send a control message >>> independently >>>> of the data message, I would have to define something like an empty >>> flight >>>> data message that has custom metadata. Why not support a container >>>> object >>>> with a oneof{FlightData, Any} in it instead so users can add more data >>>> as >>>> desired. The default impl could be a noop for the Any messages. >>>> >>>> On Tue, Oct 15, 2019 at 6:50 PM David Li <li.david...@gmail.com> wrote: >>>> >>>>> Hi Jacques, >>>>> >>>>> Thanks for the comments. >>>>> >>>>> - I do agree DoExchange is a better name! >>>>> - FlightData already has metadata fields as a result of prior >>>>> proposals, so I don't think we need a new message to carry that kind >>>>> of information. >>>>> - I like the suggestion of an async handler to handle incoming >>>>> messages as the fundamental API; it would actually be quite natural to >>>>> implement in Flight/Java. I will note that it's not possible in >>>>> C++/Python without spawning a thread, though. (In essence, gRPC-Java >>>>> is async-always and gRPC-C++ is sync-always.) There are experimental >>>>> C++ APIs that would let us do something similar to Java, but those are >>>>> only in relatively recent gRPC versions and are still under >>>>> development (contrary to the interceptor APIs which have been around >>>>> for quite a while). >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> On 10/15/19, Jacques Nadeau <jacq...@apache.org> wrote: >>>>>> I like it. Added some comments to the doc. Might worth discussion >>>>>> here >>>>>> depending on your thoughts. >>>>>> >>>>>> On Tue, Oct 15, 2019 at 7:11 AM David Li <li.david...@gmail.com> >>> wrote: >>>>>> >>>>>>> Hey Ryan, >>>>>>> >>>>>>> Thanks for the comments. >>>>>>> >>>>>>> Concrete example: I've edited the doc to provide a Python strawman. >>>>>>> >>>>>>> Sync vs async: while I don't touch on it, you could interleave >>> uploads >>>>>>> and downloads if you were so inclined. Right now, synchronous APIs >>>>>>> make this error-prone, e.g. if both client and server wait for each >>>>>>> other due to an application logic bug. (gRPC doesn't give us the >>>>>>> ability to have per-read timeouts, only an overall timeout.) As an >>>>>>> example of this happening with DoPut, see ARROW-6063: >>>>>>> https://issues.apache.org/jira/browse/ARROW-6063 >>>>>>> >>>>>>> This is mostly tangential though, eventually we will want to design >>>>>>> asynchronous APIs for Flight as a whole. A bidirectional stream >>>>>>> like >>>>>>> this (and like DoPut) just makes these pitfalls easier to run into. >>>>>>> >>>>>>> Using DoPut+DoGet: I discussed this in the proposal, but the main >>>>>>> concern is that depending on how you deploy, two separate calls >>>>>>> could >>>>>>> get routed to different instances. Additionally, gRPC has some >>>>>>> reconnection behaviors; if the server goes away in between the two >>>>>>> calls, but it then restarts or there is another instance available, >>>>>>> the client will happily reconnect to the new server without >>>>>>> warning. >>>>>>> >>>>>>> Thanks, >>>>>>> David >>>>>>> >>>>>>> On 10/15/19, Ryan Murray <rym...@dremio.com> wrote: >>>>>>>> Hey David, >>>>>>>> >>>>>>>> I think this proposal makes a lot of sense. I like it and the >>>>>>>> possibility >>>>>>>> of remote compute via arrow buffers. One thing that would help me >>>>> would >>>>>>> be >>>>>>>> a concrete example of the API in a real life use case. Also, what >>>>> would >>>>>>> the >>>>>>>> client experience be in terms of sync vs asyc? Would the client >>>>>>>> block >>>>>>> till >>>>>>>> the bidirectional call return ie c = flight.vector_mult(a, b) or >>>>>>>> would >>>>>>> the >>>>>>>> client wait to be signaled that computation was done. If the >>>>>>>> later >>>>>>>> how >>>>>>>> is >>>>>>>> that different from a DoPut then DoGet? I suppose that this could >>> be >>>>>>>> implemented without extending the RPC interface but rather by a >>>>>>>> function/util? >>>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> Ryan >>>>>>>> >>>>>>>> On Sun, Oct 13, 2019 at 9:24 PM David Li <li.david...@gmail.com> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> We've been using Flight quite successfully so far, but we have >>>>>>>>> identified a new use case on the horizon: being able to both >>>>>>>>> send >>>>>>>>> and >>>>>>>>> retrieve Arrow data within a single RPC call. To that end, I've >>>>>>>>> written up a proposal for a new RPC method: >>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> https://docs.google.com/document/d/1Hh-3Z0hK5PxyEYFxwVxp77jens3yAgC_cpp0TGW-dcw/edit?usp=sharing >>>>>>>>> >>>>>>>>> Please let me know if you can't view or comment on the document. >>>>>>>>> I'd >>>>>>>>> appreciate any feedback; I think this is a relatively >>>>>>>>> straightforward >>>>>>>>> addition - it is essentially "DoPutThenGet". >>>>>>>>> >>>>>>>>> This is a format change and would require a vote. I've decided >>>>>>>>> to >>>>>>>>> table the other format change I had proposed (on DoPut), as it >>>>> doesn't >>>>>>>>> functionally change Flight, just the interpretation of the >>>>>>>>> semantics. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> David >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Ryan Murray | Principal Consulting Engineer >>>>>>>> >>>>>>>> +447540852009 | rym...@dremio.com >>>>>>>> >>>>>>>> <https://www.dremio.com/> >>>>>>>> Check out our GitHub <https://www.github.com/dremio>, join our >>>>>>>> community >>>>>>>> site <https://community.dremio.com/> & Download Dremio >>>>>>>> <https://www.dremio.com/download> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
