No worries. I filed https://issues.apache.org/jira/browse/ARROW-15921 for this.
On Fri, Mar 11, 2022, at 16:43, Jacques Nadeau wrote: > Agreed. Sorry I didn't do a better job in the original comments in the proto > file. > > On Fri, Mar 11, 2022, 1:42 PM David Li <[email protected]> wrote: >> __ >> Ah, thanks for the clarification Jacques - we should perhaps clarify that in >> the spec itself. >> >> On Fri, Mar 11, 2022, at 16:37, Jacques Nadeau wrote: >>> The intention was that a flight consumer could use any of the location >>> values and they produced equivalent data. Similar to the concept of block >>> location in the hdfs api. This could allow a single flight dataset to be >>> served from multiple locations (and/or protocols) and temhe consumer can >>> choose the optimal path of access (to maximize locality, etc). >>> >>> It was never meant to be application specified. >>> >>> On Fri, Mar 11, 2022, 1:24 PM Kyle Porter <[email protected]> wrote: >>>> I would second that - let's just make it part of implementing Flight SQL >>>> in terms of how this should be interpreted. I don't have a specific >>>> opinion on what the best interpretation would be, however. >>>> >>>> *Kyle Porter* >>>> CEO >>>> Bit Quill Technologies Inc. >>>> Office: +1.778.331.3355 | Direct: +1.604.441.7318 | [email protected] >>>> https://www.bitquill.com >>>> >>>> This email message is for the sole use of the intended recipient(s) and >>>> may contain confidential and privileged information. Any unauthorized >>>> review, use, disclosure, or distribution is prohibited. If you are not >>>> the intended recipient, please contact the sender by reply email and >>>> destroy all copies of the original message. Thank you. >>>> >>>> >>>> On Fri, Mar 11, 2022 at 1:21 PM David Li <[email protected]> wrote: >>>>> __ >>>>> Are existing services really a concern? Implementing Flight SQL means >>>>> you're in control already, and I'd rather see a single clear >>>>> specification over a lot of toggles/flags. >>>>> >>>>> On Fri, Mar 11, 2022, at 16:08, James Duong wrote: >>>>>> I propose for Flight SQL, we make this an always existing GetSqlInfo >>>>>> property, so clients can understand the server's intent. (As opposed to >>>>>> mandating behavior one way or the other, which could make Flight SQL >>>>>> harder to adapt to existing Flight services if we pick the wrong way). >>>>>> >>>>>> With regular Flight, the application will already be tuned to the >>>>>> server's behavior so there's no need for metadata around this there. >>>>>> >>>>>> On Fri, Mar 11, 2022 at 1:02 PM Kyle Porter <[email protected]> >>>>>> wrote: >>>>>>> It seems like we should probably specify the behaviour for Flight SQL, >>>>>>> at the very least, to ensure that the behaviour is consistent, which is >>>>>>> one of the goals of Flight SQL. >>>>>>> >>>>>>> *Kyle Porter* >>>>>>> CEO >>>>>>> Bit Quill Technologies Inc. >>>>>>> Office: +1.778.331.3355 | Direct: +1.604.441.7318 | >>>>>>> [email protected] >>>>>>> https://www.bitquill.com >>>>>>> >>>>>>> This email message is for the sole use of the intended recipient(s) and >>>>>>> may contain confidential and privileged information. Any unauthorized >>>>>>> review, use, disclosure, or distribution is prohibited. If you are not >>>>>>> the intended recipient, please contact the sender by reply email and >>>>>>> destroy all copies of the original message. Thank you. >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 11, 2022 at 1:01 PM David Li <[email protected]> wrote: >>>>>>>> __ >>>>>>>> The perhaps unsatisfactory answer is that it is up to the application >>>>>>>> how to interpret this. For instance, it could be used to indicate >>>>>>>> multiple servers that can all handle the request, allowing the client >>>>>>>> to pick one at its choosing (say, for load balancing, or because the >>>>>>>> client only supports certain protocols). >>>>>>>> >>>>>>>> On Fri, Mar 11, 2022, at 15:58, James Duong wrote: >>>>>>>>> This seems inconsistent. >>>>>>>>> >>>>>>>>> In this c++ performance test it looks like it just gets the first >>>>>>>>> location in every endpoint (though it gets data from every endpoint >>>>>>>>> in parallel): >>>>>>>>> https://github.com/apache/arrow/blob/dc2e0b2e44fdaa3d5ad0bb358ff8ce9db3bc7416/cpp/src/arrow/flight/flight_benchmark.cc#L297 >>>>>>>>> >>>>>>>>> The protobuf definition is a bit unclear too. It says the client can >>>>>>>>> redeem the ticket at every location, but doesn't say the client >>>>>>>>> must retrieve data at every location to get the whole dataset (so >>>>>>>>> it's not clear if this is for redundancy or partitioning): >>>>>>>>> https://github.com/apache/arrow/blob/dc2e0b2e44fdaa3d5ad0bb358ff8ce9db3bc7416/format/Flight.proto#L283 >>>>>>>>> >>>>>>>>> The Java and C++ examples posted also do not handle the case where >>>>>>>>> the location is empty. >>>>>>>>> >>>>>>>>> On Fri, Mar 11, 2022 at 12:53 PM Rafael Telles <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> Hi guys, >>>>>>>>>> >>>>>>>>>> What is the user supposed to do with the Location list in a >>>>>>>>>> FlightEndpoint? >>>>>>>>>> Looking at this integration test >>>>>>>>>> (https://github.com/apache/arrow/blob/dc2e0b2e44fdaa3d5ad0bb358ff8ce9db3bc7416/java/flight/flight-integration-tests/src/main/java/org/apache/arrow/flight/integration/tests/IntegrationTestClient.java#L151) >>>>>>>>>> it's iterating on Location list and issuing multiple getStream() >>>>>>>>>> calls with the same ticket. >>>>>>>>>> >>>>>>>>>> Best Regards, >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *James Duong* >>>>>>>>> Lead Software Developer >>>>>>>>> Bit Quill Technologies Inc. >>>>>>>>> Direct: +1.604.562.6082 | [email protected] >>>>>>>>> https://www.bitquilltech.com >>>>>>>>> >>>>>>>>> >>>>>>>>> This email message is for the sole use of the intended recipient(s) >>>>>>>>> and may contain confidential and privileged information. Any >>>>>>>>> unauthorized review, use, disclosure, or distribution is prohibited. >>>>>>>>> If you are not the intended recipient, please contact the sender by >>>>>>>>> reply email and destroy all copies of the original message. Thank >>>>>>>>> you. >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *James Duong* >>>>>> Lead Software Developer >>>>>> Bit Quill Technologies Inc. >>>>>> Direct: +1.604.562.6082 | [email protected] >>>>>> https://www.bitquilltech.com >>>>>> >>>>>> >>>>>> This email message is for the sole use of the intended recipient(s) and >>>>>> may contain confidential and privileged information. Any unauthorized >>>>>> review, use, disclosure, or distribution is prohibited. If you are not >>>>>> the intended recipient, please contact the sender by reply email and >>>>>> destroy all copies of the original message. Thank you. >>>>> >>
