I've filed/found the following: ARROW-4575 [1] for adding Python to the Flight integration tests ARROW-5875 [2] for adding RPC features (like auth) to integration tests ARROW-1009 [3] for async APIs ARROW-5876 [4] for implementing basic auth across all languages ARROW-5877 [5] for documenting the caveats around auth
[1]: https://issues.apache.org/jira/browse/ARROW-4575 [2]: https://issues.apache.org/jira/browse/ARROW-5875 [3]: https://issues.apache.org/jira/browse/ARROW-1009 [4]: https://issues.apache.org/jira/browse/ARROW-5876 [5]: https://issues.apache.org/jira/browse/ARROW-5877 Best, David On 7/6/19, Wes McKinney <wesmck...@gmail.com> wrote: > Are there some action items (JIRA issues) to follow up here? At > minimum having documentation about this for the Python client side > would seem to be in order. > > On Thu, Jul 4, 2019 at 2:20 PM Ryan Murray <rym...@dremio.com> wrote: >> >> Hey David, >> >> I was actually testing test_flight.test_http_basic_auth(). But I think >> the >> same applies. The Java default implementation expects a handshake. More >> to >> the point it expects a BasicAuth protobuf which I believe is not exposed >> at >> all in python. Always returning true in >> BasicServerAuthHandler.authenticate() allows for the test implementations >> of Java and Python authentication to speak to each other. >> >> Thanks for the below link, that really clarified things for me. I would >> add >> to the list that we should normalise the use of BasicAuth protobuf >> message >> between java and cpp. >> >> Apologies for the urgency yesterday, I am glad it was sorted and more my >> fault than Arrow's. >> >> Best, >> Ryan >> >> >> On Thu, Jul 4, 2019 at 11:29 AM David Li <li.david...@gmail.com> wrote: >> >> > Hmm, interesting. I assume you mean test_flight.test_token_auth() as >> > the client? The tests weren't written to be explicitly compatible, but >> > there's no reason you should get an indefinite stall. >> > >> > We don't use Handshake/ServerAuthHandler#authenticate, so that would >> > explain why we don't see issues. I commented on this in the initial >> > implementation: >> > https://github.com/apache/arrow/pull/4125#discussion_r273935691 >> > >> > > there is not a 1-1 mapping between connected clients and connected >> > peers, and so you can >> > > only know the identity of the peer at the moment it makes a call. >> > > Just >> > doing a handshake >> > > (Authenticate) isn't enough to identify who is on the other end of a >> > particular connection. >> > >> > the reason being that a layer 7 load balancer (e.g. Envoy) means one >> > gRPC connection can represent multiple clients. Conversely, >> > client-side load balancing (built into gRPC) means one client-side >> > "connection" can actually represent multiple servers. Of course, you >> > have to consciously deploy in this manner, so Handshake is still >> > useful if you know you won't ever need these features. >> > >> > As I see it, this means there's a few things to work on: >> > - Flight RPC feature compatibility needs to be tested, not just format >> > compatibility. >> > - We should start thinking about async APIs and/or timeouts in any >> > sort of API that makes a network call (there's already a JIRA: >> > https://issues.apache.org/jira/browse/ARROW-1009), since "never >> > returns" is a terrible failure mode >> > >> > Best, >> > David >> > >> > On 7/4/19, Ryan Murray <rym...@dremio.com> wrote: >> > > Hey David, >> > > >> > > I am curious to see what you are doing different from me. I am >> > > running >> > the >> > > Java ExampleFlightServer.java against the python auth flight tests >> > > and >> > they >> > > are not passing. The particular issue is that incoming.next() never >> > returns >> > > in BasicServerAuthHandler.java:56 >> > > >> > > It doesn't appear to be anything wrong w/ the auth piece specifically >> > > rather the server appears to not be getting the auth info to verify. I >> > > am >> > > still investigating my issue but I am glad that someone else has >> > > gotten >> > > this to work. >> > > >> > > Best, >> > > Ryan >> > > >> > > On Thu, Jul 4, 2019 at 9:13 AM Antoine Pitrou <anto...@python.org> >> > wrote: >> > > >> > >> >> > >> It may be worth opening a JIRA for the flaky tests if not already >> > >> done. >> > >> >> > >> Regards >> > >> >> > >> Antoine. >> > >> >> > >> >> > >> Le 04/07/2019 à 18:11, David Li a écrit : >> > >> > I'm also curious as to what the issue was, as we've been doing >> > >> > Python-client-Java-server auth with development builds without >> > >> > trouble. >> > >> > >> > >> > Regardless - this does point out a need for more cross-language >> > >> > Flight >> > >> > testing (perhaps a Flight-specific integration suite?), and to get >> > >> > existing tests running more consistently in CI (Flight/Java in >> > >> > particular has a lot of flaky tests, though the auth tests are >> > >> > enabled >> > >> > in Travis). >> > >> > >> > >> > Best, >> > >> > David >> > >> > >> > >> > On 7/4/19, Jacques Nadeau <jacq...@apache.org> wrote: >> > >> >> Which is exactly why I was withholding a vote until there was >> > >> >> more >> > >> >> information. >> > >> >> >> > >> >> On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou <solip...@pitrou.net> >> > >> wrote: >> > >> >> >> > >> >>> On Thu, 4 Jul 2019 09:04:34 -0500 >> > >> >>> Wes McKinney <wesmck...@gmail.com> wrote: >> > >> >>>> >> > >> >>>> That being said, with Ryan's issue, he is using a feature >> > >> >>>> (cross-language auth in Flight) that isn't being tested. The >> > >> >>>> Flight >> > >> >>>> integration tests do not use authentication AFAIK so I'm not >> > >> >>>> surprised >> > >> >>>> to hear that there may be an issue with it. >> > >> >>> >> > >> >>> OTOH, it's a bit unlikely. Flight authentication is implemented >> > >> >>> is: >> > >> >>> - the Arrow codebase simply passes opaque tokens around >> > >> >>> - interpretation of tokens is handled by application code >> > >> >>> - marshalling of tokens is handled by Protocol Buffers >> > >> >>> >> > >> >>> So unless something silly is going on (such as "passing an empty >> > >> >>> string >> > >> >>> instead of the actual token") there's not much potential for >> > >> >>> auth interoperability issues in the core Flight codebase. >> > >> >>> >> > >> >>> Regards >> > >> >>> >> > >> >>> Antoine. >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >> >> > >> >> > > >> > > >> > > -- >> > > >> > > Ryan Murray | Principal Consulting Engineer >> > > >> > > +447540852009 | rym...@dremio.com >> > > >> > > <https://www.dremio.com/> >> > > Check out our GitHub <https://www.github.com/dremio>, join our >> > > community >> > > site <https://community.dremio.com/> & Download Dremio >> > > <https://www.dremio.com/download> >> > > >> > >> >> >> -- >> >> Ryan Murray | Principal Consulting Engineer >> >> +447540852009 | rym...@dremio.com >> >> <https://www.dremio.com/> >> Check out our GitHub <https://www.github.com/dremio>, join our community >> site <https://community.dremio.com/> & Download Dremio >> <https://www.dremio.com/download> >> I >
