To close the loop on this discussion, I’ve filed the following issue with the gRPC folks:
https://github.com/grpc/grpc/issues/35638 Thank you again for all of your help. I would not have been able to understand what’s going on without it. > On Jan 23, 2024, at 11:43 AM, Brad House <b...@brad-house.com> wrote: > > Yeah, it does clearly show them enqueuing IPv4 and IPv6 requests separately. > So either they need to add logic similar to c-ares has internally with > https://github.com/c-ares/c-ares/pull/551 or just use ares_getaddrinfo() > instead of ares_gethostbyname() with address family AF_UNSPEC and let c-ares > do the right thing. > > > > On 1/23/24 11:25 AM, Nicholas Chammas wrote: >> Thank you for all the troubleshooting help, Brad. >> >> I am using gRPC via Apache Spark Connect (a Python library), so I am two >> levels removed from c-ares itself. Looking in the Python virtual environment >> where gRPC is installed, I’m not sure what file to run otool on. The only >> seemingly relevant file I could find is called cygrpc.cpython-311-darwin.so, >> and otool didn’t turn up anything interesting on it. >> >> I will take this issue up with the gRPC folks. >> >> I see in several places that the gRPC folks are using ares_gethostbyname: >> https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293 >> https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758 >> https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086 >> >> >>> On Jan 22, 2024, at 1:39 PM, Brad House <b...@brad-house.com> >>> <mailto:b...@brad-house.com> wrote: >>> >>> Are you using gRPC installed via homebrew or is it bundled with something >>> else? Usually package maintainers like homebrew will dynamically link to >>> the system versions of dependencies so they can be updated independently. >>> You might be able to run otool -L on grpc to see what c-ares library its >>> picking up (and if none are listed, it might be compiled in statically). >>> >>> That said, according to your grpc logs, it appears that grpc may be itself >>> performing both A and AAAA queries and expect responses to both of those. >>> I see the "A" reply comes back but the "AAAA" reply never comes and it >>> bails at that point. Many years ago c-ares didn't have a way to request >>> both A and AAAA records with one query, but does these days via >>> ares_getaddrinfo(), and it was recently enhanced with logic to assist in >>> the exact scenario you are seeing, basically it will stop retrying when at >>> least one address family is returned. >>> >>> You might need to escalate this to the gRPC folks. >>> >>> On 1/22/24 12:10 PM, Nicholas Chammas wrote: >>>> Here’s the output of adig and ahost >>>> <https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-adig-ahost-txt>, >>>> both with and without the DNS servers set directly on the network >>>> interface (vs. just on the router). >>>> >>>> I also learned that gRPC 1.60.0 may be using c-ares 1.19.1 >>>> <https://github.com/grpc/grpc/tree/v1.60.0/third_party/cares>, though >>>> again that’s just via looking at the gRPC source and not via some runtime >>>> query. >>>> >>>> >>>>> On Jan 21, 2024, at 7:34 AM, Brad House <b...@brad-house.com> >>>>> <mailto:b...@brad-house.com> wrote: >>>>> >>>>> I think homebrew distributes the 'adig' and 'ahost' utilities from >>>>> c-ares. Can you try using those to do the same lookup so we can see the >>>>> results? >>>>> >>>>> On 1/19/24 11:01 AM, Nicholas Chammas wrote: >>>>>> >>>>>>> On Jan 17, 2024, at 3:38 PM, Brad House <b...@brad-house.com> >>>>>>> <mailto:b...@brad-house.com> wrote: >>>>>>> What version of c-ares is installed? >>>>>>> >>>>>> Sorry about the delay in responding. Answering this question is more >>>>>> difficult than I expected. >>>>>> >>>>>> I know that Spark Connect is running gRPC 1.160.0. Looking through the >>>>>> gRPC repo, I see mention of c-ares 1.13.0 >>>>>> <https://github.com/grpc/grpc/blob/v1.60.0/cmake/cares.cmake#L42>, but I >>>>>> don’t know how that translates to my runtime. Homebrew tells me I have >>>>>> c-ares 1.25.0 installed, but again, I’m not sure if that’s what I’m >>>>>> actually running. >>>>>> >>>>>> Is there a way I can directly query the version of c-ares being run via >>>>>> Spark Connect / gRPC? I asked this question on the gRPC forum >>>>>> <https://groups.google.com/g/grpc-io/c/3tZCa48Xvh8> but no response yet. >>>>>> >>>>>> For the record, I know that c-ares is involved because if I tell gRPC to >>>>>> not use it (via GRPC_DNS_RESOLVER=native >>>>>> <https://github.com/grpc/grpc/blob/b34d98fbd47834845e3f9cdaa4aa706f1aa4eddb/doc/environment_variables.md>) >>>>>> then my problem disappears. >>>>>>> What DNS servers are configured on your MacOS system when its not >>>>>>> operating properly? The output of "scutil --dns" would be helpful here. >>>>>>> >>>>>> Here’s that output. >>>>>> <https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-scutil-dns-txt> >>>>>> I believe 192.168.1.1 is just my local router, and on there is where I >>>>>> have the default DNS servers set to 1.1.1.1 and 1.0.0.1. >>>>>> >>>> >>
-- c-ares mailing list c-ares@lists.haxx.se https://lists.haxx.se/mailman/listinfo/c-ares
