Can you try running with the following environment variables set, and share the log? That might help us figure out what's going on here.
GRPC_VERBOSITY=DEBUG GRPC_TRACE=client_channel_routing,pick_first,cares_resolver In general, the c-ares resolver should return an error when there's an empty address list, so it should automatically retry the resolution periodically until it succeeds. The only exception I see in the code is if there are balancer addresses successfully returned <https://github.com/grpc/grpc/blob/9794038ae03842573517411df4ef6ac87a377be0/src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc#L325>, but that shouldn't be the case if you're using pick_first. Unless maybe you're using a service config in DNS, but the service config lookup is failing also? Anyway, getting some additional logs will probably help us understand what's going wrong here. On Wed, Aug 10, 2022 at 6:41 AM 'Peter Hurley' via grpc.io < grpc-io@googlegroups.com> wrote: > Thanks for the reply. > > > And would it be possible for you to upgrade your gRPC library and try to > reproduce this? > I didn't see any similar issue (marked fixed or not) in > https://github.com/grpc/grpc/issues; we were hoping the community could > confirm whether this has been observed and fixed already but went > unreported in github. > > > v1.36.4 is over a year old, and a fair handful of bug fixes have gone in > since then. > We're using the still experimental TLSCredentials so every version bump is > non-trivial, and we've already found fixed a number of core bugs > ourselves, so it'll be a while before we're upgrading again in production. > > > Regarding that, are you able to reproduce the conditions in which the > failure occurs, or are they maybe not fully understood? e.g., run a local > DNS server for testing, and modify its records. > Yeah, the exact conditions are not well understood, but almost certainly > happening during a restart of the local caching dnsmasq server due to > intermittent connection loss. > > > On Fri, Aug 5, 2022 at 8:35 PM 'AJ Heller' via grpc.io < > grpc-io@googlegroups.com> wrote: > >> That's mysterious, do you know what the state of the DNS records are when >> this occurs? And would it be possible for you to upgrade your gRPC library >> and try to reproduce this? v1.36.4 is over a year old, and a fair handful >> of bug fixes have gone in since then. >> >> We've been unable to reproduce this failure in testing, and would >>> appreciate any pointers: >>> >> >> Regarding that, are you able to reproduce the conditions in which the >> failure occurs, or are they maybe not fully understood? e.g., run a local >> DNS server for testing, and modify its records. >> >> >>> >>> - what is supposed to re-kick a new DNS resolve if the server list >>> is empty? >>> - where to check in the resolver code for an empty server list? >>> - or any other ideas for how to track down the problem >>> >>> >>> We're using grpc v1.36.4 w/ libcares2 1.14 >>> >>> Regards, >>> Peter Hurley >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "grpc.io" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to grpc-io+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/306779dd-0a68-4b95-851e-0a5979a4e872n%40googlegroups.com >> <https://groups.google.com/d/msgid/grpc-io/306779dd-0a68-4b95-851e-0a5979a4e872n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/CAKzaEUf00rkYWHD6aq1nks8WhVo59wrTcaspkMk2EHUDc1b0JQ%40mail.gmail.com > <https://groups.google.com/d/msgid/grpc-io/CAKzaEUf00rkYWHD6aq1nks8WhVo59wrTcaspkMk2EHUDc1b0JQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Mark D. Roth <r...@google.com> Software Engineer Google, Inc. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJgPXp6brPPuWnVVvArWy3jCZyXXTfL%2BNThjZwTbU5ik__82ng%40mail.gmail.com.
