RE: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
> I glanced at the code, I'm pretty sure you're wrong. Yes, I think you are right, and I am wrong. đ I didnât look deep into c-ares code and thought that going to a next server after timeout will either close the server socket or or late responses for such timed-out servers will be ignored. The process_answer() function knows which server responded, but this is used for error handling. So, it seems that there is already parallel query mechanism that I need, which is just invoked too late. > Again, I'm pretty sure all that needs to be done to meet your needs within > c-ares is: > 1) Reduce query response timeout from 5s default to something more reasonable > like 200ms > 2) Have the ability to set an overall query timeout (rather than relying on > number of tries) -- this will properly handle high latency connections > 3) Feedback loop to re-sort server list any time either a) we receive a hard > error trying to reach a server b) we receive a successful response from a > server. (a) would sort server to bottom of server list, (b) would sort > server to top of server list. Seems like a good plan now. This will cover timed-outed servers and will allow to put more responsive servers on top of the list. It is also worth making query response timeout configurable for c-ares clients, so they can let more responsive servers to get on top of the list faster. Thanks, Dmitry Karpov From: Brad House Sent: Thursday, January 20, 2022 11:47 AM To: c-ares discussions Cc: Dmitry Karpov Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I glanced at the code, I'm pretty sure you're wrong. Can you point where it actually dequeues the qid from the global table or closes the file descriptor of the prior server when it goes to the next server on a timeout? Or where it somehow stops listening for a response from the old server at any point prior to the entire query ending? As far as I can tell, its still going to accept a reply from the prior server even when sending the query to the next ... Looking at how it processes timeouts: https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L556 next_server doesn't close the fd https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L757 The qid doesn't appear to get rewritten during sending to the next server https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L799 Lookups for responses from any open fd does a lookup on qid only, there is no server expectation: https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L591 The qid is generated in ares_send(), which is only ever called once per query and doesn't change on re-sends (whether to the same server or others). Again, I'm pretty sure all that needs to be done to meet your needs within c-ares is: 1) Reduce query response timeout from 5s default to something more reasonable like 200ms 2) Have the ability to set an overall query timeout (rather than relying on number of tries) -- this will properly handle high latency connections 3) Feedback loop to re-sort server list any time either a) we receive a hard error trying to reach a server b) we receive a successful response from a server. (a) would sort server to bottom of server list, (b) would sort server to top of server list. On 1/20/22 2:24 PM, Dmitry Karpov via c-ares wrote: > I'm pretty sure that c-ares is already doing this next server as a parallel > query, just the default timeout isn't where you expect. If you set it lower, > it will start a second request at that point the timeout is hit, but if the > first request responds, > it will still use that response if the next server on the list hasn't yet > responded its been a while since I looked at the code, but that seems to > be what I recall. Nope. C-ares iterates name servers sequentially and waits until DNS timeout occurs before switching to the other name server in the list. It matches the expected behavior for resolv.conf on Linux, which prescribes resolver to iterate name servers sequentially. For resolv.conf c-ares honors only the ârotateâ option, which allows to start not from the first server in the name server list, but not any other options. While sequential approach makes sense in general, it doesnât work well for cases with bad name servers (either dual or single stack) where the fastest name resolution is very critical, and it also makes overall DNS timeout non-deterministic depending on a number of bad servers in the list. So, for such cases we either need to have internal sorting putting good servers on top, or use some kind of parallel approach. Thanks, Dmitry Karpov From: Brad House <mailto:b...@brad-house.com> Sent: Wednesday, January 19, 2022 4:24 PM To: c-ares discussions <mailto:c-ares@lists.h
Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
I glanced at the code, I'm pretty sure you're wrong. Can you point where it actually dequeues the qid from the global table or closes the file descriptor of the prior server when it goes to the next server on a timeout? Or where it somehow stops listening for a response from the old server at any point prior to the entire query ending? As far as I can tell, its still going to accept a reply from the prior server even when sending the query to the next ... Looking at how it processes timeouts: https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L556 next_server doesn't close the fd https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L757 The qid doesn't appear to get rewritten during sending to the next server https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L799 Lookups for responses from any open fd does a lookup on qid only, there is no server expectation: https://github.com/c-ares/c-ares/blob/main/src/lib/ares_process.c#L591 The qid is generated in ares_send(), which is only ever called once per query and doesn't change on re-sends (whether to the same server or others). Again, I'm pretty sure all that needs to be done to meet your needs within c-ares is: 1) Reduce query response timeout from 5s default to something more reasonable like 200ms 2) Have the ability to set an overall query timeout (rather than relying on number of tries) -- this will properly handle high latency connections 3) Feedback loop to re-sort server list any time either a) we receive a hard error trying to reach a server b) we receive a successful response from a server. (a) would sort server to bottom of server list, (b) would sort server to top of server list. On 1/20/22 2:24 PM, Dmitry Karpov via c-ares wrote: > I'm pretty sure that c-ares is already doing this next server as a parallel query, just the default timeout isn't where you expect. If you set it lower, it will start a second request at that point the timeout is hit, but if the first request responds, >  it will still use that response if the next server on the list hasn't yet responded its been a while since I looked at the code, but that seems to be what I recall. Nope. C-ares iterates name servers sequentially and waits until DNS timeout occurs before switching to the other name server in the list. It matches the expected behavior for resolv.conf on Linux, which prescribes resolver to iterate name servers sequentially. For resolv.conf c-ares honors only the ârotateâ option, which allows to start not from the first server in the name server list, but not any other options. While sequential approach makes sense in general, it doesnât work well for cases with bad name servers (either dual or single stack) where the fastest name resolution is very critical, and it also makes overall DNS timeout non-deterministic depending on a number of bad servers in the list. So, for such cases we either need to have internal sorting putting good servers on top, or use some kind of parallel approach. Thanks, Dmitry Karpov *From:* Brad House *Sent:* Wednesday, January 19, 2022 4:24 PM *To:* c-ares discussions *Cc:* Dmitry Karpov *Subject:* Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I'm pretty sure that c-ares is already doing this next server as a parallel query, just the default timeout isn't where you expect. If you set it lower, it will start a second request at that point the timeout is hit, but if the first request responds, it will still use that response if the next server on the list hasn't yet responded its been a while since I looked at the code, but that seems to be what I recall. What c-ares does NOT have is an overall query timeout ... that has been requested previously, but it doesn't currently exist (though I agree it should). The logic for retries once it hits the end of the list of nameservers is a bit weird so predicting when a query will return a failed result is basically impossible from what I recall. So this seems to be converging on what I originally suggested then, except now it sounds like also adding the ability to set an overall query timeout. On 1/19/22 7:04 PM, Dmitry Karpov via c-ares wrote: > Again, there's a reason happy eyeballs doesn't just hammer all endpoints returned from getaddrinfo() simultaneously, I'd think the same reasoning would go for DNS servers ... be kind ... start a second query after a short delay if we haven't received a response yet (e.g. 200ms). > It doesn't make sense to hammer more than 1 DNS server if they're all responsive, you just doubled the network load for DNS for no reason. Very true! But in my parallel approach, I didnât mean to start all parallel queries simultaneously.
RE: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
> I'm pretty sure that c-ares is already doing this next server as a parallel > query, just the default timeout isn't where you expect. If you set it lower, > it will start a second request at that point the timeout is hit, but if the > first request responds, > it will still use that response if the next server on the list hasn't yet > responded its been a while since I looked at the code, but that seems to > be what I recall. Nope. C-ares iterates name servers sequentially and waits until DNS timeout occurs before switching to the other name server in the list. It matches the expected behavior for resolv.conf on Linux, which prescribes resolver to iterate name servers sequentially. For resolv.conf c-ares honors only the ârotateâ option, which allows to start not from the first server in the name server list, but not any other options. While sequential approach makes sense in general, it doesnât work well for cases with bad name servers (either dual or single stack) where the fastest name resolution is very critical, and it also makes overall DNS timeout non-deterministic depending on a number of bad servers in the list. So, for such cases we either need to have internal sorting putting good servers on top, or use some kind of parallel approach. Thanks, Dmitry Karpov From: Brad House Sent: Wednesday, January 19, 2022 4:24 PM To: c-ares discussions Cc: Dmitry Karpov Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I'm pretty sure that c-ares is already doing this next server as a parallel query, just the default timeout isn't where you expect. If you set it lower, it will start a second request at that point the timeout is hit, but if the first request responds, it will still use that response if the next server on the list hasn't yet responded its been a while since I looked at the code, but that seems to be what I recall. What c-ares does NOT have is an overall query timeout ... that has been requested previously, but it doesn't currently exist (though I agree it should). The logic for retries once it hits the end of the list of nameservers is a bit weird so predicting when a query will return a failed result is basically impossible from what I recall. So this seems to be converging on what I originally suggested then, except now it sounds like also adding the ability to set an overall query timeout. On 1/19/22 7:04 PM, Dmitry Karpov via c-ares wrote: > Again, there's a reason happy eyeballs doesn't just hammer all endpoints > returned from getaddrinfo() simultaneously, I'd think the same reasoning > would go for DNS servers ... be kind ... start a second query after a short > delay if we haven't received a response yet (e.g. 200ms). > It doesn't make sense to hammer more than 1 DNS server if they're all > responsive, you just doubled the network load for DNS for no reason. Very true! But in my parallel approach, I didnât mean to start all parallel queries simultaneously. I didnât nail the details, but obviously such approach should be similar to the Happy Eyeballs even for single stacks. So, parallel queries in the parallel approach should be started with some small delays like 200ms in Happy Eyeballs, but the whole name resolution should be controlled by one constant and deterministic timeout â i.e. 5s, which shouldnât depend on the number of the name servers in the list, as it is currently the case with c-ares. In my use cases, using c-ares with libcurl, I see different name resolution timeouts: 5s, 15s,⌠depending on a number of bad name servers in the list, which cause some my time critical services to fail. And we canât just use 200ms as a DNS timeout per name server and iterate name servers sequentially, because there are high-latency satellite links with big RTTs, which require 2s and sometimes more for name resolutions. Thatâs why the parallel approach (with delays between parallel queries) seems to me as a better solution for bad name servers than the sequential one. But as I said, any improvements in this area will be very welcomed c-ares extensions, especially if they help libcurl with c-ares, used by a lot of people, to better handle issues with bad name servers. Thanks, Dmitry Karpov From: Brad House <mailto:b...@brad-house.com> Sent: Wednesday, January 19, 2022 2:37 PM To: c-ares discussions <mailto:c-ares@lists.haxx.se> Cc: Dmitry Karpov <mailto:dkar...@roku.com> Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I guess it always depends on the design of whatever is using c-ares. In my own use cases, I have a single ares_channel running on an event loop and enqueue my lookups to there ... so it keeps state. Nothing with thread local storage or anythin
Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
I'm pretty sure that c-ares is already doing this next server as a parallel query, just the default timeout isn't where you expect. If you set it lower, it will start a second request at that point the timeout is hit, but if the first request responds, it will still use that response if the next server on the list hasn't yet responded its been a while since I looked at the code, but that seems to be what I recall. What c-ares does NOT have is an overall query timeout ... that has been requested previously, but it doesn't currently exist (though I agree it should). The logic for retries once it hits the end of the list of nameservers is a bit weird so predicting when a query will return a failed result is basically impossible from what I recall. So this seems to be converging on what I originally suggested then, except now it sounds like also adding the ability to set an overall query timeout. On 1/19/22 7:04 PM, Dmitry Karpov via c-ares wrote: > Again, there's a reason happy eyeballs doesn't just hammer all endpoints returned from getaddrinfo() simultaneously, I'd think the same reasoning would go for DNS servers ... be kind ... start a second query after a short delay if we haven't received a response yet (e.g. 200ms). > It doesn't make sense to hammer more than 1 DNS server if they're all responsive, you just doubled the network load for DNS for no reason. Very true! But in my parallel approach, I didnât mean to start all parallel queries simultaneously. I didnât nail the details, but obviously such approach should be similar to the Happy Eyeballs even for single stacks. So, parallel queries in the parallel approach should be started with some small delays like 200ms in Happy Eyeballs, but the whole name resolution should be controlled by one constant and deterministic timeout â i.e. 5s, which shouldnât depend on the number of the name servers in the list, as it is currently the case with c-ares. In my use cases, using c-ares with libcurl, I see different name resolution timeouts: 5s, 15s,⌠depending on a number of bad name servers in the list, which cause some my time critical services to fail. And we canât just use 200ms as a DNS timeout per name server and iterate name servers sequentially, because there are high-latency satellite links with big RTTs, which require 2s and sometimes more for name resolutions. Thatâs why the parallel approach (with delays between parallel queries) seems to me as a better solution for bad name servers than the sequential one. But as I said, any improvements in this area will be very welcomed c-ares extensions, especially if they help libcurl with c-ares, used by a lot of people, to better handle issues with bad name servers. Thanks, Dmitry Karpov *From:* Brad House *Sent:* Wednesday, January 19, 2022 2:37 PM *To:* c-ares discussions *Cc:* Dmitry Karpov *Subject:* Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I guess it always depends on the design of whatever is using c-ares. In my own use cases, I have a single ares_channel running on an event loop and enqueue my lookups to there ... so it keeps state. Nothing with thread local storage or anything, just dispatching to that event loop for any DNS queries that need to be performed. The single ares_channel can handle multiple simultaneous DNS queries. Also, since there is a proposed feedback loop, if a DNS server is no longer reachable, it will re-sort the list for any future requests, so it would only impact a single request (ok, well, whatever number of requests came in before the timeout or error occurred). Again, there's a reason happy eyeballs doesn't just hammer all endpoints returned from getaddrinfo() simultaneously, I'd think the same reasoning would go for DNS servers ... be kind ... start a second query after a short delay if we haven't received a response yet (e.g. 200ms). It doesn't make sense to hammer more than 1 DNS server if they're all responsive, you just doubled the network load for DNS for no reason. On 1/19/22 5:25 PM, Dmitry Karpov via c-ares wrote: > I wasn't suggesting this be outside of c-ares, I was talking about implementing this inside of c-ares as a simpler alternative to your proposal. OK, I got it know. :) Pre-sorting name servers based on reachability from previous queries or/and protocol family may help in some cases, but the sequential approach, even with sorting, still will have some issues that the parallel approach allows to solve more efficiently. For example, the first query when nothing is sorted, may cause critical connection timeouts aborting some applications, and storing name server âreachability metricsâ which name servers will be sorted on will require either thread local storage (
RE: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
> Again, there's a reason happy eyeballs doesn't just hammer all endpoints > returned from getaddrinfo() simultaneously, I'd think the same reasoning > would go for DNS servers ... be kind ... start a second query after a short > delay if we haven't received a response yet (e.g. 200ms). > It doesn't make sense to hammer more than 1 DNS server if they're all > responsive, you just doubled the network load for DNS for no reason. Very true! But in my parallel approach, I didnât mean to start all parallel queries simultaneously. I didnât nail the details, but obviously such approach should be similar to the Happy Eyeballs even for single stacks. So, parallel queries in the parallel approach should be started with some small delays like 200ms in Happy Eyeballs, but the whole name resolution should be controlled by one constant and deterministic timeout â i.e. 5s, which shouldnât depend on the number of the name servers in the list, as it is currently the case with c-ares. In my use cases, using c-ares with libcurl, I see different name resolution timeouts: 5s, 15s,⌠depending on a number of bad name servers in the list, which cause some my time critical services to fail. And we canât just use 200ms as a DNS timeout per name server and iterate name servers sequentially, because there are high-latency satellite links with big RTTs, which require 2s and sometimes more for name resolutions. Thatâs why the parallel approach (with delays between parallel queries) seems to me as a better solution for bad name servers than the sequential one. But as I said, any improvements in this area will be very welcomed c-ares extensions, especially if they help libcurl with c-ares, used by a lot of people, to better handle issues with bad name servers. Thanks, Dmitry Karpov From: Brad House Sent: Wednesday, January 19, 2022 2:37 PM To: c-ares discussions Cc: Dmitry Karpov Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) I guess it always depends on the design of whatever is using c-ares. In my own use cases, I have a single ares_channel running on an event loop and enqueue my lookups to there ... so it keeps state. Nothing with thread local storage or anything, just dispatching to that event loop for any DNS queries that need to be performed. The single ares_channel can handle multiple simultaneous DNS queries. Also, since there is a proposed feedback loop, if a DNS server is no longer reachable, it will re-sort the list for any future requests, so it would only impact a single request (ok, well, whatever number of requests came in before the timeout or error occurred). Again, there's a reason happy eyeballs doesn't just hammer all endpoints returned from getaddrinfo() simultaneously, I'd think the same reasoning would go for DNS servers ... be kind ... start a second query after a short delay if we haven't received a response yet (e.g. 200ms). It doesn't make sense to hammer more than 1 DNS server if they're all responsive, you just doubled the network load for DNS for no reason. On 1/19/22 5:25 PM, Dmitry Karpov via c-ares wrote: > I wasn't suggesting this be outside of c-ares, I was talking about > implementing this inside of c-ares as a simpler alternative to your proposal. OK, I got it know. :) Pre-sorting name servers based on reachability from previous queries or/and protocol family may help in some cases, but the sequential approach, even with sorting, still will have some issues that the parallel approach allows to solve more efficiently. For example, the first query when nothing is sorted, may cause critical connection timeouts aborting some applications, and storing name server âreachability metricsâ which name servers will be sorted on will require either thread local storage (thus requiring each thread to go through the same âname server discoveryâ procedure as the other app threads using c-ares) or some global access to the metrics data with proper read/write accesses, needed by multi-threaded apps. Also, if run-time conditions change from the previous query then the sorted list may be not sorted correctly for the current conditions, and thus not the best server or even bad server may be tried first, thus increasing name resolution time. The parallel approach, on the other hand, will provide the fastest name resolution regardless the previous queries, so it doesnât need to store any name server metrics and do pre-processing of the name server list from OS. But I agree that implementing parallel approach may be not very easy and any improvements in this area will be a very welcomed extension, anyway. So, if you think that updated sequential approach with smart sorting is much easier to implement than the parallel one, then hopefully we can get it in next c-ares updates. Thanks, Dmitry Ka
Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
I guess it always depends on the design of whatever is using c-ares. In my own use cases, I have a single ares_channel running on an event loop and enqueue my lookups to there ... so it keeps state. Nothing with thread local storage or anything, just dispatching to that event loop for any DNS queries that need to be performed. The single ares_channel can handle multiple simultaneous DNS queries. Also, since there is a proposed feedback loop, if a DNS server is no longer reachable, it will re-sort the list for any future requests, so it would only impact a single request (ok, well, whatever number of requests came in before the timeout or error occurred). Again, there's a reason happy eyeballs doesn't just hammer all endpoints returned from getaddrinfo() simultaneously, I'd think the same reasoning would go for DNS servers ... be kind ... start a second query after a short delay if we haven't received a response yet (e.g. 200ms). It doesn't make sense to hammer more than 1 DNS server if they're all responsive, you just doubled the network load for DNS for no reason. On 1/19/22 5:25 PM, Dmitry Karpov via c-ares wrote: > I wasn't suggesting this be outside of c-ares, I was talking about implementing this inside of c-ares as a simpler alternative to your proposal. OK, I got it know. :) Pre-sorting name servers based on reachability from previous queries or/and protocol family may help in some cases, but the sequential approach, even with sorting, still will have some issues that the parallel approach allows to solve more efficiently. For example, the first query when nothing is sorted, may cause critical connection timeouts aborting some applications, and storing name server âreachability metricsâ which name servers will be sorted on will require either thread local storage (thus requiring each thread to go through the same âname server discoveryâ procedure as the other app threads using c-ares) or some global access to the metrics data with proper read/write accesses, needed by multi-threaded apps. Also, if run-time conditions change from the previous query then the sorted list may be not sorted correctly for the current conditions, and thus not the best server or even bad server may be tried first, thus increasing name resolution time. The parallel approach, on the other hand, will provide the fastest name resolution regardless the previous queries, so it doesnât need to store any name server metrics and do pre-processing of the name server list from OS. But I agree that implementing parallel approach may be not very easy and any improvements in this area will be a very welcomed extension, anyway. So, if you think that updated sequential approach with smart sorting is much easier to implement than the parallel one, then hopefully we can get it in next c-ares updates. Thanks, Dmitry Karpov *From:* Brad House *Sent:* Wednesday, January 19, 2022 12:10 PM *To:* c-ares discussions *Cc:* Dmitry Karpov *Subject:* Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) Commenting below ... On 1/19/22 2:51 PM, Dmitry Karpov via c-ares wrote: > Infact, happyeyeballs itself doesn't always do parallel connection attempts, its an implementation-defined delay before also attempting the next address in the list. In case of Happy Eyeballs, a delay between IPv4 and IPv6 connections is constant and typically relatively short â 200-300ms. But non-functional IPv6 name servers in the server list may create dynamic delays in connection establishment which can be very large. By default, c-ares uses 5s timeout per name server, so it may take 5s and more (if several IPv6 name servers are in the list)  to get to the connection Happy Eyeballs thus taking much more than expected 200-300ms. It would be assumed as part of this patch set, this timer would be reduced. > It would be much easier to stay closer to happy eyeballs and just sort the dns server list using prior result success/fail (even upfront sorting using some algorithm to interleave ipv6/ipv4 in a pattern would help, >  maybe with using logic such as from RFC6724 sec 2.1 like we do in ares_getaddrinfo for returned addresses, but instead of the nameservers themselves). Yes, of course, it is possible that c-ares client can implement some kind of name server sorting/filtering logic outside of c-ares and just pass a list of âgoodâ name servers to c-ares,  but in this case it has to be more involved into the name resolution business than it would be desired. I wasn't suggesting this be outside of c-ares, I was talking about implementing this inside of c-ares as a simpler alternative to your proposal. -Brad -- c-ares mailing list c-ares@lists.haxx.se https://lists.haxx.se/listinfo/c-ares
RE: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
> I wasn't suggesting this be outside of c-ares, I was talking about > implementing this inside of c-ares as a simpler alternative to your proposal. OK, I got it know. :) Pre-sorting name servers based on reachability from previous queries or/and protocol family may help in some cases, but the sequential approach, even with sorting, still will have some issues that the parallel approach allows to solve more efficiently. For example, the first query when nothing is sorted, may cause critical connection timeouts aborting some applications, and storing name server âreachability metricsâ which name servers will be sorted on will require either thread local storage (thus requiring each thread to go through the same âname server discoveryâ procedure as the other app threads using c-ares) or some global access to the metrics data with proper read/write accesses, needed by multi-threaded apps. Also, if run-time conditions change from the previous query then the sorted list may be not sorted correctly for the current conditions, and thus not the best server or even bad server may be tried first, thus increasing name resolution time. The parallel approach, on the other hand, will provide the fastest name resolution regardless the previous queries, so it doesnât need to store any name server metrics and do pre-processing of the name server list from OS. But I agree that implementing parallel approach may be not very easy and any improvements in this area will be a very welcomed extension, anyway. So, if you think that updated sequential approach with smart sorting is much easier to implement than the parallel one, then hopefully we can get it in next c-ares updates. Thanks, Dmitry Karpov From: Brad House Sent: Wednesday, January 19, 2022 12:10 PM To: c-ares discussions Cc: Dmitry Karpov Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) Commenting below ... On 1/19/22 2:51 PM, Dmitry Karpov via c-ares wrote: > Infact, happyeyeballs itself doesn't always do parallel connection attempts, > its an implementation-defined delay before also attempting the next address > in the list. In case of Happy Eyeballs, a delay between IPv4 and IPv6 connections is constant and typically relatively short â 200-300ms. But non-functional IPv6 name servers in the server list may create dynamic delays in connection establishment which can be very large. By default, c-ares uses 5s timeout per name server, so it may take 5s and more (if several IPv6 name servers are in the list) to get to the connection Happy Eyeballs thus taking much more than expected 200-300ms. It would be assumed as part of this patch set, this timer would be reduced. > It would be much easier to stay closer to happy eyeballs and just sort the > dns server list using prior result success/fail (even upfront sorting using > some algorithm to interleave ipv6/ipv4 in a pattern would help, > maybe with using logic such as from RFC6724 sec 2.1 like we do in > ares_getaddrinfo for returned addresses, but instead of the nameservers > themselves). Yes, of course, it is possible that c-ares client can implement some kind of name server sorting/filtering logic outside of c-ares and just pass a list of âgoodâ name servers to c-ares, but in this case it has to be more involved into the name resolution business than it would be desired. I wasn't suggesting this be outside of c-ares, I was talking about implementing this inside of c-ares as a simpler alternative to your proposal. -Brad -- c-ares mailing list c-ares@lists.haxx.se https://lists.haxx.se/listinfo/c-ares
Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
Commenting below ... On 1/19/22 2:51 PM, Dmitry Karpov via c-ares wrote: > Infact, happyeyeballs itself doesn't always do parallel connection attempts, its an implementation-defined delay before also attempting the next address in the list. In case of Happy Eyeballs, a delay between IPv4 and IPv6 connections is constant and typically relatively short â 200-300ms. But non-functional IPv6 name servers in the server list may create dynamic delays in connection establishment which can be very large. By default, c-ares uses 5s timeout per name server, so it may take 5s and more (if several IPv6 name servers are in the list) Â to get to the connection Happy Eyeballs thus taking much more than expected 200-300ms. It would be assumed as part of this patch set, this timer would be reduced. > It would be much easier to stay closer to happy eyeballs and just sort the dns server list using prior result success/fail (even upfront sorting using some algorithm to interleave ipv6/ipv4 in a pattern would help, > Â maybe with using logic such as from RFC6724 sec 2.1 like we do in ares_getaddrinfo for returned addresses, but instead of the nameservers themselves). Yes, of course, it is possible that c-ares client can implement some kind of name server sorting/filtering logic outside of c-ares and just pass a list of âgoodâ name servers to c-ares, Â but in this case it has to be more involved into the name resolution business than it would be desired. I wasn't suggesting this be outside of c-ares, I was talking about implementing this inside of c-ares as a simpler alternative to your proposal. -Brad-- c-ares mailing list c-ares@lists.haxx.se https://lists.haxx.se/listinfo/c-ares
RE: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
Hi Brad, Thanks for reply. A few comments. > Infact, happyeyeballs itself doesn't always do parallel connection attempts, > its an implementation-defined delay before also attempting the next address > in the list. In case of Happy Eyeballs, a delay between IPv4 and IPv6 connections is constant and typically relatively short â 200-300ms. But non-functional IPv6 name servers in the server list may create dynamic delays in connection establishment which can be very large. By default, c-ares uses 5s timeout per name server, so it may take 5s and more (if several IPv6 name servers are in the list) to get to the connection Happy Eyeballs thus taking much more than expected 200-300ms. > It would be much easier to stay closer to happy eyeballs and just sort the > dns server list using prior result success/fail (even upfront sorting using > some algorithm to interleave ipv6/ipv4 in a pattern would help, > maybe with using logic such as from RFC6724 sec 2.1 like we do in > ares_getaddrinfo for returned addresses, but instead of the nameservers > themselves). Yes, of course, it is possible that c-ares client can implement some kind of name server sorting/filtering logic outside of c-ares and just pass a list of âgoodâ name servers to c-ares, but in this case it has to be more involved into the name resolution business than it would be desired. A cross-platform client will have to implement a multi-platform logic to get a list of name servers like c-ares, thus potentially duplicating the code doing the same thing, and then also implement a mechanism of checking and filtering out bad servers to create a list of good servers to feed it to c-ares or some c-ares client application. In my opinion, this is too much complexity to ask from a resolver client, which only desires to get host name resolutions from the resolver relying on the resolver to do it using the best and the fastest way. Using parallel DNS requests for IPv4 and IPv6 stacks in c-ares will help to avoid complicated workarounds in all clients trying to resolve dual-stack issues, and option to use parallel requests for all servers in the list will help to address a general problem of bad name servers going before good ones. I think these issues are quite common, so I thought that it would be a good c-ares extension to provide an option to deal more easily with bad name servers for its clients using parallel approach. Thanks, Dmitry Karpov From: Brad House Sent: Tuesday, January 18, 2022 2:56 PM To: c-ares discussions Cc: Dmitry Karpov Subject: Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6) We would certainly take patches to accomplish something like this. I'm not sure how easy it would be to work the internals of c-ares to understand parallel nameservers for the same query, it would probably be a substantial change though. Also, I'm also not sure how "friendly" it is to actually perform the requests in parallel at all times. Infact, happyeyeballs itself doesn't always do parallel connection attempts, its an implementation-defined delay before also attempting the next address in the list. It also has you maintain a feedback loop in order to sort known bad addresses to be least prefered. It would be much easier to stay closer to happy eyeballs and just sort the dns server list using prior result success/fail (even upfront sorting using some algorithm to interleave ipv6/ipv4 in a pattern would help, maybe with using logic such as from RFC6724 sec 2.1 like we do in ares_getaddrinfo for returned addresses, but instead of the nameservers themselves). Obviously, if you're using something that isn't maintaining state, the feedback loop won't help, but maybe the pre-sorting suggestion would. Assuming you're using libcurl, and not a stateless command line curl, I think it would ultimately accomplish your goal. -Brad On 1/18/22 4:23 PM, Dmitry Karpov via c-ares wrote: Hello, Using libcurl with c-ares for dual-stack scenarios, I observed that c-ares doesnât distinguish between IPv4 and IPv6 nameservers in the name server list (i.e. listed in resolv.conf on Linux systems) and iterates through them sequentially. Such approach creates problems for dual-stack systems, when one stack is either not fully functional or have not responding/reachable name servers put on top of the list. In such scenarios, problems with one stack (i.e. IPv6) may create name resolution delays and timeouts for the other fully functional stack (i.e. IPv4) if name servers from the not functioning stack go before the good stack like: [IPv6 - BAD] 2001:4860:4860:: 2001:4860:4860::8844 [IPv4 - GOOD] 8.8.8.8 8.8.4.4 In this scenario, it will take two resolution timeouts for bad IPv6 name servers before good IPv4 name servers are reached, and this negatively impacts Happy Eyeball impl
Re: Feature request for parallel queries for name servers from different protocol families (IPv4 vs IPv6)
We would certainly take patches to accomplish something like this. I'm not sure how easy it would be to work the internals of c-ares to understand parallel nameservers for the same query, it would probably be a substantial change though. Also, I'm also not sure how "friendly" it is to actually perform the requests in parallel at all times. Infact, happyeyeballs itself doesn't always do parallel connection attempts, its an implementation-defined delay before also attempting the next address in the list. It also has you maintain a feedback loop in order to sort known bad addresses to be least prefered. It would be much easier to stay closer to happy eyeballs and just sort the dns server list using prior result success/fail (even upfront sorting using some algorithm to interleave ipv6/ipv4 in a pattern would help, maybe with using logic such as from RFC6724 sec 2.1 like we do in ares_getaddrinfo for returned addresses, but instead of the nameservers themselves). Obviously, if you're using something that isn't maintaining state, the feedback loop won't help, but maybe the pre-sorting suggestion would. Assuming you're using libcurl, and not a stateless command line curl, I think it would ultimately accomplish your goal. -Brad On 1/18/22 4:23 PM, Dmitry Karpov via c-ares wrote: Hello, Using libcurl with c-ares for dual-stack scenarios, I observed that c-ares doesnât distinguish between IPv4 and IPv6 nameservers in the name server list (i.e. listed in resolv.conf on Linux systems)  and iterates through them sequentially. Such approach creates problems for dual-stack systems, when one stack is either not fully functional or have not responding/reachable name servers put on top of the list. In such scenarios, problems with one stack (i.e. IPv6) may create name resolution delays and timeouts for the other fully functional stack (i.e. IPv4) if name servers from the not functioning stack go before the good stack like: [IPv6 - BAD] 2001:4860:4860:: 2001:4860:4860::8844 [IPv4 - GOOD] 8.8.8.8 8.8.4.4 In this scenario, it will take two resolution timeouts for bad IPv6 name servers before good IPv4 name servers are reached, and this negatively impacts Happy Eyeball implementations in client applications (i.e. libcurl) which have to wait for too long before they can start dual-stack connections. So here is the feature request which should help dual-stack client applications to work more efficiently when name servers from not functioning stack are listed before name servers from a good stack: * Split the flat name server list for both stacks into two lists for each stack. * Execute parallel DNS queries for each stack list, iterating each stack list sequentially as it is currently done for the whole dual-stack list. * Return the result whichever comes first. This feature request can be also considered in a broader scope: * Run parallel queries for each name server in the dual-stack list regardless of whether it is IPv4 or IPv6 address. * Return the result whichever comes first. Such broader scope will also allow to skip over bad name servers and get host resolution results much more quickly even for the same stack (i.e. when name server list for a single stack contains some not responding name servers at the top). But in this case, it should be probably controlled by some new c-ares option, so the current sequential approach may be enforced if needed. Thanks, Dmitry Karpov -- c-ares mailing list c-ares@lists.haxx.se https://lists.haxx.se/listinfo/c-ares
