Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
On Fri, May 19, 2023 at 01:40:30PM +0200, Petr Menšík wrote: > When analysing report [1] for non-responding queries over TCP, I have found > forwarded TCP connections have quite high timeout. If for whatever reason > the forwarder currently set as a last used forwarder is dropping packets > without reject, the TCP will timeout for about 120 seconds on my system. > That is way too much, I think any TCP clients will give up far before that. > This is just quick workaround to improve the situation, not final fix. > > The problem is if the client chooses to use TCP for whatever reason, dnsmasq > will never switch to actually working server until some UDP request arrives > to trigger re-evaluation of last_server responsiveness. It might do so, but > just inside single TCP process. It just stubbornly forwards even 5th retry > to the same server now, when another one might be able to answer right away. > > I think the proper solution would be implementation of event driven reading > of TCP sockets in the main process. I don't think using threads is possible, > because quite a lot of globals used. It should not fork another processes > without --no-daemon parameter, but just use poll() to watch socket > readiness, then read whatever is prepared already. Since TCP DNS message > says its length at the start, it can just allocate buffer big enough for > that connection and iteratively read without blocking. Once it is read, it > can parse it, process it. A bit of socket magic would be required, but > similar approach could solve also sending with multiple calls without > blocking. That would be big change however. > > I think some feedback should be delivered to main dnsmasq process from tcp > processing children, just like cache is updated from cache_end_insert(). I > think it should be able to switch last_server used due to feedback from tcp > client process. I even think there should be different last_server for UDP > and different for TCP, but not untill TCP can report issues too. But not > sure what approach to choose. At first I though about special F_flag, but > all bits for cache record (struct crec) are already used. > > Alternative quick-fix might be in case the TCP request sending fails to some > server to generate UDP request with EDNS header added from tcp child process > to main process UDP socket. It would ensure UDP check is done at the main > process, which might change current used resolver for following TCP > connections too. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2160466 > > -- > Petr Menšík > Software Engineer, RHEL > Red Hat, http://www.redhat.com/ > PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB > From c02cfcb0a358e04636ffd2bcc595860b25b3a440 Mon Sep 17 00:00:00 2001 > From: =?UTF-8?q?Petr=20Men=C5=A1=C3=ADk?= > Date: Wed, 17 May 2023 21:40:19 +0200 > Subject: [PATCH] Add --dns-tcp-timeout option > > Changes send timeout of outgoing TCP connections. Allows waiting just > short time for successful connection before trying another server. > Makes possible faster switch to working server if previous is not > responding. Default socket timeout seems too high for DNS connections. > --- > man/dnsmasq.8 | 4 > src/config.h | 1 + > src/dnsmasq.h | 1 + > src/forward.c | 10 -- > src/option.c | 10 +- > 5 files changed, 23 insertions(+), 3 deletions(-) > > diff --git a/man/dnsmasq.8 b/man/dnsmasq.8 > index 30429df..94fd5b1 100644 > --- a/man/dnsmasq.8 > +++ b/man/dnsmasq.8 > @@ -860,6 +860,10 @@ where this needs to be increased is when using > web-server log file > resolvers, which can generate large numbers of concurrent queries. This > parameter actually controls the number of concurrent queries per server > group, where a server group is the set of server(s) associated with a single > domain. So if a domain has it's own server via --server=/example.com/1.2.3.4 > and 1.2.3.4 is not responding, but queries for *.example.com cannot go > elsewhere, then other queries will not be affected. On configurations with > many such server groups and tight resources, this value may need to be > reduced. > .TP > +.B --dns-tcp-timeout= > +Sets send timeout for forwarded TCP connections. Can be used to reduce time > of waiting > +for successful TCP connection. Default value 0 skips the change of it. > +.TP Missing information about sane values. > .B --dnssec > Validate DNS replies and cache DNSSEC data. When forwarding DNS queries, > dnsmasq requests the > DNSSEC records needed to validate the replies. The replies are validated and > the result returned as > diff --git a/src/config.h b/src/config.h > index 88cf72e..5fd5cdf 100644 > --- a/src/config.h > +++ b/src/config.h > @@ -61,6 +61,7 @@ > #define LOOP_TEST_TYPE T_TXT > #define DEFAULT_FAST_RETRY 1000 /* ms, default delay before fast retry */ > #define STALE_CACHE_EXPIRY 86400 /* 1 day in secs, default maximum expiry > time for stale cache data */ > +#define TCP_TIMEOUT 0 /* timeout of tcp
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
On Wed, May 31, 2023 at 01:40:30PM +0200, Petr Menšík wrote:
> Attached my bigger attempt to solve this problem.
> 7 separate patches included.
>
> Because in rare circumstances servers might have changed since TCP
> connection process started, I serialize domain, server address and its last
> position in server array. Hacked a redirection with special -2 value from
> cache inserting, where it always reads just one server. It then tries server
> at reported position first, iterating on all servers for this domain.
>
> In rare cases there might be a problem, because it does not send source
> address or interface, which might identify correct server. But I doubt those
> would make it different in any real world examples. We have no simple
> identification for changed servers. It works in basic testing well.
>
> Added also separation of TCP and UDP last servers. It should be able to
> forward UDP to server responding just over UDP and TCP to server responding
> just TCP. That should be quite rare case, more teoretical than real-world.
> Maybe change of UDP server should change also TCP, because UDP test can be
> done in parallel.
>
> I have found also unwanted difference from UDP queries. If the response is
> REFUSED, even that were accepted as valid last_server response. Now it sets
> TCP last_server just after non-refused response, not just successful
> connection.
>
> I have tried to look into glibc, that does not seem to set any timeout for
> TCP (vc) queries. Default timeout in dig tool is 10 seconds, it does not
> seem to tweak number of SYN packets sent. I think it just measures time
> before reply arrives. I think ideally we should be able to spawn another TCP
> connection to the other server if it didn't respond in few seconds. And wait
> for fastest response from any of those. But that would require quite
> significant rework of current code.
>
> Did just a basic testing, but those changes improve tested situation.
>
> What do you think about it?
I haven't yet reviewed the patch at the begin of this thread ...
Below some feedback
> Cheers,
> Petr
> From c02cfcb0a358e04636ffd2bcc595860b25b3a440 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Petr=20Men=C5=A1=C3=ADk?=
> Date: Wed, 17 May 2023 21:40:19 +0200
> Subject: [PATCH 1/7] Add --dns-tcp-timeout option
>
> Changes send timeout of outgoing TCP connections. Allows waiting just
> short time for successful connection before trying another server.
> Makes possible faster switch to working server if previous is not
> responding. Default socket timeout seems too high for DNS connections.
> ---
> man/dnsmasq.8 | 4
> src/config.h | 1 +
> src/dnsmasq.h | 1 +
> src/forward.c | 10 --
> src/option.c | 10 +-
> 5 files changed, 23 insertions(+), 3 deletions(-)
>
> diff --git a/man/dnsmasq.8 b/man/dnsmasq.8
> index 30429df..94fd5b1 100644
> --- a/man/dnsmasq.8
> +++ b/man/dnsmasq.8
> @@ -860,6 +860,10 @@ where this needs to be increased is when using
> web-server log file
> resolvers, which can generate large numbers of concurrent queries. This
> parameter actually controls the number of concurrent queries per server
> group, where a server group is the set of server(s) associated with a single
> domain. So if a domain has it's own server via --server=/example.com/1.2.3.4
> and 1.2.3.4 is not responding, but queries for *.example.com cannot go
> elsewhere, then other queries will not be affected. On configurations with
> many such server groups and tight resources, this value may need to be
> reduced.
> .TP
> +.B --dns-tcp-timeout=
> +Sets send timeout for forwarded TCP connections. Can be used to reduce time
> of waiting
> +for successful TCP connection. Default value 0 skips the change of it.
> +.TP
> .B --dnssec
> Validate DNS replies and cache DNSSEC data. When forwarding DNS queries,
> dnsmasq requests the
> DNSSEC records needed to validate the replies. The replies are validated and
> the result returned as
...
> diff --git a/src/forward.c b/src/forward.c
> index ecfeebd..f323fee 100644
> --- a/src/forward.c
> +++ b/src/forward.c
> @@ -1929,8 +1929,14 @@ static ssize_t tcp_talk(int first, int last, int
> start, unsigned char *packet,
> /* Copy connection mark of incoming query to outgoing connection. */
> if (have_mark)
> setsockopt(serv->tcpfd, SOL_SOCKET, SO_MARK, &mark, sizeof(unsigned
> int));
> -#endif
> -
Thanks for removing those trailing white spaces.
> +#endif
> +
> + if (daemon->tcp_timeout>0)
> + {
> + struct timeval tv = {daemon->tcp_timeout, 0};
> + setsockopt(serv->tcpfd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv));
> + }
> +
> if ((!local_bind(serv->tcpfd, &serv->source_addr, serv->interface,
> 0, 1)))
> {
> close(serv->tcpfd);
> diff --git a/src/option.c b/src/option.c
> index 8322725..b94a5ff 100644
> --- a/src/option.c
> +++ b/src/option
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
On Fri, May 26, 2023 at 05:19:49PM +0100, Simon Kelley wrote:
> On 25/05/2023 20:32, Petr Menšík wrote:
> > This problem is best tested by an example, taken from [2] but a bit
> > modified.
> >
> > Let's create hepothetical network issue with one forwarder, which worked
> > fine a while ago.
> >
> > $ sudo iptables -I INPUT -i lo -d 127.0.0.255 -j DROP
> >
> > Now start dnsmasq and send tcp query to it
> >
> > $ dnsmasq -d --log-queries --port 2053 --no-resolv --conf-file=/dev/null
> > --server=127.0.0.255 --server=127.0.0.1
> > $ dig +tcp @localhost -p 2053 test
> >
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to 127.0.0.1#2053: timed out
> >
> > ; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
> > ; (2 servers found)
> > ;; global options: +cmd
> > ;; no servers could be reached
> >
> > Because dig waits much shorter time than dnsmasq does, it never receives
> > any reply. Even when the other server is responding just fine. That is
> > main advantage of having local cache running, isn't it? It should
> > improve things!
> >
> > Now lets be persistent and keep trying:
> >
> > $ time for TRY in {1..6}; do dig +tcp @localhost -p 2053 test; done
> >
> > After few timeouts, it will finally notice something is wrong and tries
> > also the second server, which will answer fast. However this works only
> > with dnsmasq -d, which is not used in production. If I replace it with
> > dnsmasq -k, it will not answer at all!
> >
> > $ dnsmasq -k --log-queries --port 2053 --no-resolv --conf-file=/dev/null
> > --server=127.0.0.255 --server=127.0.0.1
> > $ time for TRY in {1..8}; do dig +tcp @localhost -p 2053 test; done
> >
> > ...
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to 127.0.0.1#2053: timed out
> >
> > ; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
> > ; (2 servers found)
> > ;; global options: +cmd
> > ;; no servers could be reached
> >
> >
> > real 5m20,602s
> > user 0m0,094s
> > sys 0m0,115s
> >
> > This is because with -k it spawns tcp workers, which start always with
> > whatever last_server prepared by last UDP. And until any UDP query
> > arrives to save the day, it will stubbornly try non-responding server
> > first. Even when the other one answers in miliseconds. Notice it have
> > been trying 5 minutes without success.
> >
> > I think this has to be fixed somehow. This is corner case, because TCP
> > queries are usually caused by UDP queries with TC bit set. But there
> > exist real-world examples, where TCP only query makes sense. But dnsmasq
> > does not handle them well. Summarized this at [3].
> >
> > My proposal would be sending UDP query + EDNS0 header in case sending
> > query failed to the main process, which can then trigger forwarders
> > responsiveness and change the last_server to a working one. So
> > subsequent attempts do not fall into the blackhole again and again.
> > EDNS0 header would be there to increase chance for a positive reply from
> > upstream, which can be cached.
> >
> > Would you have other ideas, how to solve this problem?
> >
> > Cheers,
> > Petr
> >
>
>
> The long delay awaiting a connection from a non-responsive server may be
> improved by reducing the value of the TCP_SYNCNT socket option, at least on
> Linux.
>
>
> I think it's pretty easy to pass back the identity of a server which is
> responding to TCP connections to the main process using the same mechanism
> that passes back cache entries. The only wrinkle is if the list of servers
> changes between forking the child process and is sending back data about
> which server worked, for instance is the srever list gets reconfigured.
> Detecting that just needs an "epoch" counter to be included. It's rare, so
> just rejecting a "use this server" update from a child that was spawned in a
> different epoch from the current one should avoid problems. Provided the
> epoch is the same, indices into the server[] array are valid to send across
> the pipe.
>
> I like the idea of using a different valid server for TCP and UDP.
>
> Note that the TCP code does try to pick a good server. It's not currently
> much good with long connection delays, but it does cope with ignoring a
> server which accepts connections and then immediately closes them.
> I guess that must have been a real-world problem sometime.
No one claimt yet it is a real-world problem.
> Cheers,
>
> Simon.
> > [2] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c6
> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c13
> >
> > On 19. 05. 23 13:40, Petr Menšík wrote:
> > > When analysing report [1] for non-responding queries over TCP, I
> > > have found forwarded TCP connections have quite high timeout. If for
> > > whatever reason th
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
This relies on default settings of tcp tuning values set in system. I like the change in 50adf82199c362da6c542f1d22be2eeab7481211 adding timeout variable check. But using SO_SNDTIMEO would allow setting it in time units, regardless default tcp values. And should be supported also on BSD and in general more portable variant. Do you have a reason to use only linux-specific variant? 2 tries seems too little if 6 is the default. I wish we had some value to tell system just retry (a bit) faster than usual. Haven't found that. I would use 3-4 to overcome short connection break at least. 20-30 seconds should be better default value. I think the final fix would be making connections non-blocking and in parallel, accepting first reply which arrives. This is just small tuning to be not so terrible. It has to be sooner than alarm kills the TCP client, at least 3 servers should be tried before that happens. I think previous default allowed just one, so my changed server report over pipe did not happen. I am also not sure we should log every truncated packet without some limit. I think it might lead to denial of service attack. Just stats counter should be incremented, which can be watched by polling. Number of accepted TCP connections is not counted if I see well. As is not number of truncated messages. Would be useful for the service monitoring. On 26. 05. 23 18:53, Simon Kelley wrote: Setting TCP_SYNCNT to 2 limits the delay for a non responsive address to about 8 seconds, which is mouch more sensible. Simon. -- Petr Menšík Software Engineer, RHEL Red Hat, http://www.redhat.com/ PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
Attached my bigger attempt to solve this problem. 7 separate patches
included.
Because in rare circumstances servers might have changed since TCP
connection process started, I serialize domain, server address and its
last position in server array. Hacked a redirection with special -2
value from cache inserting, where it always reads just one server. It
then tries server at reported position first, iterating on all servers
for this domain.
In rare cases there might be a problem, because it does not send source
address or interface, which might identify correct server. But I doubt
those would make it different in any real world examples. We have no
simple identification for changed servers. It works in basic testing well.
Added also separation of TCP and UDP last servers. It should be able to
forward UDP to server responding just over UDP and TCP to server
responding just TCP. That should be quite rare case, more teoretical
than real-world. Maybe change of UDP server should change also TCP,
because UDP test can be done in parallel.
I have found also unwanted difference from UDP queries. If the response
is REFUSED, even that were accepted as valid last_server response. Now
it sets TCP last_server just after non-refused response, not just
successful connection.
I have tried to look into glibc, that does not seem to set any timeout
for TCP (vc) queries. Default timeout in dig tool is 10 seconds, it does
not seem to tweak number of SYN packets sent. I think it just measures
time before reply arrives. I think ideally we should be able to spawn
another TCP connection to the other server if it didn't respond in few
seconds. And wait for fastest response from any of those. But that would
require quite significant rework of current code.
Did just a basic testing, but those changes improve tested situation.
What do you think about it?
Cheers,
Petr
On 26. 05. 23 18:19, Simon Kelley wrote:
On 25/05/2023 20:32, Petr Menšík wrote:
This problem is best tested by an example, taken from [2] but a bit
modified.
Let's create hepothetical network issue with one forwarder, which
worked fine a while ago.
$ sudo iptables -I INPUT -i lo -d 127.0.0.255 -j DROP
Now start dnsmasq and send tcp query to it
$ dnsmasq -d --log-queries --port 2053 --no-resolv
--conf-file=/dev/null --server=127.0.0.255 --server=127.0.0.1
$ dig +tcp @localhost -p 2053 test
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
Because dig waits much shorter time than dnsmasq does, it never
receives any reply. Even when the other server is responding just
fine. That is main advantage of having local cache running, isn't it?
It should improve things!
Now lets be persistent and keep trying:
$ time for TRY in {1..6}; do dig +tcp @localhost -p 2053 test; done
After few timeouts, it will finally notice something is wrong and
tries also the second server, which will answer fast. However this
works only with dnsmasq -d, which is not used in production. If I
replace it with dnsmasq -k, it will not answer at all!
$ dnsmasq -k --log-queries --port 2053 --no-resolv
--conf-file=/dev/null --server=127.0.0.255 --server=127.0.0.1
$ time for TRY in {1..8}; do dig +tcp @localhost -p 2053 test; done
...
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
real 5m20,602s
user 0m0,094s
sys 0m0,115s
This is because with -k it spawns tcp workers, which start always
with whatever last_server prepared by last UDP. And until any UDP
query arrives to save the day, it will stubbornly try non-responding
server first. Even when the other one answers in miliseconds. Notice
it have been trying 5 minutes without success.
I think this has to be fixed somehow. This is corner case, because
TCP queries are usually caused by UDP queries with TC bit set. But
there exist real-world examples, where TCP only query makes sense.
But dnsmasq does not handle them well. Summarized this at [3].
My proposal would be sending UDP query + EDNS0 header in case sending
query failed to the main process, which can then trigger forwarders
responsiveness and change the last_server to a working one. So
subsequent attempts do not fall into the blackhole again and again.
EDNS0 header would be there to increase chance for a positive reply
from upstream, which can be cached.
Would you have other ideas, how to solve this problem?
Cheers,
Petr
The long delay awaiting a connection from a non-res
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
On 26/05/2023 17:19, Simon Kelley wrote: The long delay awaiting a connection from a non-responsive server may be improved by reducing the value of the TCP_SYNCNT socket option, at least on Linux. Setting TCP_SYNCNT to 2 limits the delay for a non responsive address to about 8 seconds, which is mouch more sensible. Simon. ___ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
On 25/05/2023 20:32, Petr Menšík wrote:
This problem is best tested by an example, taken from [2] but a bit
modified.
Let's create hepothetical network issue with one forwarder, which worked
fine a while ago.
$ sudo iptables -I INPUT -i lo -d 127.0.0.255 -j DROP
Now start dnsmasq and send tcp query to it
$ dnsmasq -d --log-queries --port 2053 --no-resolv --conf-file=/dev/null
--server=127.0.0.255 --server=127.0.0.1
$ dig +tcp @localhost -p 2053 test
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
Because dig waits much shorter time than dnsmasq does, it never receives
any reply. Even when the other server is responding just fine. That is
main advantage of having local cache running, isn't it? It should
improve things!
Now lets be persistent and keep trying:
$ time for TRY in {1..6}; do dig +tcp @localhost -p 2053 test; done
After few timeouts, it will finally notice something is wrong and tries
also the second server, which will answer fast. However this works only
with dnsmasq -d, which is not used in production. If I replace it with
dnsmasq -k, it will not answer at all!
$ dnsmasq -k --log-queries --port 2053 --no-resolv --conf-file=/dev/null
--server=127.0.0.255 --server=127.0.0.1
$ time for TRY in {1..8}; do dig +tcp @localhost -p 2053 test; done
...
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
real 5m20,602s
user 0m0,094s
sys 0m0,115s
This is because with -k it spawns tcp workers, which start always with
whatever last_server prepared by last UDP. And until any UDP query
arrives to save the day, it will stubbornly try non-responding server
first. Even when the other one answers in miliseconds. Notice it have
been trying 5 minutes without success.
I think this has to be fixed somehow. This is corner case, because TCP
queries are usually caused by UDP queries with TC bit set. But there
exist real-world examples, where TCP only query makes sense. But dnsmasq
does not handle them well. Summarized this at [3].
My proposal would be sending UDP query + EDNS0 header in case sending
query failed to the main process, which can then trigger forwarders
responsiveness and change the last_server to a working one. So
subsequent attempts do not fall into the blackhole again and again.
EDNS0 header would be there to increase chance for a positive reply from
upstream, which can be cached.
Would you have other ideas, how to solve this problem?
Cheers,
Petr
The long delay awaiting a connection from a non-responsive server may be
improved by reducing the value of the TCP_SYNCNT socket option, at least
on Linux.
I think it's pretty easy to pass back the identity of a server which is
responding to TCP connections to the main process using the same
mechanism that passes back cache entries. The only wrinkle is if the
list of servers changes between forking the child process and is sending
back data about which server worked, for instance is the srever list
gets reconfigured. Detecting that just needs an "epoch" counter to be
included. It's rare, so just rejecting a "use this server" update from a
child that was spawned in a different epoch from the current one should
avoid problems. Provided the epoch is the same, indices into the
server[] array are valid to send across the pipe.
I like the idea of using a different valid server for TCP and UDP.
Note that the TCP code does try to pick a good server. It's not
currently much good with long connection delays, but it does cope with
ignoring a server which accepts connections and then immediately closes
them. I guess that must have been a real-world problem sometime.
Cheers,
Simon.
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c6
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c13
On 19. 05. 23 13:40, Petr Menšík wrote:
When analysing report [1] for non-responding queries over TCP, I have
found forwarded TCP connections have quite high timeout. If for
whatever reason the forwarder currently set as a last used forwarder
is dropping packets without reject, the TCP will timeout for about 120
seconds on my system. That is way too much, I think any TCP clients
will give up far before that. This is just quick workaround to improve
the situation, not final fix.
...
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2160466
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.o
Re: [Dnsmasq-discuss] [PATCH] TCP client timeout setting
This problem is best tested by an example, taken from [2] but a bit
modified.
Let's create hepothetical network issue with one forwarder, which worked
fine a while ago.
$ sudo iptables -I INPUT -i lo -d 127.0.0.255 -j DROP
Now start dnsmasq and send tcp query to it
$ dnsmasq -d --log-queries --port 2053 --no-resolv --conf-file=/dev/null
--server=127.0.0.255 --server=127.0.0.1
$ dig +tcp @localhost -p 2053 test
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
Because dig waits much shorter time than dnsmasq does, it never receives
any reply. Even when the other server is responding just fine. That is
main advantage of having local cache running, isn't it? It should
improve things!
Now lets be persistent and keep trying:
$ time for TRY in {1..6}; do dig +tcp @localhost -p 2053 test; done
After few timeouts, it will finally notice something is wrong and tries
also the second server, which will answer fast. However this works only
with dnsmasq -d, which is not used in production. If I replace it with
dnsmasq -k, it will not answer at all!
$ dnsmasq -k --log-queries --port 2053 --no-resolv --conf-file=/dev/null
--server=127.0.0.255 --server=127.0.0.1
$ time for TRY in {1..8}; do dig +tcp @localhost -p 2053 test; done
...
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to ::1#2053: timed out
;; communications error to 127.0.0.1#2053: timed out
; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
; (2 servers found)
;; global options: +cmd
;; no servers could be reached
real 5m20,602s
user 0m0,094s
sys 0m0,115s
This is because with -k it spawns tcp workers, which start always with
whatever last_server prepared by last UDP. And until any UDP query
arrives to save the day, it will stubbornly try non-responding server
first. Even when the other one answers in miliseconds. Notice it have
been trying 5 minutes without success.
I think this has to be fixed somehow. This is corner case, because TCP
queries are usually caused by UDP queries with TC bit set. But there
exist real-world examples, where TCP only query makes sense. But dnsmasq
does not handle them well. Summarized this at [3].
My proposal would be sending UDP query + EDNS0 header in case sending
query failed to the main process, which can then trigger forwarders
responsiveness and change the last_server to a working one. So
subsequent attempts do not fall into the blackhole again and again.
EDNS0 header would be there to increase chance for a positive reply from
upstream, which can be cached.
Would you have other ideas, how to solve this problem?
Cheers,
Petr
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c6
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c13
On 19. 05. 23 13:40, Petr Menšík wrote:
When analysing report [1] for non-responding queries over TCP, I have
found forwarded TCP connections have quite high timeout. If for
whatever reason the forwarder currently set as a last used forwarder
is dropping packets without reject, the TCP will timeout for about 120
seconds on my system. That is way too much, I think any TCP clients
will give up far before that. This is just quick workaround to improve
the situation, not final fix.
...
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2160466
--
Petr Menšík
Software Engineer, RHEL
Red Hat, http://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
OpenPGP_0x4931CA5B6C9FC5CB.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
