On Fri, May 26, 2023 at 05:19:49PM +0100, Simon Kelley wrote:
> On 25/05/2023 20:32, Petr Menšík wrote:
> > This problem is best tested by an example, taken from [2] but a bit
> > modified.
> >
> > Let's create hepothetical network issue with one forwarder, which worked
> > fine a while ago.
> >
> > $ sudo iptables -I INPUT -i lo -d 127.0.0.255 -j DROP
> >
> > Now start dnsmasq and send tcp query to it
> >
> > $ dnsmasq -d --log-queries --port 2053 --no-resolv --conf-file=/dev/null
> > --server=127.0.0.255 --server=127.0.0.1
> > $ dig +tcp @localhost -p 2053 test
> >
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to 127.0.0.1#2053: timed out
> >
> > ; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
> > ; (2 servers found)
> > ;; global options: +cmd
> > ;; no servers could be reached
> >
> > Because dig waits much shorter time than dnsmasq does, it never receives
> > any reply. Even when the other server is responding just fine. That is
> > main advantage of having local cache running, isn't it? It should
> > improve things!
> >
> > Now lets be persistent and keep trying:
> >
> > $ time for TRY in {1..6}; do dig +tcp @localhost -p 2053 test; done
> >
> > After few timeouts, it will finally notice something is wrong and tries
> > also the second server, which will answer fast. However this works only
> > with dnsmasq -d, which is not used in production. If I replace it with
> > dnsmasq -k, it will not answer at all!
> >
> > $ dnsmasq -k --log-queries --port 2053 --no-resolv --conf-file=/dev/null
> > --server=127.0.0.255 --server=127.0.0.1
> > $ time for TRY in {1..8}; do dig +tcp @localhost -p 2053 test; done
> >
> > ...
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to ::1#2053: timed out
> > ;; communications error to 127.0.0.1#2053: timed out
> >
> > ; <<>> DiG 9.18.15 <<>> +tcp @localhost -p 2053 test
> > ; (2 servers found)
> > ;; global options: +cmd
> > ;; no servers could be reached
> >
> >
> > real 5m20,602s
> > user 0m0,094s
> > sys 0m0,115s
> >
> > This is because with -k it spawns tcp workers, which start always with
> > whatever last_server prepared by last UDP. And until any UDP query
> > arrives to save the day, it will stubbornly try non-responding server
> > first. Even when the other one answers in miliseconds. Notice it have
> > been trying 5 minutes without success.
> >
> > I think this has to be fixed somehow. This is corner case, because TCP
> > queries are usually caused by UDP queries with TC bit set. But there
> > exist real-world examples, where TCP only query makes sense. But dnsmasq
> > does not handle them well. Summarized this at [3].
> >
> > My proposal would be sending UDP query + EDNS0 header in case sending
> > query failed to the main process, which can then trigger forwarders
> > responsiveness and change the last_server to a working one. So
> > subsequent attempts do not fall into the blackhole again and again.
> > EDNS0 header would be there to increase chance for a positive reply from
> > upstream, which can be cached.
> >
> > Would you have other ideas, how to solve this problem?
> >
> > Cheers,
> > Petr
> >
>
>
> The long delay awaiting a connection from a non-responsive server may be
> improved by reducing the value of the TCP_SYNCNT socket option, at least on
> Linux.
>
>
> I think it's pretty easy to pass back the identity of a server which is
> responding to TCP connections to the main process using the same mechanism
> that passes back cache entries. The only wrinkle is if the list of servers
> changes between forking the child process and is sending back data about
> which server worked, for instance is the srever list gets reconfigured.
> Detecting that just needs an "epoch" counter to be included. It's rare, so
> just rejecting a "use this server" update from a child that was spawned in a
> different epoch from the current one should avoid problems. Provided the
> epoch is the same, indices into the server[] array are valid to send across
> the pipe.
>
> I like the idea of using a different valid server for TCP and UDP.
>
> Note that the TCP code does try to pick a good server. It's not currently
> much good with long connection delays, but it does cope with ignoring a
> server which accepts connections and then immediately closes them.
> I guess that must have been a real-world problem sometime.
No one claimt yet it is a real-world problem.
> Cheers,
>
> Simon.
> > [2] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c6
> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=2160466#c13
> >
> > On 19. 05. 23 13:40, Petr Menšík wrote:
> > > When analysing report [1] for non-responding queries over TCP, I
> > > have found forwarded TCP connections have quite high timeout. If for
> > > whatever reason the forwarder currently set as a last used forwarder
> > > is dropping packets without reject, the TCP will timeout for about
> > > 120 seconds on my system. That is way too much, I think any TCP
> > > clients will give up far before that. This is just quick workaround
> > > to improve the situation, not final fix.
> > >
> > > ...
> > >
> > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2160466
> > >
Groeten
Geert Stappers
--
Silence is hard to parse
_______________________________________________
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
