Tim,
> Partial responses should not happen with TCP being used. I didn't follow > along with all the DNS stuff, but does the resolver support TCP by now > or is it still limited to UDP? Using TCP doesn't avoid the truncation, it only delays the problem to a higher limit. From https://datatracker.ietf.org/doc/html/rfc1035#section-4.2.2 The message is prefixed with a two byte length field which gives the message length, excluding the two byte length field. And that limits the length to 2^16. (65536). In our HAproxy setup/scale, we're already hitting those limits. A typical example, I just did a typical query, even with DNS compression enabled, TCP response size with around 2000 answers occupies around 33k (pasting specific output from dig) ;; flags: qr rd; QUERY: 1, ANSWER: 2105, AUTHORITY: 0, ADDITIONAL: 0 ;; Query time: 73 msec ;; SERVER: ::1#53(::1) ;; WHEN: Thu Mar 31 15:22:33 UTC 2022 ;; MSG SIZE rcvd: 33739 It is perfectly reasonable to reach the 65k limit as well. When that limit is exceeded, you might see behavior where the DNS server is randomly choosing "answers" to fit within the 65k limit. Cheers, Abhijeet (https://abhi.host) On Thu, Mar 31, 2022 at 5:01 AM Tim Düsterhus <[email protected]> wrote: > Willy, > > On 3/31/22 08:31, Willy Tarreau wrote: > >> Can you clarify what *exactly* is expected to be removed and what will > >> remain? Is it just SRV DNS records or more? > > > > watchdog was triggered due to this, and in my opinion the concept is > > fundamentally flawed since responses are often partial. As soon as you > > suspect that all active addresses were not delivered, you know that you > > have to put lots of hacks in place. > > Partial responses should not happen with TCP being used. I didn't follow > along with all the DNS stuff, but does the resolver support TCP by now > or is it still limited to UDP? > > I hope this clarifies the situation and doesn't start to make anyone > > worry :-) Anyway there's no emergency, the code is still there, and > > It makes me worry :-) On one box I'm relying on a server-template being > filled with A records from the resolver. Let me explain my use-case: > > For that service I need to proxy (static) resources from a service > *external* to me to: > > a) Prevent IP addresses of my end users being sent to this external > service (for privacy reasons) > b) Locally cache the files to reduce the load on the external service. > > The IP address(es) of the external service are not under my control and > can and will unpredictably change. The service also exposes multiple A > records pointing to different backend servers. I must spread my requests > across multiple of those backend servers to be "nice to them" and not > tie up resources on a single of their backends. > > Sure, DNS-based load-balancing comes with great limitations, due to > browser's behavior, caching resolvers and TTLs, but apparently it works > for the service. And as the service is external to me, that's what I > need to work with. > > Currently it's working well: > > Requests from the user's browser for the proxied resources come in via > my HAProxy, are then routed to my nginx (which handles the caching), > which then uses a dedicated HAProxy 'listen' section for the upstream > requests (as nginx is too dumb to perform DNS resolving properly). This > 'listen' section then uses 'server-template' to spread the requests > across the external service's servers. > > Browser -> HAProxy -> nginx -> HAProxy -> External Service (uncached) > Browser -> HAProxy -> nginx (cached) > > nginx: > > > upstream some_service { > > server unix:/var/lib/haproxy/some-service-internal.sock; > > } > > > > proxy_cache_path /var/lib/some_service/cache/ levels=2 > keys_zone=some_service:50m max_size=7G inactive=30d; > > > > server { > > listen unix:/var/lib/haproxy/some-service.sock; > > real_ip_header X-Forwarded-For; > > > > server_name some-service.example.net; > > > > error_log /var/log/nginx/some_service.log; > > > > proxy_cache some_service; > > proxy_cache_key "https://some_service$request_uri";; > > proxy_cache_background_update on; > > proxy_cache_valid 200 30d; > > proxy_cache_use_stale error timeout invalid_header updating > http_500 http_502 http_503 http_504 http_429; > > proxy_http_version 1.1; > > proxy_temp_path /var/lib/some_service/tmp/; > > add_header X-Proxy-Cache $upstream_cache_status; > > > > location / { > > proxy_pass http://some_service; > > } > > } > > HAProxy: > > > > > listen some-service-internal > > mode http > > > > bind [email protected] mode 666 > > > > http-request set-header Host example.com > > > > server-template some-service 1-4 example.com:443 resolvers > my-resolver check inter 60s ssl sni req.hdr(host) verify required ca-file > ca-certificates.crt resolve-prefer ipv4 > > To replicate this set-up without support for server-template + grabbing > A records from the DNS response I would need to: > > 1) Use some other software (Varnish or Squid might or might not capable > to do this. I neither used either). > 2) Write some custom sidecar script that looks up the IP addresses of > the external service and then updates the HAProxy or nginx config. > > > my concern is more about how we can encourage such existing users to > > start to think about revisiting their approach with new tools and > > practices. And this will also require that we have working alternatives > > to suggest. While I'm pretty confident that the dataplane-api, ingress > > controller and such things already offer a valid response, I don't know > > for sure if they can be considered as drop-in replacement nor if they > > support everything, and this will have to be studied as well before > > starting to scare users! > > > > Best regards > Tim Düsterhus > >
