Re: [squid-users] Possible access via v6 when no interfaces present, fixable with dns_v4_first

[Amos Jeffries]

On 19/05/18 14:14, Luke wrote:
> 
> We're interested to know why this copy of Squid acted differently to all the
> others.  One potential clue we noticed is that this particular LAN has a lot
> of v6 trafflic flying around on it, compared to the rest of our networks.  I
> understand Squid runs it's own DNS resolver.  Could something on the local
> LAN be announcing itself and the resolver in Squid picking up on that,
> switching around the order of DNS resolution so that it tries that v6
> address first?

You have not mentioned overriding Squids normal DNS hijacking protection
with the ignore_unknown_nameservers directive. So probably not.

What you are seeing in that DNS result ordering is RFC 6540 (aka BCP
177) required behaviour. When a server is advertising IPv6 addresses,
they have preference over IPv4 ones.

The dns_v4_first option is provided to allow to you "force" use of the
outdated IPv4 protocol. It is a workaround, not a fix.

In normal operation Squid will test those addresses and immediately be
told by the network they are not routed, so move on to the IPv4 ones.
Updating its internal DNS cache appropriately not to try them again
until DNS TTL expires the record or IPv4 starts failing too.


> 
> Anything else we should be looking for?  Guidance appreciated!
> 

Anything that would cause an IPv6 TCP SYN request to result in a timeout
instead of ICMPv6 "not available" response packet. If IPv6 between Squid
and those edge.icloud.com IPv6 servers was working you would not have
noticed anything. That includes values of working such as "blocked at
the firewall".

Since Squid is actually getting DNS results I assume DNS lag is not the
problem. But if it is taking a long time to get those results it can
still be playing a part of the issue as it reduces the amount of time
that can be spent on IPv6 connection attempts.

Next thing after DNS to check is ICMP(v6). ICMP is not an optional
protocol and admin blocking it in firewalls can cause major problems
when IPv6 relies on it for routing. Specifically for MTU detection.
 
 

With port 443 traffic is it increasingly likely that the protocol is not
compatible with HTTP/1.1. If the TCP SYN works fine, but the HTTP
messaging hangs (ICMP fault usually) or produces some weird protocol
response that Squid does not handle it can result in errors.

Some web servers have IPv6 records but hang or crash when IPv6 is
delivered to them. Cloudflare have pretty good IPv6 service, so that is
highly unlikely but worth a check if all else fails to bring up a clue.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Possible access via v6 when no interfaces present, fixable with dns_v4_first

[squid-users]

Hello squid users,

I'm trying to understand a strange problem with requests to edge.apple.com,
which I think may be related to IPv6 DNS resolution.

To set the scene - we operate a large (1,000+) fleet of Squid 3.5.25 caches.
Each runs on a separate LAN, connected to the internet via another upstream
proxy, accessed over a wide-area network.  Each local cache runs on a CentOS
6 box, incuding BIND for name resolution.  For DNS resolution, each local
CentOS server runs BIND, which is configured to resolve against a local
Microsoft DNS server, which then resolves internet queries using a
whole-of-WAN BIND service operated by the carrier.  The WAN does not support
IPv6, and CentOS does not have any v6 network interfaces configured.

Recently we became aware of a fault on a single cache serving requests for
edge.icloud.com.  Requests would time out with a TAG_NONE/503 written to the
log.  The error could be replicated with cURL at the CLI using this URL:
https://edge.icloud.com/perf.css.  This was a strange error, because at the
time it happened, it was possible to connect to edge.icloud.com on port 443.
The error was happening in just one site.

To isolate the fault we stripped the Squid config at the affected site right
back to the following:

# Skeleton Squid 3.5.25 config
shutdown_lifetime 2 seconds
max_filedesc 16384
coredump_dir /var/spool/squid
dns_timeout 5 seconds
error_directory /var/www/squid-errors
logfile_rotate 0
http_port 3128
cache_dir ufs /var/spool/squid 8192 16 256
maximum_object_size 536870912 bytes
cache_replacement_policy heap LFUDA
http_access allow localhost
debug_options ALL,5

Here's the messages written to the log when fetching
https://edge.icloud.com/perf.css with curl:

2018/05/08 16:25:46.321 kid1| 14,3| ipcache.cc(362) ipcacheParse: 18 answers
for 'edge.icloud.com'
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #0 [2403:300:a50:105::f]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #1 [2403:300:a50:105::9]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #2 [2403:300:a50:100::e]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #3 [2403:300:a50:101::5]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #4 [2403:300:a50:104::e]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #5 [2403:300:a50:104::9]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #6 [2403:300:a50:104::5]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(431) ipcacheParse:
edge.icloud.com #7 [2403:300:a50:101::6]
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #8 17.248.155.107
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #9 17.248.155.142
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #10 17.248.155.110
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #11 17.248.155.80
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #12 17.248.155.114
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #13 17.248.155.77
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #14 17.248.155.145
2018/05/08 16:25:46.322 kid1| 14,3| ipcache.cc(420) ipcacheParse:
edge.icloud.com #15 17.248.155.89
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths:
Found sources for 'edge.icloud.com:443'
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:
always_direct = DENIED
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:
never_direct = DENIED
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:105::f]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:105::9]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:100::e]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:101::5]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:104::e]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:104::9]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:104::5]:443 flags=1
2018/05/08 16:25:46.322 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:
DIRECT = local=[::] remote=[2403:300:a50:101::6]:443 flags=1