> > I wonder if turning on apt's Debug::Acquire::http would give more of a > > clue on where things go wrong? OTOH given this is highly intermittent > > it'd be quite noisy... Christoph, would you be able to give that a try? > > I'll do that now. The first two retries with that setting didn't > reproduce the problem, though.
20:20:00 Get: 31 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB] 20:22:05 GET /debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44%2bdfsg-5%2bdeb9u4_amd64.deb HTTP/1.1 20:22:05 Host: security.debian.org 20:22:05 User-Agent: Debian APT-HTTP/1.3 (1.4.10) 20:22:05 20:22:05 20:22:05 Answer for: http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44+dfsg-5+deb9u4_amd64.deb 20:22:05 HTTP/1.1 200 OK 20:22:05 Server: Apache 20:22:05 X-Content-Type-Options: nosniff 20:22:05 X-Frame-Options: sameorigin 20:22:05 Referrer-Policy: no-referrer 20:22:05 X-Xss-Protection: 1 20:22:05 Last-Modified: Thu, 23 Apr 2020 05:40:59 GMT 20:22:05 ETag: "35840-5a3eeb18b3cf9" 20:22:05 Cache-Control: public, max-age=2592000 20:22:05 Expires: Tue, 28 Apr 2020 19:09:10 GMT 20:22:05 X-Clacks-Overhead: GNU Terry Pratchett 20:22:05 Content-Type: application/x-debian-package 20:22:05 Via: 1.1 varnish 20:22:05 Content-Length: 219200 20:22:05 Accept-Ranges: bytes 20:22:05 Date: Wed, 03 Jun 2020 18:22:05 GMT 20:22:05 Via: 1.1 varnish 20:22:05 Age: 515696 20:22:05 Connection: keep-alive 20:22:05 X-Served-By: cache-fra19137-FRA, cache-hhn4026-HHN 20:22:05 X-Cache: HIT, HIT 20:22:05 X-Cache-Hits: 1, 1 20:22:05 X-Timer: S1591208526.784738,VS0,VE0 20:22:05 20:22:05 Get: 32 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB] 20:24:10 GET /debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44%2bdfsg-5%2bdeb9u4_amd64.deb HTTP/1.1 20:24:10 Host: security.debian.org 20:24:10 User-Agent: Debian APT-HTTP/1.3 (1.4.10) 20:24:10 20:24:10 20:24:10 Answer for: http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.44+dfsg-5+deb9u4_amd64.deb 20:24:10 HTTP/1.1 200 OK 20:24:10 Server: Apache 20:24:10 X-Content-Type-Options: nosniff 20:24:10 X-Frame-Options: sameorigin 20:24:10 Referrer-Policy: no-referrer 20:24:10 X-Xss-Protection: 1 20:24:10 Last-Modified: Thu, 23 Apr 2020 05:40:59 GMT 20:24:10 ETag: "35840-5a3eeb18b3cf9" 20:24:10 Cache-Control: public, max-age=2592000 20:24:10 Expires: Tue, 28 Apr 2020 19:09:10 GMT 20:24:10 X-Clacks-Overhead: GNU Terry Pratchett 20:24:10 Content-Type: application/x-debian-package 20:24:10 Via: 1.1 varnish 20:24:10 Content-Length: 219200 20:24:10 Accept-Ranges: bytes 20:24:10 Date: Wed, 03 Jun 2020 18:24:10 GMT 20:24:10 Via: 1.1 varnish 20:24:10 Age: 515821 20:24:10 Connection: keep-alive 20:24:10 X-Served-By: cache-fra19137-FRA, cache-hhn4074-HHN 20:24:10 X-Cache: HIT, HIT 20:24:10 X-Cache-Hits: 1, 2 20:24:10 X-Timer: S1591208651.836599,VS0,VE0 20:24:10 20:24:10 Get: 33 http://security.debian.org/debian-security stretch/updates/main amd64 libldap-2.4-2 amd64 2.4.44+dfsg-5+deb9u4 [219 kB] 20:24:10 Fetched 16.6 MB in 8min 30s (32.4 kB/s) 20:24:10 E: Failed to fetch http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-common_2.4.44+dfsg-5+deb9u4_all.deb: Connection failed [IP: 151.101.112.204 80] 20:24:10 E: Unable to fetch some packages; try '-o APT::Get::Fix-Missing=true' to continue with missing packages 20:24:11 Reading package lists... I wonder if the 2min delay before the 2nd last package points at something. Possibly the transfer was ok for that .deb, but then apt tries http keepalive but that's already closed? It could be that the NAT layer in the build chroots here have bad iptables rules that break this (they have isolated network namespaces using newpid/newnet). But then, why does it only happen for security.d.o only, and only for jessie+stretch when buster has also security? It's also restricted to a set of VMs at Hetzner, while other machines are fine. Also, the phenomenon is new (~3 months old or so), while the (buster) buildhosts are much older and the config hasn't been touched except for kernel updates. Christoph