Package: jenkins.debian.org Severity: important The proxy running on ionos-1 is frequently overloaded and slow to respond. This frequently results in apt or mmdebstrap failing to download packages and makes a lot of jobs fail. For rebootstrap it seems to roughly kill 1/3 of all jobs. As such the situation is a significant waste of resources.
Looking into munin (thanks for providing that!), there seems to be a good correlation between significant iowait https://jenkins.debian.net/munin/debian.net/ionos1-amd64.debian.net/cpu.html and the temporary failures. The system has 40G swap in use at the moment and significant swap traffic. It seems unsurprising that a squid running on such a system would not perform well and time out requests occasionally. A long term solution would be separating the proxy component from the component that causes heavy swapping (whatever that is). The simplest solution would be dedicating a separate small vm at the same datacenter to the proxying. I suppose that 1 vcpu, 1G ram and like 30G of disk space would be the minimum to support a proxy-only vm. Do you agree with the proposed solution? Helmut