Bug#973581: apt: Occasional pipelined requests failing against localhost nginx mirror

[Jesse Hathaway]

On Tue, 8 Feb 2022 13:21:44 -0600 Jesse Hathaway
 wrote:
> I hit this bug as well with our debian mirror, mirrors.wikimedia.org.
> We also use nginx to serve our mirror and bullseye clients frequently
> receive resets while pulling down packages. I tested reverting commit,
> fa375493c5a4ed9c10d4e5257ac82c6e687862d3, and it does indeed stop the
> connection resets. I also tested against apache2, but I was unable to
> reproduce the issue, so it seems to be nginx specific. I am happy to provide
> pcaps as well.

Using the same script I was able to reproduce the issue against the
following mirror as well:

- http://ftp.us.debian.org/debian/  Server: nginx/1.18.0
- http://debian.unnoba.edu.ar/debian/ Server: nginx/1.18.0
- http://mirrors.tuna.tsinghua.edu.cn/debian/ Server: nginx/1.18.0
- http://repo.ialab.dsu.edu/debian/ Server: nginx/1.18.0
- http://mirror.us.leaseweb.net/debian/ Server: nginx
- http://plug-mirror.rcac.purdue.edu/debian/  Server: nginx/1.14.2

Bug#973581: apt: Occasional pipelined requests failing against localhost nginx mirror

[Jesse Hathaway]

On Sun, 1 Nov 2020 20:55:36 -0800 Stefano Rivera  wrote:
> Recently I noticed repeated sbuild failures during package installation,
> due to connection resets. Can't reproduce with apt 2.1.8 but can with
> >= 2.1.9, so presumably the regression is due to
> fa375493c5a4ed9c10d4e5257ac82c6e687862d3 disabling retries.

I hit this bug as well with our debian mirror, mirrors.wikimedia.org.
We also use nginx to serve our mirror and bullseye clients frequently
receive resets while pulling down packages. I tested reverting commit,
fa375493c5a4ed9c10d4e5257ac82c6e687862d3, and it does indeed stop the
connection resets. I also tested against apache2, but I was unable to
reproduce the issue, so it seems to be nginx specific. I am happy to provide
pcaps as well.

Here is the script I used to reproduce:

  #!/bin/bash

  set -o errexit

  if [[ $EUID -ne 0 ]]; then
printf 'MUST BE ROOT\n' >&2
exit 1
  fi

  sed -i 's/deb.debian.org/mirrors.wikimedia.org/' /etc/apt/sources.list
  apt-get update
  while true; do
apt-get clean
declare -a install_opts=(--yes --download-only --no-install-recommends)
declare -a install_pkgs=(firefox-esr chromium)
if ! apt-get install "${install_opts[@]}" "${install_pkgs[@]}"; then
  printf 'ERROR: We caught a fish!\n' >&2
  tput bel
  exit 1
fi
  done

Here is how I built apt to test the revert:

  #!/bin/bash
  set -o errexit

  echo 'deb-src http://deb.debian.org/debian bullseye main'
>>/etc/apt/sources.list
  apt-get update
  apt-get install --yes git build-essential
  apt-get build-dep --yes apt
  git clone https://salsa.debian.org/apt-team/apt.git
  git config --global user.email "jhatha...@wikimedia.org"
  git config --global user.name "Jesse Hathaway"
  pushd apt
  git checkout 2.2.4
  git checkout -b butter
  git revert --no-edit fa375493c5a4ed9c10d4e5257ac82c6e687862d3
  dpkg-buildpackage -jauto -us -nc -b
  popd
  dpkg -i -- *.deb

I tested with buildah and this Dockerfile:

  FROM docker.io/debian:bullseye
  ENV LANG C.UTF-8
  COPY build /root/
  COPY resets /root/

Bug#973581: apt: Occasional pipelined requests failing against localhost nginx mirror

[Stefano Rivera]

Package: apt
Version: 2.1.11
Severity: normal

I have a local Debian mirror on a buster machine that I occasionally
build packages on too. Pretty stock Debian nginx config.

Recently I noticed repeated sbuild failures during package installation,
due to connection resets. Can't reproduce with apt 2.1.8 but can with
>= 2.1.9, so presumably the regression is due to
fa375493c5a4ed9c10d4e5257ac82c6e687862d3 disabling retries.

Clearly pipelining messups are still present.

It's quite repeatable on an "apt build-dep chromium" in a sid chroot.
Usually 2 packages will fail to download.

I took a stab at finding the source of the resets, but it's nothing
obvious. I see nginx resetting the connection about 100MiB into it,
immediately after an APT GET request. Those kind of pcaps are too big to
share and not that useful. But I can give you access to experiment, if
desired. Prod me by private email / IRC.

It's entirely possible that I simply haven't built anything on that
machine since apt 2.1.9 that triggers this bug.

SR