Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
David Kalnischkies dixit: system e.g. now even figures out that specific files mentioned within a Release file haven't changed since the previous Release file we have and doesn't even try to download it then (which mostly solves problems with servers who do not even support I-M-S hits), it doesn't calculate checksums it doesn't have in the Release file to compare with (like Woah, you rock! A heartfelt Thanks! bye, //mirabilos -- 21:49⎜allamoox:#sendmail I have a question guys, ⎜Can I use my PC as SMTP server, I use Windows 7 . ⎜Already googled and Installed IIS ⎜but Still I can't send E-mail
Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
Control: tags -1 pending Hi On Sat, Sep 20, 2014 at 09:15:15AM +0200, Michael Vogt wrote: The downside if of course that if a Packages file is missing (because of a earlier download error for example) then it won't be fetched again, so the patch is probably too simplistic. But I think something like this should be done to avoid the extra verify step if we know that nothing has changed. This is actually implemented now, some of it in 1.1~exp8 I believe, but most in the version currently in git in all its glory. The acquire system e.g. now even figures out that specific files mentioned within a Release file haven't changed since the previous Release file we have and doesn't even try to download it then (which mostly solves problems with servers who do not even support I-M-S hits), it doesn't calculate checksums it doesn't have in the Release file to compare with (like SHA512 currently) and a bunch of other nifty stuff. See also the test/integration/test-apt-update-* tests for details on what it can do (and prevents attackers from doing). Best regards David Kalnischkies signature.asc Description: Digital signature
Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
On Fri, Sep 19, 2014 at 09:00:43AM +, Thorsten Glaser wrote:
Package: apt
Version: 1.0.9.1
Severity: minor
Thanks for your bugreport.
Michael Vogt requested I file this as official bugreport:
The recent revalidation changes lead to a simple apt-get update
on my buildd machine (m68k VM with 198 MHz CPU) with only the
incoming.d-p.o repository modified, but not unstable, taking:
Fetched 156 kB in 8min 33s (304 B/s)
The sources.list on that chroot is just:
deb http://ftp.de.debian.org/debian-ports unstable main
deb http://ftp.de.debian.org/debian-ports unreleased main
deb-src http://ftp.de.debian.org/debian unstable main
# plus the “incoming” repository
With almost 200 MHz, this is one of the faster “machines”.
Our regular buildds have 25, 33, 50 MHz.
[..]
This is indeed a pretty big hit on this hardware :/
The following patch should fix most of this:
diff --git a/apt-pkg/acquire-item.cc b/apt-pkg/acquire-item.cc
index 5df4372..28ede30 100644
--- a/apt-pkg/acquire-item.cc
+++ b/apt-pkg/acquire-item.cc
@@ -1558,8 +1558,9 @@ void pkgAcqMetaIndex::AuthDone(string Message)
/*{{{*/
}
- // Download further indexes with verification
- QueueIndexes(true);
+ // Download further indexes with verification if something has changed
+ if(IMSHit == false)
+ QueueIndexes(true);
// is it a clearsigned MetaIndex file?
if (DestFile == SigFile)
The downside if of course that if a Packages file is missing (because
of a earlier download error for example) then it won't be fetched
again, so the patch is probably too simplistic. But I think something
like this should be done to avoid the extra verify step if we know
that nothing has changed.
Cheers,
Michael
The cause for this is that even a not-modified now triggers
reverification of all downloaded files. The visible effect
is a process called “copy” taking 90% CPU for minutes, in
top(1).
Michael and I think that apt should, when the Release and
Release.gpg (or InRelease, but we aren’t using them any
more, IIRC?) files are not modified, just reverify these,
and, if they pass, assume the other data on the disc is
valid/verified (because the files only get moved out of
partial/ once verified AFAIK, and because if an unmodified
Release file passes reverification, it would have passed
initial verification as well, normally).
Michael also thinks apt could cache more information, such
as the original hashes, to speed up things more.
-- Package-specific info:
-- apt-config dump --
APT ;
APT::Architecture m68k;
APT::Build-Essential ;
APT::Build-Essential:: build-essential;
APT::Install-Recommends 0;
APT::Install-Suggests 0;
APT::NeverAutoRemove ;
APT::NeverAutoRemove:: ^firmware-linux.*;
APT::NeverAutoRemove:: ^linux-firmware$;
APT::NeverAutoRemove:: ^linux-image-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-image-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^linux-headers-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-headers-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^linux-image-extra-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-image-extra-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^linux-signed-image-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-signed-image-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^kfreebsd-image-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^kfreebsd-image-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^kfreebsd-headers-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^kfreebsd-headers-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^gnumach-image-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^gnumach-image-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^.*-modules-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^.*-modules-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^.*-kernel-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^.*-kernel-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^linux-backports-modules-.*-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-backports-modules-.*-3\.16-1-m68k$;
APT::NeverAutoRemove:: ^linux-tools-3\.14-2-m68k$;
APT::NeverAutoRemove:: ^linux-tools-3\.16-1-m68k$;
APT::VersionedKernelPackages ;
APT::VersionedKernelPackages:: linux-image;
APT::VersionedKernelPackages:: linux-headers;
APT::VersionedKernelPackages:: linux-image-extra;
APT::VersionedKernelPackages:: linux-signed-image;
APT::VersionedKernelPackages:: kfreebsd-image;
APT::VersionedKernelPackages:: kfreebsd-headers;
APT::VersionedKernelPackages:: gnumach-image;
APT::VersionedKernelPackages:: .*-modules;
APT::VersionedKernelPackages:: .*-kernel;
APT::VersionedKernelPackages:: linux-backports-modules-.*;
APT::VersionedKernelPackages:: linux-tools;
APT::Never-MarkAuto-Sections ;
APT::Never-MarkAuto-Sections:: metapackages;
APT::Never-MarkAuto-Sections:: restricted/metapackages;
APT::Never-MarkAuto-Sections:: universe/metapackages;
APT::Never-MarkAuto-Sections:: multiverse/metapackages;
APT::Never-MarkAuto-Sections:: oldlibs;
APT::Never-MarkAuto-Sections:: restricted/oldlibs;
APT::Never-MarkAuto-Sections:: universe/oldlibs;
Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
Package: apt Version: 1.0.9.1 Severity: minor Hi, Michael Vogt requested I file this as official bugreport: The recent revalidation changes lead to a simple apt-get update on my buildd machine (m68k VM with 198 MHz CPU) with only the incoming.d-p.o repository modified, but not unstable, taking: Fetched 156 kB in 8min 33s (304 B/s) The sources.list on that chroot is just: deb http://ftp.de.debian.org/debian-ports unstable main deb http://ftp.de.debian.org/debian-ports unreleased main deb-src http://ftp.de.debian.org/debian unstable main # plus the “incoming” repository With almost 200 MHz, this is one of the faster “machines”. Our regular buildds have 25, 33, 50 MHz. The cause for this is that even a not-modified now triggers reverification of all downloaded files. The visible effect is a process called “copy” taking 90% CPU for minutes, in top(1). Michael and I think that apt should, when the Release and Release.gpg (or InRelease, but we aren’t using them any more, IIRC?) files are not modified, just reverify these, and, if they pass, assume the other data on the disc is valid/verified (because the files only get moved out of partial/ once verified AFAIK, and because if an unmodified Release file passes reverification, it would have passed initial verification as well, normally). Michael also thinks apt could cache more information, such as the original hashes, to speed up things more. -- Package-specific info: -- apt-config dump -- APT ; APT::Architecture m68k; APT::Build-Essential ; APT::Build-Essential:: build-essential; APT::Install-Recommends 0; APT::Install-Suggests 0; APT::NeverAutoRemove ; APT::NeverAutoRemove:: ^firmware-linux.*; APT::NeverAutoRemove:: ^linux-firmware$; APT::NeverAutoRemove:: ^linux-image-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-image-3\.16-1-m68k$; APT::NeverAutoRemove:: ^linux-headers-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-headers-3\.16-1-m68k$; APT::NeverAutoRemove:: ^linux-image-extra-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-image-extra-3\.16-1-m68k$; APT::NeverAutoRemove:: ^linux-signed-image-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-signed-image-3\.16-1-m68k$; APT::NeverAutoRemove:: ^kfreebsd-image-3\.14-2-m68k$; APT::NeverAutoRemove:: ^kfreebsd-image-3\.16-1-m68k$; APT::NeverAutoRemove:: ^kfreebsd-headers-3\.14-2-m68k$; APT::NeverAutoRemove:: ^kfreebsd-headers-3\.16-1-m68k$; APT::NeverAutoRemove:: ^gnumach-image-3\.14-2-m68k$; APT::NeverAutoRemove:: ^gnumach-image-3\.16-1-m68k$; APT::NeverAutoRemove:: ^.*-modules-3\.14-2-m68k$; APT::NeverAutoRemove:: ^.*-modules-3\.16-1-m68k$; APT::NeverAutoRemove:: ^.*-kernel-3\.14-2-m68k$; APT::NeverAutoRemove:: ^.*-kernel-3\.16-1-m68k$; APT::NeverAutoRemove:: ^linux-backports-modules-.*-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-backports-modules-.*-3\.16-1-m68k$; APT::NeverAutoRemove:: ^linux-tools-3\.14-2-m68k$; APT::NeverAutoRemove:: ^linux-tools-3\.16-1-m68k$; APT::VersionedKernelPackages ; APT::VersionedKernelPackages:: linux-image; APT::VersionedKernelPackages:: linux-headers; APT::VersionedKernelPackages:: linux-image-extra; APT::VersionedKernelPackages:: linux-signed-image; APT::VersionedKernelPackages:: kfreebsd-image; APT::VersionedKernelPackages:: kfreebsd-headers; APT::VersionedKernelPackages:: gnumach-image; APT::VersionedKernelPackages:: .*-modules; APT::VersionedKernelPackages:: .*-kernel; APT::VersionedKernelPackages:: linux-backports-modules-.*; APT::VersionedKernelPackages:: linux-tools; APT::Never-MarkAuto-Sections ; APT::Never-MarkAuto-Sections:: metapackages; APT::Never-MarkAuto-Sections:: restricted/metapackages; APT::Never-MarkAuto-Sections:: universe/metapackages; APT::Never-MarkAuto-Sections:: multiverse/metapackages; APT::Never-MarkAuto-Sections:: oldlibs; APT::Never-MarkAuto-Sections:: restricted/oldlibs; APT::Never-MarkAuto-Sections:: universe/oldlibs; APT::Never-MarkAuto-Sections:: multiverse/oldlibs; APT::Get ; APT::Get::AllowUnauthenticated false; APT::Architectures ; APT::Architectures:: m68k; APT::Compressor ; APT::Compressor::. ; APT::Compressor::.::Name .; APT::Compressor::.::Extension ; APT::Compressor::.::Binary ; APT::Compressor::.::Cost 1; APT::Compressor::gzip ; APT::Compressor::gzip::Name gzip; APT::Compressor::gzip::Extension .gz; APT::Compressor::gzip::Binary gzip; APT::Compressor::gzip::Cost 2; APT::Compressor::gzip::CompressArg ; APT::Compressor::gzip::CompressArg:: -9n; APT::Compressor::gzip::UncompressArg ; APT::Compressor::gzip::UncompressArg:: -d; APT::Compressor::bzip2 ; APT::Compressor::bzip2::Name bzip2; APT::Compressor::bzip2::Extension .bz2; APT::Compressor::bzip2::Binary bzip2; APT::Compressor::bzip2::Cost 3; APT::Compressor::bzip2::CompressArg ; APT::Compressor::bzip2::CompressArg:: -9; APT::Compressor::bzip2::UncompressArg ; APT::Compressor::bzip2::UncompressArg:: -d; APT::Compressor::xz ; APT::Compressor::xz::Name xz; APT::Compressor::xz::Extension .xz; APT::Compressor::xz::Binary xz; APT::Compressor::xz::Cost 4; APT::Compressor::xz::CompressArg ;
