Your message dated Fri, 24 Jun 2011 18:47:10 +0000
with message-id <[email protected]>
and subject line Bug#627737: fixed in debhelper 8.9.0
has caused the Debian Bug report #627737,
regarding dh_movefiles too slow under certain circumstances
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
627737: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=627737
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: debhelper
Version: 7.0.15
Tag: patch
Hi,
I've recently encountered a weird situation. Routine rebuilding
of Debian packages during daily development started taking
enormous amounts of time compared to past behaviour.
Debugging narrowed the slowdown down to calling `rm -f' from
within dh_movefiles via xargs.
(Now, please do not question usage of dh_movefiles instead of
dh_install, that's a separate topic.)
The part of dh_movefiles that is responsible for removal of
files after copying them does that in a very inefficient way.
The approach tracks back to a patch submitted by Yann Dirson
with the bug report #233226. The goal was noble, the solution
was... inferior (other expletives suppressed).
tr \\n \\0 < movelist | xargs -0 -i rm -f '{}'
A separate pair of shell and `rm' processes gets spawned per
individual file to be removed. The `tr' is redundant, but
insignificant.
I've researched the problem in a Lenny environment, but the same
code is present in the latest debhelper 8.1.6 (wheezy). It may
have been going unnoticed for many years, perhaps adding to the
worldwide resistance to Moore's law. Indeed, fast modern
computers may have masked the inefficiency of this code. In my
case, a certain set of circumstances had to align:
- ~1300 files under dh_movefiles attention in one package;
- building within a networked AFS filespace;
- other processes taking much CPU in parallel (it was this
factor that had to grow over a certain threshold to
avalanche the problem).
As a result, the removal of ~1300 files was taking up to ~150
seconds. Removal of the same files on a local disk took up to
~10 seconds under the same other circumstances, which still is
too much comparatively. Hence, the AFS aspect of the
environment wasn't decisive, it merely added a factor of about
15 to the worst case (much less slowdown normally).
Here's a trivial fix to the dh_movefiles:
--- /usr/bin/dh_movefiles~ 2011-05-24 14:17:37.000000000 +1200
+++ /usr/bin/dh_movefiles 2011-05-24 14:20:06.000000000 +1200
@@ -145,7 +145,7 @@
complex_doit("(cd $sourcedir >/dev/null ; tar --create
--files-from=$pwd/debian/movelist --file -) | (cd $tmp >/dev/null ;tar xpf -)");
# --remove-files is not used above because tar then doesn't
# preserve hard links
- complex_doit("(cd $sourcedir >/dev/null ; tr '\\n' '\\0' <
$pwd/debian/movelist | xargs -0 -i rm -f '{}')");
+ complex_doit("(cd $sourcedir >/dev/null ; <
$pwd/debian/movelist xargs -rd'\\n' rm -f)");
doit("rm","-f","debian/movelist");
}
}
Minimal number of processes spawn. The effect is removal of the
same ~1300 files within ~0.6 seconds (speedup factor of 250)
from AFS filespace and within ~0.15 seconds from local disk
(speedup factor of 67) under the same other circumstances.
Below I put another trivial patch, following the one above.
--- /usr/bin/dh_movefiles~ 2011-05-24 15:18:50.000000000 +1200
+++ /usr/bin/dh_movefiles 2011-05-24 15:32:19.000000000 +1200
@@ -142,10 +142,10 @@
}
my $pwd=`pwd`;
chomp $pwd;
- complex_doit("(cd $sourcedir >/dev/null ; tar --create
--files-from=$pwd/debian/movelist --file -) | (cd $tmp >/dev/null ;tar xpf -)");
+ complex_doit("tar c -C $sourcedir -T $pwd/debian/movelist | tar
xp -C $tmp");
# --remove-files is not used above because tar then doesn't
# preserve hard links
- complex_doit("(cd $sourcedir >/dev/null ; <
$pwd/debian/movelist xargs -rd'\\n' rm -f)");
+ complex_doit("cd $sourcedir && < $pwd/debian/movelist xargs
-rd'\\n' rm -f");
doit("rm","-f","debian/movelist");
}
}
It doesn't have to to with the main problem I'm reporting, but
makes further enhancements to the same area of dh_movefiles:
- spawns no redundant shell copies;
- removes useless `tar' options and abridges useful ones;
- does away with useless suppression of stdout of `cd'
builtin (it doesn't output anything to stdout);
- doesn't proceed to copy or remove any files if changing to
the prescribed directory fails for any reason (currently
it's possible to litter the filespace or remove important
files inadvertently).
Feel free to use just the first one or both.
Cheers,
--
/Dennis Vshivkov <[email protected]>
--- End Message ---
--- Begin Message ---
Source: debhelper
Source-Version: 8.9.0
We believe that the bug you reported is fixed in the latest version of
debhelper, which is due to be installed in the Debian FTP archive:
debhelper_8.9.0.dsc
to main/d/debhelper/debhelper_8.9.0.dsc
debhelper_8.9.0.tar.gz
to main/d/debhelper/debhelper_8.9.0.tar.gz
debhelper_8.9.0_all.deb
to main/d/debhelper/debhelper_8.9.0_all.deb
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Joey Hess <[email protected]> (supplier of updated debhelper package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Fri, 24 Jun 2011 14:28:52 -0400
Source: debhelper
Binary: debhelper
Architecture: source all
Version: 8.9.0
Distribution: unstable
Urgency: low
Maintainer: Joey Hess <[email protected]>
Changed-By: Joey Hess <[email protected]>
Description:
debhelper - helper programs for debian/rules
Closes: 541458 544844 627534 627737 628053 629139 630826
Changes:
debhelper (8.9.0) unstable; urgency=low
.
* dh: In v9, any standard rules file targets, including build-arch,
build-indep, build, install, etc, can be defined in debian/rules
without needing to explicitly tell make the dependencies between
the targets. Closes: #629139
(Thanks, Roger Leigh)
* dh_auto_configure: In v9, does not include the source package name
in --libexecdir when using autoconf. Closes: #541458
* dh_auto_build, dh_auto_configure, dh: Set environment variables
listed by dpkg-buildflags --export. Any environment variables that
are already set to other values will not be changed.
Closes: #544844
* dh_movefiles: Optimise use of xargs. Closes: #627737
* Correct docs about multiarch and v9. Closes: #630826
* Fix example. Closes: #627534
* Fix error message. Closes: #628053
* dh_auto_configure: If there is a problem with cmake, display
the CMakeCache.txt.
Checksums-Sha1:
04fc16f2247506059765f2031a7982269705020a 1568 debhelper_8.9.0.dsc
a24326c4c2c0984783be408a90e857375f399c3a 383819 debhelper_8.9.0.tar.gz
ca13253bfa8a960fd628a3fd3921eb679cba2d9d 559388 debhelper_8.9.0_all.deb
Checksums-Sha256:
d6fb5dfeaacadc5574c619560d950a623261ad229848a2bf1e02bad7ae89939b 1568
debhelper_8.9.0.dsc
c9a1c2ceb4fcccd794199d738314f52cce9248ef24f360a1a101622df0b766c2 383819
debhelper_8.9.0.tar.gz
35767a7c9937d103fddfcc7f810667509c26797150704a5f9ac511770d1fbf1f 559388
debhelper_8.9.0_all.deb
Files:
0dc2ec18b832d0aad04f680427d6d7c6 1568 devel optional debhelper_8.9.0.dsc
0b3dbca729105330453f05498b71c288 383819 devel optional debhelper_8.9.0.tar.gz
e4ccb25e81e935053d9aed5c0a9a66d2 559388 devel optional debhelper_8.9.0_all.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iQIVAwUBTgTYK8kQ2SIlEuPHAQhASA/8DAvEwz+nTdaJvJSZzpwqCrWYmClBtbdb
qdn0xbDBgGwHjbchoTGMX07HgWJWEC0SOBnGrtD8krUfHpOFJML+jgnsBjTEg5KW
A293CYxdHcXgWyhEEpbUbD++hmWlF2F4tWU1hWRg2y3pFf8sY1b7TIfv7RdETpsJ
h2FXW3a1EO1zCdc0EJPmcN3bZg+++ZKxjOLp9p9i1xwwd8sC2tz4cCkxdkNFLWsD
FZ+NQrDQLa4AT9s7lFUvb1VEaL/H7OXGioKP97eoj91DKUumMBmMuSTkEi5PLjGT
IBIwzdEdgwtdUi+4lnuz+dr7+Fbm7TVA6RyP6NLrLUXfCIO15vXyUdbwaNTJNnS9
xtH21+HWGq5W898CQcYXHegBiUQtt4vyfQF2N36UMTHPreih92P6ipnGtAWSup2P
jXAK2bDUEeyjbdHwR5CgfodCzNMVravJRuv/Z2vs91kwewSWKDNG9ctcjbhnbvTp
D/1GbFdVZd9OFfU+NXL8KpJwCmZOAHwo/E5nS373zb59KTz2287l0xx+TUJo5j7A
bclLjMZYUBnsMptaESozaEz4m9W8CN0TumSy/jqZBuMvTKRoJzP67IQv1RgYkFnc
ecI3a+a96fek6coyXpmpoIm5u09YuhlRpdQV9U1gigCD1dkCohhm3ITWiowrTqbw
CGqB7k2E8h8=
=/n9E
-----END PGP SIGNATURE-----
--- End Message ---
