Hi, here are my observations after analyzing a few pbuilder --twice failures in experimental. I only did full (source+arch+indep) builds.
The command sequence being tested is roughly: 1. debian/rules clean 2. dpkg-source 3. debian/rules build binary 4. debian/rules clean 5. dpkg-source 6. debian/rules build binary 1.-3. are utilized by regular builds and tested by normal archive-wide rebuilds, therefore I'll only consider packages passing these steps. Possible failure scenarios for the second build: 4. debian/rules clean There are bugs related to make distclean and debian/rules clean, they usually don't show up during 1. since the source tree is not in a configured state (no Makefile?) and the debian/ tree is not in a built state. a) make distclean may fail, probably an upstream bug * #884898, icu/experimental, parallel distclean race b) passes, but leaves new/modified files in the source tree * This is the majority of bugs, usually generated filed don't get deleted properly. The subsequent error from dpkg-source is pretty obvious. c) passes, but leaves new/modified in the debian/ tree * #884419, #884815: override_dh_clean does not run dh_clean resulting in debian/$PKG/ package build directories and debhelper logfiles being left in the tree. Subsequent dpkg-source will choke on unknown binary files under debian/ There is a lintian check coming: #884817 "lintian: check for override_dh_clean target missing call to dh_clean". * there may be text-only changes in debian/ that don't trigger errors, but cause differing source package to be built in 2./5. d) passes, but deletes files that cannot be regenerated * #884706, #884813, #884819, #884889 This passes the subsequent dpkg-source (since it will just warn on deleted files by default, which is very convient to get rid of generated files that are shipped in the upstream tarball), but will fail during the following build 5. dpkg-source a) may fail as a consequence of 4.b), 4.c) b) may build a source package that differs from the one built in 2. Testing for these bugs easily may need some new features in pbuilder, e.g. a --one-and-a-half option :-) A possible candidate is git/experimental 1:2.15.1+next.20171214-1 which should add a 'debian/git-core/usr/share/doc/git-core -> git' symlink (#884890). 6. debian/rules build binary a) may fail as a consequence of 4.d) How could we check for these bugs? I think the interesting failures to look for are 4.a), 5.b), 6.a) (leaving out the probabe majority 5.a)), since they may make working with the packages difficult by failing in non-obvious ways. I would also consider these as RC. We would probably need some enhancements for pbuilder to allow testing points 4. and 5. without doing a full --twice run. Also I don't know whether it is easily possible to classify pbuilder --twice failures to the command that failed and whether it was first or second build. Given proper pbuilder support (deliver build result from the first build even if 4. debian/rules clean failed), testing for 4.a) should be trivially possible to do for reproducible builds with only marginal computation time requirements. Does the reproducible build effort currently test for reproducibility of source packages? In that case testing for 5.b) could hopefully be done with reproducible builds as well, it would just require another dpkg-source call. Failures for 5.a) could be filtered out unless someone volunteers for the bug filing. Testing for 6.a) effectively doubles the compute power needed for a rebuild test, so maybe that should be rather left to infrequent specialized archive-wide rebuilds. Andreas