On Thu, 16 Jan 2020 at 08:50:25 -0800, Daniel Schepler wrote: > I've been running a manual test bootstrap of Debian (starting with > cross-compiled packages amd64 -> i386 up to the point I was able to > install debhelper), and posting a few bugs I've found along the way. > These are where I found that having extra packages installed during > the dpkg-buildpackage run either failed or resulted in broken > packages. (Some examples of the type of thing I mean: #948522, > #887902.) > > However, I've been getting push back on some of these, with > maintainers of the opinion that it isn't actually a bug. So, I > thought I'd consult here to get more opinions on whether these are > true bugs, or whether I'm at fault for trying to run dpkg-buildpackage > manually instead of using it through pbuilder or sbuild.
I would say it's often a bug, but usually (always?) a non-RC bug, because there's a straightforward workaround (build in a minimal environment). Opening bugs as RC tends to put maintainers on the defensive (if nothing else because RC bugs have automatic consequences, like autoremovals), so it's often better to err on the side of opening bugs with a lower severity (and escalating if there's later consensus that this should happen) rather than starting with a severity that could be seen as unjustified. FTBFS and broken packages when built in a non-minimal environment are special cases of the more general problem of getting a different package when built in a non-minimal environment, which is sometimes trivial to solve, but somewhat intractable to detect and solve in general, because there are so many different non-minimal environments - as Ansgar mentioned, not just packages that we can capture in Build-Depends and Build-Conflicts, but also the contents of directories like /usr/local, /opt and ~/bin, and various properties of your system such as whether it has undergone the /usr merge, whether alternatives have been reconfigured, the filesystem you're running on, whether an LSM is in use, ... I think it's best-practice for package maintainers to explicitly disable optional features that are autodetected but aren't wanted in the package (./configure --disable-weasels --without-libstoat, or equivalent), but this isn't always possible without relatively extensive patches to the upstream build system, which themselves carry a cost. > The mere existence of the Build-Conflicts field [implies that we are > meant to use it] Yes to a point, but here's a somewhat devil's-advocate response to that: Suppose we lived in a parallel timeline where there is a serious QA effort put into making sure packages produce the same results, regardless of how they are built, as long as their Build-Depends and Build-Conflicts (and their -Arch, -Indep flavours) are satisfied. In that world, one rational thing for maintainers of packages with optional features to do would be to add an ever-increasing number of Build-Conflicts (every time someone notices a package that can alter or breaks the build). I could easily imagine it becoming no longer feasible to build large sets of packages unless you use sbuild or equivalent, because at least a few of those packages will have a Build-Conflicts on something you needed for another of the packages - which might not even be necessary any more, but as an individual package maintainer, there's little incentive to *remove* Build-Conflicts on things that you yourself don't use, particularly if insufficient Build-Conflicts are considered RC. So, pragmatically, it might be a good idea to optimize away the steps where we put lots of effort into QA, report lots of bugs, get them fixed by addition of Build-Conflicts, and find ourselves having to use a minimal build environment to get your desired set of packages built - by using a minimal build environment to start with. And we're back to the advice to use pbuilder or sbuild. I don't think it's coincidence that there has been so much parallel evolution between Debian's pbuilder and sbuild (among others), Fedora's mock, openSUSE's obs-build, Flatpak apps' flatpak-builder, and lots of projects' Docker-based build systems - getting a reliable and reproducible build of a complex package/module in minimal environments is already quite difficult, which makes the harder task of making it reliably reproducible in arbitrary non-minimal environments look rather unappealing. > It would be very difficult to impossible to test for every possible > combination of packages that satisfy the Build-Depends. (Though I > would think a vast majority of such bugs would be detected by a > reproducible-builds type setup with one build being the standard > minimal chroot, and the other build using a chroot with as many > packages as possible installed.) I think I dimly remember someone setting up "the buildd from hell" which deliberately did this as a QA mechanism, but it doesn't seem to have continued in any systematic way. > requiring use of pbuilder or sbuild would place a > burden on users who previously would have made local patches by a > sequence of "apt-get source package; cd package-*/; edit; > dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb I think in an ideal world, we'd have better tools for those users to build modified packages in a way that more closely resembles what happens on the production Debian infrastructure - which might for instance mean CI services, or my vectis[1] tool, or sbuild-debian-developer-setup, or even autopkgtest (which is really for testing packages, but as a side-effect, it knows how to build packages in an environment that in practice is going to be close-to-minimal). pbuilder and the usual sbuild+schroot setup have the disadvantage of requiring root privileges and crossing privilege boundaries, but vectis uses virtual machines (in practice this means kvm group membership or udev/logind uaccess, to get write access to /dev/kvm) and as namespace/container stuff gets more powerful and more trusted, I'd hope that we'll get a better ability to install build-dependencies and do builds in unprivileged containers. A justification for non-minimal build environments that I'm personally more interested in is debugging, whether by users developing local patches or by a package's maintainer. For successful, one-off builds, throwing a .dsc or source directory at a build service and waiting for a .deb to fall out is sufficient - but when the package isn't building correctly, it's often useful to be able to try it interactively and iteratively (without doing a full rebuild every time), in a non-minimal environment that contains debugging tools. In this situation, you'll often end up with a package that you are not entirely confident is reproducible, and the next thing to do after your patch seems ready to propose is to rebuild the package "properly" to confirm that it still works. Re-bootstrapping seems a lot more like the "one-off build" use-case to me, so I would recommend something sbuild-like. smcv [1] https://salsa.debian.org/smcv/vectis