Re: Is running dpkg-buildpackage manually from the command line forbidden?
(Disclaimer: This is a xkcd:386-like response to this subthread) > Here's the current list of these packages on my system: > > $ aptitude -F '%p' search '~prequired !~E' The list omits 'apt' as a libapt internally flags it as essential to grant it the utmost protection by all clients along with its (due to that) pseudo-essential dependencies both in terms of user actions as well as (re)solver and installation ordering algorithms. So, to see the "real" list, you need something like: $ aptitude -F '%p' search '~prequired !~E' -o pkgCacheGen::ForceEssential=',' -o Dir::Cache=/dev/null (The second -o is needed to prevent libapt from using its binary caches and forces it to reparse everything in memory; the first -o is the knob defaulting to 'apt' if unset. And yes, it is really ',' and you probably don't want to know why and just accept it as meaning empty list) Now, suggesting that apt is not an integral part of a Debian system and could henceforth be removed is of course heresy! The only thing saving you vile heretics is apts heavy involvement in the creation of these chroots. Best regards David Kalnischkies signature.asc Description: PGP signature
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Fri, Jan 17, 2020 at 03:02:06AM +, Paul Wise wrote: > Personally, I think there is value in Daniel's work on bootstrapping > Debian from other operating systems and Helmut's work on bootstrapping > Debian from existing Debian architectures. Both are important projects > and we need Debian and FLOSS in general to be more bootstrappable than > it is now [...] indeed! > At some point I'd even like to see Debian > bootstrapped from whatever comes out of the the Bootstrappable Builds > project: > > https://bootstrappable.org/ Vagrant (cc:ed) was working on this at the last reproducible builds summit. -- cheers, Holger --- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C signature.asc Description: PGP signature
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Fri, 2020-01-17 at 10:58 +0100, Johannes Schauer wrote: > Quoting Simon McVittie (2020-01-16 19:47:02) > > I think I dimly remember someone setting up "the buildd from hell" which > > deliberately did this as a QA mechanism, but it doesn't seem to have > > continued in any systematic way. > > is there more information about it somewhere? My inbox has only two emails > from > xnox and pabs (in CC) about it. How did it work? I found a reference to it on the wiki: https://wiki.debian.org/qa.debian.org/ArchiveTesting#TODO Some discussions about it that I found: Subject: rebuilding the archive in a dirty chroot: results <20080125142515.ga18...@xanadu.blop.info> Subject: Meaning of the "Altering package upload rules" <20080216130228.ga32...@pcpool00.mathematik.uni-freiburg.de> <20080216181247.GA13475@garfield> Subject: Use of the Build-Conflicts field Subject: Re: Please drop anacron from task-desktop -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Fri, Jan 17, 2020 at 2:05 AM Johannes Schauer wrote:
> yes, probably a communication problem. I think you are talking about [1] from
> August 11 last year? I replied the same day, telling you to please file a bug
> with your patches to continue discussion there. But then there was no response
> from you anymore and I don't find your bug in the BTS. Maybe my mail got lost
> somehow or I fail finding the bug you filed?
It seems the reply must have gotten lost somehow - I don't see it anywhere.
> I'm also very excited about having yet another chroot backend being integrated
> into sbuild! Though my first comment would be the same as I gave those asking
> for a docker backend in #867176: maybe try adding that backend to autopkgtest
> first. Then more people would profit from having that type of backend
> available
> and no additional changes would be needed in sbuild because sbuild can already
> use autopkgtest backends for package building.
OK, I'll try to get the patch submitted either tonight or tomorrow.
(What I need to clean up is that it's interspersed with changes I made
to have it run with a personal distribution build I've been tinkering
with. On a quick review, I now notice it's also interspersed with
changes to support using eatmydata only on the apt install commands
and only if not in schroot-update mode, instead of having to put it
into schroot config to apply to all commands - which seems reasonable
to split out into a separate patch to submit. I also haven't yet
updated documentation files.)
One quick question: I don't see any mention of
$options->{'DISABLE_NETWORK'} in lib/Sbuild/ChrootAutopkgtest.pm,
whereas my new lib/Sbuild/ChrootNspawn.pm does support it. If I'm not
missing something, then I wonder what it would take to add
DISABLE_NETWORK support to the autopkgtest sbuild engine.
--
Daniel Schepler
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi,
Quoting Daniel Schepler (2020-01-17 17:58:09)
> > I'm also very excited about having yet another chroot backend being
> > integrated into sbuild! Though my first comment would be the same as I gave
> > those asking for a docker backend in #867176: maybe try adding that backend
> > to autopkgtest first. Then more people would profit from having that type
> > of backend available and no additional changes would be needed in sbuild
> > because sbuild can already use autopkgtest backends for package building.
> OK, I'll try to get the patch submitted either tonight or tomorrow.
thank you!! :)
> One quick question: I don't see any mention of $options->{'DISABLE_NETWORK'}
> in lib/Sbuild/ChrootAutopkgtest.pm, whereas my new lib/Sbuild/ChrootNspawn.pm
> does support it. If I'm not missing something, then I wonder what it would
> take to add DISABLE_NETWORK support to the autopkgtest sbuild engine.
it would require some way to tell the autopkgtest backends to disable network.
To my knowledge that is not possible. So as of today, the only backend where
you can disable network for the build (not the apt install) is the "unshare"
backend.
More info here:
https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage
Thanks!
cheers, josch
signature.asc
Description: signature
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi Daniel, On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote: > However, I've been getting push back on some of these, with > maintainers of the opinion that it isn't actually a bug. So, I > thought I'd consult here to get more opinions on whether these are > true bugs, or whether I'm at fault for trying to run dpkg-buildpackage > manually instead of using it through pbuilder or sbuild. Given that I'm also bootstrapping Debian (in a different setting), I'm running into much the same problems. However, when I file bugs about build failures in non-standard environments, I don't receive the push-back that you describe. I cannot reproduce your experience. It's a rare thing for maintainers to tell me that I'm filing a non-bug and I do file a *lot* of bugs. Could it be that your sample is very small or biased? Maybe you can give examples? In any case, the consensus seems to be that FTBFS in a non-standard environment is a bug, but not an RC one. And in general, you get to discuss less with maintainers if your bug includes a patch. ;) Helmut
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi Daniel, On Thu, Jan 16, 2020 at 12:04:12PM -0800, Daniel Schepler wrote: > Also, by the way, the amd64 -> i386 cross built core packages largely > worked OK, with the exception of gcc-9, which ended up with incorrect > include-fixed/limits.h, and with a compiler that required -lssp when > building with -fstack-protector-strong or -fstack-protector as almost > all Debian packages do. To anybody on the list: is there something I > did wrong with the cross build there, which was to run > "dpkg-buildpackage -a i386 -B -Pcross"?) This sounds a lot like you're running into https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80677. Cross building gcc tends to not work in my experience, see: http://crossqa.debian.net/src/gcc-8 http://crossqa.debian.net/src/gcc-9 For gcc-9, the --host flag is not properly passed down to the gm2 component. This is reported as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92336. I think the expectation that it works is misguided. Adding DEB_BUILD_OPTIONS=nocheck might increase your chances of a successful build. Quite certainly, more work is needed here. Unfortunately, we sorely lack porters and everyone seems to expect that things just work without anyone doing the work. Helmut
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Fri, 2020-01-17 at 11:12:50 +0100, Ansgar wrote: > Johannes Schauer writes: > > My advice would also be to switch from debootstrap to mmdebstrap because the > > latter is able to create a chroot with only Essential:yes packages in it > > while > > debootstrap includes Priority:required packages as well. Alternatively, > > debootstrap could gain a variant only installing Essential:yes packages and > > their dependencies (why doesn't it have that already?). > > Debian doesn't support systems that don't have "required" packages > installed. So buildds should have them installed. If these statements are based on the policy quote below, then I do not agree. I don't see why e2fsprogs would need to be installed on a chroot, as long as there's nothing depending on it, for example. > Policy states: > "Removing a required package may cause your system to become totally > broken and you may not even be able to use dpkg to put things back, so > only do so if you know what you are doing." That seems wrong, or inaccurate at best. dpkg should never depend on anything that is not part of the pseudo-essential set (strictly speaking only Essential:yes + awk-virtual), or that it depends on explicitly. So as long as a package has not been forced out, dpkg must work. Removing a required package (that is not Essential:yes) might indeed render your system broken, but what broken means or in what context is not qualified there. This could apply to systems that get booted for example, but not to chroots. A package that relies on another package that is Priority:required and not Essential:yes, and that it does not depend on, is just broken. Here's the current list of these packages on my system: $ aptitude -F '%p' search '~prequired !~E' debconf e2fsprogs gcc-10-base gcc-9-base libpam-modules libpam-modules-bin libpam-runtime mawk mount passwd tzdata And most of these are actually part of the pseudo-essential set anyway, and cannot be removed w/o force. That passage in policy might have made sense some time ago, when Priority:required almost matched the pseudo-essential set, and when we didn't have a separation of "dpkg-essential" and "boot-essential". Regards, Guillem
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi, On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote: > I've been running a manual test bootstrap of Debian (starting with > cross-compiled packages amd64 -> i386 up to the point I was able to > install debhelper), and posting a few bugs I've found along the way. > These are where I found that having extra packages installed during > the dpkg-buildpackage run either failed or resulted in broken > packages. (Some examples of the type of thing I mean: #948522, > #887902.) I would expect a sensible package build to not depend on package relationships that aren't declared, in either way, but things in /usr/local are outside the control of the package system. So a package that enables an optional feature if a particular other package is installed, but there is neither a build dependency nor a build conflict declared is a bug to me. The Build-Conflicts should still be avoided, by an explicit configure option "disable this feature". Simon
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Johannes Schauer writes: > My advice would also be to switch from debootstrap to mmdebstrap because the > latter is able to create a chroot with only Essential:yes packages in it while > debootstrap includes Priority:required packages as well. Alternatively, > debootstrap could gain a variant only installing Essential:yes packages and > their dependencies (why doesn't it have that already?). Debian doesn't support systems that don't have "required" packages installed. So buildds should have them installed. Policy states: "Removing a required package may cause your system to become totally broken and you may not even be able to use dpkg to put things back, so only do so if you know what you are doing." "Essential" packages just have additional requirements (in particular essential packages must work even in the "unpacked" state). Ansgar
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi Daniel, Sam and all, Quoting Sam Hartman (2020-01-17 01:27:52) > > "Daniel" == Daniel Schepler writes: > > Daniel> (Incidentally, another offshoot was creating local patches to > sbuild > Daniel> which add an operation mode using systemd-nspawn --ephemeral to > start > Daniel> a container (along with the base being a BTRFS subvolume to speed > up > Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx > Daniel> [--property=PrivateNetwork=yes] cmd..., etc. When I sent a > message to > Daniel> sbu...@packages.debian.org there didn't seem to be any interest in > Daniel> having me clean up the patches and send them on. > > This sounds like a communication problem or something. Having > systemd-nspawn container support in sbuild would be really helpful. > Would you be willing to try reaching out to the sbuild maintainers > again? If you don't get an answer, in a month or so, please reach out > to me as DPL. My job would not to be to convince the maintainer to > accept your patches, but rather to facilitate communication so you can > get a clear answer. The DPL does that from time to time when things seem to > have broken down. yes, probably a communication problem. I think you are talking about [1] from August 11 last year? I replied the same day, telling you to please file a bug with your patches to continue discussion there. But then there was no response from you anymore and I don't find your bug in the BTS. Maybe my mail got lost somehow or I fail finding the bug you filed? I'm also very excited about having yet another chroot backend being integrated into sbuild! Though my first comment would be the same as I gave those asking for a docker backend in #867176: maybe try adding that backend to autopkgtest first. Then more people would profit from having that type of backend available and no additional changes would be needed in sbuild because sbuild can already use autopkgtest backends for package building. Let me also use this opportunity to ask for help with maintaining sbuild. If any of you reading this ever felt that sbuild was the thing that is worth your time, please feel free to reach out to me. We need to make a new release fixing some easy but important bugs that accumulated in the BTS. I would be very happy to review and apply people's patches. :) Thanks! cheers, josch [1] cadf0c45pjydq+hmqetg6atdqp8ojw8cr3z1kz2ktu9z3ua3...@mail.gmail.com signature.asc Description: signature
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Hi, Quoting Simon McVittie (2020-01-16 19:47:02) > I think I dimly remember someone setting up "the buildd from hell" which > deliberately did this as a QA mechanism, but it doesn't seem to have > continued in any systematic way. is there more information about it somewhere? My inbox has only two emails from xnox and pabs (in CC) about it. How did it work? > I think in an ideal world, we'd have better tools for those users to build > modified packages in a way that more closely resembles what happens on the > production Debian infrastructure - which might for instance mean CI services, > or my vectis[1] tool, or sbuild-debian-developer-setup, or even autopkgtest > (which is really for testing packages, but as a side-effect, it knows how to > build packages in an environment that in practice is going to be > close-to-minimal). My advice would also be to switch from debootstrap to mmdebstrap because the latter is able to create a chroot with only Essential:yes packages in it while debootstrap includes Priority:required packages as well. Alternatively, debootstrap could gain a variant only installing Essential:yes packages and their dependencies (why doesn't it have that already?). > pbuilder and the usual sbuild+schroot setup have the disadvantage of > requiring root privileges and crossing privilege boundaries, but vectis uses > virtual machines (in practice this means kvm group membership or udev/logind > uaccess, to get write access to /dev/kvm) and as namespace/container stuff > gets more powerful and more trusted, I'd hope that we'll get a better ability > to install build-dependencies and do builds in unprivileged containers. That can be done with sbuild. With --chroot-mode=unshare, sbuild looks for rootfs tarballs in ~/.cache/sbuild or you can give it your own tarball with the --chroot option. Since you can create rootfs tarballs without sudo using mmdebstrap, you can setup arbitrary chroots and build packages without any process running with root privileges. If you don't like the security implications of unshared user namespaces, then you might want to try --chroot-mode=autopkgtest and --autopkgtest-virt-server=qemu. Using --autopkgtest-virt-server-opts you can then supply any qemu compatible system image. Using mmdebstrap with fakechroot mode and guestfish you can create these system images without becoming root and without having to enable kernel.unprivileged_userns_clone. Or if lxc/lxd are the unprivileged containers you are talking about, then you can just use --autopkgtest-virt-server=lxc and let sbuild do builds in your existing lxc container. Thanks! cheers, josch signature.asc Description: signature
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Thu, Jan 16, 2020 at 7:18 PM Ondřej Surý wrote: > while your effort is valiant, I see a little value in it as there’s no real > world use case. While your arguments are valid, you are imposing additional > work on generally already overloaded maintainers with unclear goal and > purpose. Personally, I think there is value in Daniel's work on bootstrapping Debian from other operating systems and Helmut's work on bootstrapping Debian from existing Debian architectures. Both are important projects and we need Debian and FLOSS in general to be more bootstrappable than it is now, just like the Reproducible Builds folks made everything more reproducible. At some point I'd even like to see Debian bootstrapped from whatever comes out of the the Bootstrappable Builds project: https://bootstrappable.org/ -- bye, pabs https://wiki.debian.org/PaulWise
Re: Is running dpkg-buildpackage manually from the command line forbidden?
> "Daniel" == Daniel Schepler writes: Daniel> (Incidentally, another offshoot was creating local patches to sbuild Daniel> which add an operation mode using systemd-nspawn --ephemeral to start Daniel> a container (along with the base being a BTRFS subvolume to speed up Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx Daniel> [--property=PrivateNetwork=yes] cmd..., etc. When I sent a message to Daniel> sbu...@packages.debian.org there didn't seem to be any interest in Daniel> having me clean up the patches and send them on. This sounds like a communication problem or something. Having systemd-nspawn container support in sbuild would be really helpful. Would you be willing to try reaching out to the sbuild maintainers again? If you don't get an answer, in a month or so, please reach out to me as DPL. My job would not to be to convince the maintainer to accept your patches, but rather to facilitate communication so you can get a clear answer. The DPL does that from time to time when things seem to have broken down. --Sam
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Thu, Jan 16, 2020 at 11:18 AM Ondřej Surý wrote: > while your effort is valiant, I see a little value in it as there’s no real > world use case. While your arguments are valid, you are imposing additional > work on generally already overloaded maintainers with unclear goal and > purpose. > > Perhaps your energy and enthusiasm (which I appreciate) could be spent on > helping fixing reproducible builds in packages or cross-building. Those are > practical and you won’t find any resistance in accepting patches for these > two use cases. OK, makes sense. This was, in fact, an offshoot of the beginning stages of a project to create some way to automate bootstrapping an architecture starting with automated cross builds of the core packages. (Incidentally, another offshoot was creating local patches to sbuild which add an operation mode using systemd-nspawn --ephemeral to start a container (along with the base being a BTRFS subvolume to speed up the cloning), systemd-run -M debian-sid-amd64-xxx [--property=PrivateNetwork=yes] cmd..., etc. When I sent a message to sbu...@packages.debian.org there didn't seem to be any interest in having me clean up the patches and send them on. Still, do you think if I posted bug reports for issues I found due to the builds running under seccomp filters, as wishlist bugs and with either suggested patches or a request for advice on further debugging it myself where I got stuck, that maintainers might be willing to consider them? Also, by the way, the amd64 -> i386 cross built core packages largely worked OK, with the exception of gcc-9, which ended up with incorrect include-fixed/limits.h, and with a compiler that required -lssp when building with -fstack-protector-strong or -fstack-protector as almost all Debian packages do. To anybody on the list: is there something I did wrong with the cross build there, which was to run "dpkg-buildpackage -a i386 -B -Pcross"?) -- Daniel Schepler
Re: Is running dpkg-buildpackage manually from the command line forbidden?
Daniel, while your effort is valiant, I see a little value in it as there’s no real world use case. While your arguments are valid, you are imposing additional work on generally already overloaded maintainers with unclear goal and purpose. Perhaps your energy and enthusiasm (which I appreciate) could be spent on helping fixing reproducible builds in packages or cross-building. Those are practical and you won’t find any resistance in accepting patches for these two use cases. Ondrej -- Ondřej Surý > On 16 Jan 2020, at 18:21, Daniel Schepler wrote: > > I've been running a manual test bootstrap of Debian (starting with > cross-compiled packages amd64 -> i386 up to the point I was able to > install debhelper), and posting a few bugs I've found along the way. > These are where I found that having extra packages installed during > the dpkg-buildpackage run either failed or resulted in broken > packages. (Some examples of the type of thing I mean: #948522, > #887902.) > > However, I've been getting push back on some of these, with > maintainers of the opinion that it isn't actually a bug. So, I > thought I'd consult here to get more opinions on whether these are > true bugs, or whether I'm at fault for trying to run dpkg-buildpackage > manually instead of using it through pbuilder or sbuild. > > My arguments in favor of such things being bugs: > > 1. I've been using Debian since before pbuilder or sbuild even > existed, and I don't remember ever seeing any announcements along the > lines of "using pbuilder or sbuild is now mandatory, running > dpkg-buildpackage manually is forbidden". (Just announcements that of > course, testing package builds using one of those before uploading is > strongly encouraged.) > 2. The mere existence of the Build-Conflicts field. > 3. The general principle that the Build-Depends are meant not to > describe every possible way the package *could* be built, but to pin > down the exact environment in which the package *should* be built, in > order to avoid unnecessary differences in the resulting packages > between architectures. > 4. The build-essential package set could evolve over time, and in a > few cases that could come back to bite maintainers. For a > hypothetical example: what if Debian eventually decided to add cmake, > ninja, and meson to the build-essential set? Or, what if there were a > source package that made a build time check of whether a libpopt.so.2 > file exists to be dlopen()ed, but if it's found enabled broken code; > and then, eventually, one of the build-essential packages added a > dependency on libpopt? > > Possible arguments I can anticipate against such things being considered bugs: > > 1. It would be very difficult to impossible to test for every possible > combination of packages that satisfy the Build-Depends. (Though I > would think a vast majority of such bugs would be detected by a > reproducible-builds type setup with one build being the standard > minimal chroot, and the other build using a chroot with as many > packages as possible installed.) > 2. It would be pointless to worry about such things, especially now > that all uploads to the archives must be source only. (To which my > answer would be: requiring use of pbuilder or sbuild would place a > burden on users who previously would have made local patches by a > sequence of "apt-get source package; cd package-*/; edit; > dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb ) > > (Somewhat related to this: I've also found a few packages that hang > when building from the command line, waiting for input from stdin; > whereas in pbuilder or sbuild, with input redirected to /dev/null or > similar, the builds succeed. Would these be considered bugs as well? > Of course, in some situations it looks like it detects an > incontrovertible bug, such as when an "rm" command hangs on the prompt > for confirmation on a read-only file, and the /dev/null stdin case > would just result in those files being left in place. I've especially > been seeing the latter sort of thing related to Perl packages now that > recent Perls install lots of files as read only.) > -- > Daniel Schepler >
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Thu, 16 Jan 2020 at 08:50:25 -0800, Daniel Schepler wrote: > I've been running a manual test bootstrap of Debian (starting with > cross-compiled packages amd64 -> i386 up to the point I was able to > install debhelper), and posting a few bugs I've found along the way. > These are where I found that having extra packages installed during > the dpkg-buildpackage run either failed or resulted in broken > packages. (Some examples of the type of thing I mean: #948522, > #887902.) > > However, I've been getting push back on some of these, with > maintainers of the opinion that it isn't actually a bug. So, I > thought I'd consult here to get more opinions on whether these are > true bugs, or whether I'm at fault for trying to run dpkg-buildpackage > manually instead of using it through pbuilder or sbuild. I would say it's often a bug, but usually (always?) a non-RC bug, because there's a straightforward workaround (build in a minimal environment). Opening bugs as RC tends to put maintainers on the defensive (if nothing else because RC bugs have automatic consequences, like autoremovals), so it's often better to err on the side of opening bugs with a lower severity (and escalating if there's later consensus that this should happen) rather than starting with a severity that could be seen as unjustified. FTBFS and broken packages when built in a non-minimal environment are special cases of the more general problem of getting a different package when built in a non-minimal environment, which is sometimes trivial to solve, but somewhat intractable to detect and solve in general, because there are so many different non-minimal environments - as Ansgar mentioned, not just packages that we can capture in Build-Depends and Build-Conflicts, but also the contents of directories like /usr/local, /opt and ~/bin, and various properties of your system such as whether it has undergone the /usr merge, whether alternatives have been reconfigured, the filesystem you're running on, whether an LSM is in use, ... I think it's best-practice for package maintainers to explicitly disable optional features that are autodetected but aren't wanted in the package (./configure --disable-weasels --without-libstoat, or equivalent), but this isn't always possible without relatively extensive patches to the upstream build system, which themselves carry a cost. > The mere existence of the Build-Conflicts field [implies that we are > meant to use it] Yes to a point, but here's a somewhat devil's-advocate response to that: Suppose we lived in a parallel timeline where there is a serious QA effort put into making sure packages produce the same results, regardless of how they are built, as long as their Build-Depends and Build-Conflicts (and their -Arch, -Indep flavours) are satisfied. In that world, one rational thing for maintainers of packages with optional features to do would be to add an ever-increasing number of Build-Conflicts (every time someone notices a package that can alter or breaks the build). I could easily imagine it becoming no longer feasible to build large sets of packages unless you use sbuild or equivalent, because at least a few of those packages will have a Build-Conflicts on something you needed for another of the packages - which might not even be necessary any more, but as an individual package maintainer, there's little incentive to *remove* Build-Conflicts on things that you yourself don't use, particularly if insufficient Build-Conflicts are considered RC. So, pragmatically, it might be a good idea to optimize away the steps where we put lots of effort into QA, report lots of bugs, get them fixed by addition of Build-Conflicts, and find ourselves having to use a minimal build environment to get your desired set of packages built - by using a minimal build environment to start with. And we're back to the advice to use pbuilder or sbuild. I don't think it's coincidence that there has been so much parallel evolution between Debian's pbuilder and sbuild (among others), Fedora's mock, openSUSE's obs-build, Flatpak apps' flatpak-builder, and lots of projects' Docker-based build systems - getting a reliable and reproducible build of a complex package/module in minimal environments is already quite difficult, which makes the harder task of making it reliably reproducible in arbitrary non-minimal environments look rather unappealing. > It would be very difficult to impossible to test for every possible > combination of packages that satisfy the Build-Depends. (Though I > would think a vast majority of such bugs would be detected by a > reproducible-builds type setup with one build being the standard > minimal chroot, and the other build using a chroot with as many > packages as possible installed.) I think I dimly remember someone setting up "the buildd from hell" which deliberately did this as a QA mechanism, but it doesn't seem to have continued in any systematic way. > requiring use of pbuilder or sbuild
Re: Is running dpkg-buildpackage manually from the command line forbidden?
> "Daniel" == Daniel Schepler writes: Daniel> However, I've been getting push back on some of these, with Daniel> maintainers of the opinion that it isn't actually a bug. So, I Daniel> thought I'd consult here to get more opinions on whether these are Daniel> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage Daniel> manually instead of using it through pbuilder or sbuild. I think it is often a bug, but rarely if ever a serious bug. I'd say that a good test is whether you can produce a clean, simple patch. (adding build-conflicts seems like a fine answer in a lot of places). If something like build-conflicts is sufficient, then sure, file the bug with a patch if it actually annoys you. If the patch is going to be ugly, then perhaps it's not worth it.
Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Thu, 2020-01-16 at 08:50 -0800, Daniel Schepler wrote: > These are where I found that having extra packages installed during > the dpkg-buildpackage run either failed or resulted in broken > packages. (Some examples of the type of thing I mean: #948522, > #887902.) If you build outside a controlled environment the set of additional packages are just a small part of your worries. There are others such as - more creative solutions to satisfy Build-Depends, - anything in /usr/local, - anything in /opt (if PATH or other configuration includes /opt), - which alternative is active, - diversions, - local configuration, including environment variables (PATH, variables changing behavior of the Python interpreter, ...), - modified files, - local packages that do not exist in Debian's archive, or - modified packages. In many cases this will not be a problem, but trying to make any guarantees that the package building process will still work reliably seems to be a lost cause to me. It is fine to ask a maintainer to include some `--disable-something` flag if this is easy to do and reasonably likely to be a problem, but I would not expect people to actively spend effort on this. Ansgar
