Bug#1054125: dh-builtusing: Please backport dh-builtusing to bookworm
Hi Nicolas, On Sat, Nov 18, 2023 at 06:27:42PM +0100, Nicolas Boulenguez wrote: > Have you seen this? > https://lists.debian.org/debian-devel/2023/08/ What exactly are you referring to? That's the whole month's archive :) Yes I do read d-devel so I've likely seen whatever you're pointing to :p > > I've been toying with the idea of setting up a Debian-wide system to nag > > maintainers about out-of-date, inconsistent or plain broken packaging git > > repos. This logic to diff the dsc against one built from unstable could be > > part of that effort. > > Some will probably argue that you are trying to influence their > priorities, that non-git .dsc formats are obsolete, and so on. And that's a bad thing because ... ? There's a huge difference between nudging and forcing :) > Moreover, even if you guess the git tag and whether patches are > applied, there is probably no deterministic way to tell if .gitignore > matches the workflow of upstream, Debian or both. Sounds like a job for a new config option in debian/source/ or similar, this is conceptually no different than lintian false-positives. As long as there's a way to override I don't see a problem. > For the avoidance of doubt, I would appreciate such alerts, a Debian > policy about tags and patches, and that .gitignore is only allowed for > self defense... I for sure don't have all this thought yet, I'm happy to collaborate on designing this system if you're interested. BTW: are you at ccc congress? > > Even with git what may be missing is a hook in dpkg-buildpackage to abort > > the (source) build when the worktree is unclean, > > I often build with manual changes in debian/ that I want tested before > committed. Right yeah same. So that wouldn't really help. gbp's --git-ignore-new is already annoying me enough :) > The script I am using is too lazy for public exposure. You can always send me those shameful scripts directly, I don't judge.sh. > Here are the parts unrelated with pbuilder. > > # After a successful build, clean and compare the source directory > # with the contents extracted by dpkg-source. > # The diff must match debian/clean.diff if it exists, else be empty. > # Example: > # Only in ../dsc: Makefile.in > # Only in ../dsc: config.h > # Only in ../dsc: configure > # If lots of files are removed, repackaging with Files-Excluded is > # probably a better option, see uscan(1). > > #!/bin/sh > set -Ceux > > source=$(dpkg-parsechangelog -SSource) > version=$(dpkg-parsechangelog -SVersion) > dsc_dir=../dsc > > debian/rules clean > dpkg-source -x ../$source_$version.dsc $dsc_dir > if ! (diff -qr $dsc_dir . || true) | diff -N debian/clean.diff -; then > # Display a full diagnostic while $dsc_dir is available. > diff -ru $dsc_dir . || true > # When -ru is empty, -N above already reported the obsolete clean.diff. > fi > rm -fr $dsc_dir I see, this is actually really clever. I don't see why we couldn't do this diff on the buildds to get reporting for this Debian wide. I've been meaning to give sbuild some love anyhow. --Daniel signature.asc Description: PGP signature
Bug#1054125: dh-builtusing: Please backport dh-builtusing to bookworm
Have you seen this? https://lists.debian.org/debian-devel/2023/08/ > I've been toying with the idea of setting up a Debian-wide system to nag > maintainers about out-of-date, inconsistent or plain broken packaging git > repos. This logic to diff the dsc against one built from unstable could be > part of that effort. Some will probably argue that you are trying to influence their priorities, that non-git .dsc formats are obsolete, and so on. Moreover, even if you guess the git tag and whether patches are applied, there is probably no deterministic way to tell if .gitignore matches the workflow of upstream, Debian or both. For the avoidance of doubt, I would appreciate such alerts, a Debian policy about tags and patches, and that .gitignore is only allowed for self defense... > Even with git what may be missing is a hook in dpkg-buildpackage to abort > the (source) build when the worktree is unclean, I often build with manual changes in debian/ that I want tested before committed. > > [...] script cleaning after the build, > > then reporting any difference between the source and the .dsc. > > The output is almost always either > > * trivial to work-around in Debian > >(echo '*.o' > debian/clean) > >(fixing upstream may be another story) > > * short enough to be easily ignored > >(./configure), > > * or a real issue. > [..] could you elaborate or show an example? The script I am using is too lazy for public exposure. Here are the parts unrelated with pbuilder. # After a successful build, clean and compare the source directory # with the contents extracted by dpkg-source. # The diff must match debian/clean.diff if it exists, else be empty. # Example: # Only in ../dsc: Makefile.in # Only in ../dsc: config.h # Only in ../dsc: configure # If lots of files are removed, repackaging with Files-Excluded is # probably a better option, see uscan(1). #!/bin/sh set -Ceux source=$(dpkg-parsechangelog -SSource) version=$(dpkg-parsechangelog -SVersion) dsc_dir=../dsc debian/rules clean dpkg-source -x ../$source_$version.dsc $dsc_dir if ! (diff -qr $dsc_dir . || true) | diff -N debian/clean.diff -; then # Display a full diagnostic while $dsc_dir is available. diff -ru $dsc_dir . || true # When -ru is empty, -N above already reported the obsolete clean.diff. fi rm -fr $dsc_dir
Bug#1054125: dh-builtusing: Please backport dh-builtusing to bookworm
Hi Nicolas, Sorry for taking so long to get back to you I've had some family issues to deal with. On Thu, Oct 19, 2023 at 12:42:18PM +0200, Nicolas Boulenguez wrote: > > would it be possible to get dh-builtusing backported for bookworm? We > > want to use dh-builtusing in ghdl (as you know ;), but I run my > > package builds on bookworm, inside an sbuild chroot, so gbp breaks > > when building the source package it's going to hand to the chroot. > > Please first consider a simple solution that I have been using for > years. > > Dpkg-buildpackage --build=source (run by $tool) cleans the source tree > before preparing the .dsc (that $tool then copies into the chroot). Incidentally, I've done some light code review of dpkg-dev a while ago. I was looking at how/when patches are applied/unapplied but the cleaning logic is close by. > The motivation for the cleaning was probably to prevent objects > embedded into the .dsc, but I think it might also be related to the automatic patch generation some source formats support. From that perspective: if you don't clean up before building the dsc the generated patch could contain build cruft. Of course there's extend-diff-ignore to prevent this so I'm not entirely sure this tracks. > * dpkg-buildpackage now (source format 3.0) aborts when it detects a > file present in the source tree but missing or different in the > .orig tarball I had this same throught "why can't we just disable pre-clean for 3.0 (quilt)" but that's forgetting about debian/. Build products end up there so we need to clean those up or we could include build cruft in the dsc and potentially change the behaviour of the build that way. > The clean step may even be harmful. > * It requires build dependencies outside the chroot >(your issue with dh-builtusing). > * Even if these are installed, the versions and behaviour may differ. >(probable cause of your issue with dh-ada-library) I suppose in theory this behavioural difference could reflect in the dsc content. I don't think that's a really a huge problem, it's just another of a number of reasons why a dsc freshly built from packaging git may differ from whats in the archive. I kind of see this (source) build diversity to be a good thing. It means I'm going to encounter (and fix) issues users doing their own ghetto backports (like I used to ;) are going to encounter. Perhaps it would be worthwhile to setup some system (salsa CI?) to detect this case tho. I've been toying with the idea of setting up a Debian-wide system to nag maintainers about out-of-date, inconsistent or plain broken packaging git repos. This logic to diff the dsc against one built from unstable could be part of that effort. > * The upstream clean process may execute surprising commands like > find . -name '*~' -delete > git reset --hard >or make applied debian/patches hard to revert. > * Various packages require incompatible Build-Dependencies. > * ... > > You can disable the clean with > # dpkg-buildpackage -nc > # pdebuild --debbuildopts -nc > or a similar gbp-buildpackage option. I enabled this on my workstation by way of $ echo no-pre-clean > ~/.config/dpkg/buildpackage.conf and after using it for a while I can't say I enjoy it or consider it safe. It's wayyy too easy to forget to clean this way. Even with git what may be missing is a hook in dpkg-buildpackage to abort the (source) build when the worktree is unclean, but even then I'm not sure gbp export-orig will apply debian/.gitignore if it exists, which could be another pitfall. > In case you want to keep a basic check of the clean target, the right > place is a pbuilder script cleaning inside the chroot after the build, > then reporting any difference between the source and the .dsc. > The output is almost always either > * trivial to work-around in Debian >(echo '*.o' > debian/clean) >(fixing upstream may be another story) > * short enough to be easily ignored >(./configure), > * or a real issue. I never really used pbuilder, I went straight to sbuild so I'm not sure what this setup would look like, could you elaborate or show an example? --Daniel signature.asc Description: PGP signature
Bug#1054125: dh-builtusing: Please backport dh-builtusing to bookworm
Hello. > would it be possible to get dh-builtusing backported for bookworm? We > want to use dh-builtusing in ghdl (as you know ;), but I run my > package builds on bookworm, inside an sbuild chroot, but gbp breaks > when building the source package it's going to hand to the chroot. Please first consider a simple solution that I have been using for years. Dpkg-buildpackage --build=source (run by $tool) cleans the source tree before preparing the .dsc (that $tool then copies into the chroot). The motivation for the cleaning was probably to prevent objects embedded into the .dsc, but * dpkg-buildpackage now (source format 3.0) aborts when it detects a file present in the source tree but missing or different in the .orig tarball * chroot users rarely build objects locally * git users (keeping away from .gitignore) rarely forget objects in the source tree. The clean step may even be harmful. * It requires build dependencies outside the chroot (your issue with dh-builtusing). * Even if these are installed, the versions and behaviour may differ. (probable cause of your issue with dh-ada-library) * The upstream clean process may execute surprising commands like find . -name '*~' -delete git reset --hard or make applied debian/patches hard to revert. * Various packages require incompatible Build-Dependencies. * ... You can disable the clean with # dpkg-buildpackage -nc # pdebuild --debbuildopts -nc or a similar gbp-buildpackage option. In case you want to keep a basic check of the clean target, the right place is a pbuilder script cleaning inside the chroot after the build, then reporting any difference between the source and the .dsc. The output is almost always either * trivial to work-around in Debian (echo '*.o' > debian/clean) (fixing upstream may be another story) * short enough to be easily ignored (./configure), * or a real issue.
Bug#1054125: dh-builtusing: Please backport dh-builtusing to bookworm
Source: dh-builtusing Version: 0.0.5 Severity: wishlist X-Debbugs-Cc: d...@darkboxed.org Hi Nicolas, would it be possible to get dh-builtusing backported for bookworm? We want to use dh-builtusing in ghdl (as you know ;), but I run my package builds on bookworm, inside an sbuild chroot, but gbp breaks when building the source package it's going to hand to the chroot. ``` dh /usr/share/ada/packaging.mk dh: error: unable to load addon builtusing: Can't locate Debian/Debhelper/Sequence/builtusing.pm in @INC (you may need to install the Debian::Debhelper::Sequence::builtusing module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at (eval 15) line 1. BEGIN failed--compilation aborted at (eval 15) line 1. debian/rules:10: /usr/share/ada/packaging.mk: No such file or directory make: *** [debian/rules:65: /usr/share/ada/packaging.mk] Error 255 E: Failed to clean source directory /home/dxld/share/dev/deb/pkg/ghdl (/home/dxld/share/dev/deb/pkg/ghdl_3.0.0+dfsg2-1.dsc) gbp:error: 'sbuild -d unstable --anything-failed-commands %SBUILD_SHELL' failed: it exited with 1 ``` I've reported the dh-ada-library error seperately, I think either of those problems would break the srcpkg build. Thanks, --Daniel
