Bug#977674: Corrupt changes file when built with --source-only-changes
On Mon, 25 Jan 2021 17:45:29 +0100 Johannes Schauer Marin Rodrigues wrote: > The problem is, that when you combine --source-only-changes with --keyid, then > debsign will be run twice (once for the normal changes file and once for the > source-only changes file) and both times with --re-sign. This means, that the > second invocation will possibly also change the signature of files that were > already processed by the first invocation and this means that the checksum of > the first changes file doesn't match anymore. > > To fix the problem, one might suggest to just run the second invocation of > debsign with --no-re-sign so that everything that is already signed does not > get changed and only those things that don't have a signature get signed. > > But this triggers a bug in debsign where the dsc will not even be considered > for signing if the buildinfo was already signed. Since the buildinfo file of an upload contains the checksums of the dsc, this behaviour makes sense, as signing the dsc would break the buildinfo. On the other hand for the same reason, if the buildinfo is signed the dsc should already be signed as well. In the sbuild --source --source-only-changes case it most certainly will be, because the first debsign invocation signed it. What use case would using --no-re-sign for the second call break? -- Regards, Feri.
Bug#977674: Corrupt changes file when built with --source-only-changes
Hi, Quoting Mathias Behrle (2021-01-27 11:57:04) > > This code was introduced in devscripts back in 2017, so you can see that I > > was correct when I said that the sbuild codepath of combining > > --source-only-changes with --keyid is indeed seldom used. > > Yes, the real use case appeared with mandatory source only uploads. > > On one side I need the source only changes for uploads to ftp.debian.org, on > the other side I need binaries with according changes files suitable for > testing and upload to the backports reprepro. I think what most sbuild users do, is to run sbuild without --keyid and then let dput do the signing. Myself, I'm using "dgit push-source" to do my source-only uploads. I never run sbuild with --source-only-changes because dgit will take care of doing all of that for me. Clearly, this bug has to be fixed. I just wanted to suggest some workaround for you and explanations why this was only found so late. Thanks! cheers, josch signature.asc Description: signature
Bug#977674: Corrupt changes file when built with --source-only-changes
* Johannes Schauer Marin Rodrigues: " Re: Bug#977674: Corrupt changes file when
built with --source-only-changes" (Mon, 25 Jan 2021 17:45:29 +0100):
Hi Josh,
> I got a bit further on this bug.
Big thanks for taking a look.
> The problem is, that when you combine --source-only-changes with --keyid, then
> debsign will be run twice (once for the normal changes file and once for the
> source-only changes file) and both times with --re-sign. This means, that the
> second invocation will possibly also change the signature of files that were
> already processed by the first invocation and this means that the checksum of
> the first changes file doesn't match anymore.
That sounds exactly like the problem I have.
> To fix the problem, one might suggest to just run the second invocation of
> debsign with --no-re-sign so that everything that is already signed does not
> get changed and only those things that don't have a signature get signed.
>
> But this triggers a bug in debsign where the dsc will not even be considered
> for signing if the buildinfo was already signed. Consider this code from
> debsign:
>
> maybesign_buildinfo() {
> [...]
> if check_already_signed "$buildinfo" "buildinfo"; then
>echo "Leaving current signature unchanged." >&2
>return
> fi
>
> if [ -n "$dsc" ]; then
> maybesign_dsc "$signas" "$remotehost" "$dsc"
> withtempfile buildinfo "$buildinfo" fixup_buildinfo "$dsc"
> fi
> [...]
>
> As you can see, the function will return immediately without checking the dsc
> if the buildinfo is already signed.
>
> This code was introduced in devscripts back in 2017, so you can see that I was
> correct when I said that the sbuild codepath of combining
> --source-only-changes with --keyid is indeed seldom used.
Yes, the real use case appeared with mandatory source only uploads.
On one side I need the source only changes for uploads to ftp.debian.org, on
the other side I need binaries with according changes files suitable for testing
and upload to the backports reprepro.
> I reported this as devscripts bug #981021 but I suggest that you have a look
> into it or I fear that the chances of $somebody doing the work for us are
> slim.
Will have a look ASAP.
> Thanks!
Thanks to you!
Cheers
Mathias
--
Mathias Behrle
PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6
AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6
Bug#977674: Corrupt changes file when built with --source-only-changes
Hi,
I got a bit further on this bug.
The problem is, that when you combine --source-only-changes with --keyid, then
debsign will be run twice (once for the normal changes file and once for the
source-only changes file) and both times with --re-sign. This means, that the
second invocation will possibly also change the signature of files that were
already processed by the first invocation and this means that the checksum of
the first changes file doesn't match anymore.
To fix the problem, one might suggest to just run the second invocation of
debsign with --no-re-sign so that everything that is already signed does not
get changed and only those things that don't have a signature get signed.
But this triggers a bug in debsign where the dsc will not even be considered
for signing if the buildinfo was already signed. Consider this code from
debsign:
maybesign_buildinfo() {
[...]
if check_already_signed "$buildinfo" "buildinfo"; then
echo "Leaving current signature unchanged." >&2
return
fi
if [ -n "$dsc" ]; then
maybesign_dsc "$signas" "$remotehost" "$dsc"
withtempfile buildinfo "$buildinfo" fixup_buildinfo "$dsc"
fi
[...]
As you can see, the function will return immediately without checking the dsc
if the buildinfo is already signed.
This code was introduced in devscripts back in 2017, so you can see that I was
correct when I said that the sbuild codepath of combining --source-only-changes
with --keyid is indeed seldom used.
I reported this as devscripts bug #981021 but I suggest that you have a look
into it or I fear that the chances of $somebody doing the work for us are slim.
Thanks!
cheers, josch
signature.asc
Description: signature
Bug#977674: Corrupt changes file when built with --source-only-changes
Control: tag -1 - newcomer Hi, Quoting Mathias Behrle (2021-01-25 08:55:34) > Probably I shall try this next time resp. I will try with an old version of > sbuild. > > IIRC this bug is a regression introduced (probably between August and > September > 2020). Before the signing process used to work for changes as well as for > source-only-changes. From my unexperienced view the following commit comes to > my > attention > https://salsa.debian.org/debian/sbuild/-/commit/bafc5362e0bbf29c5604fcaa73454e0fd6d0e3fc > that could have caused the impact. that this is maybe a regression piqued my interest. I tried to reproduce the problem you are seeing by running sbuild with --keyid, --source-only-changes and --source. This is probably the combination that causes your problem. Unfortunately, I was not able to reproduce your problem. How did you verify that the checksums indeed are wrong? Did you try running dscverify on the amd64 and source changes files? I tried with a test package and it all checks out: tmp/test-pkg_1.0_source.changes: validating test-pkg_1.0.dsc validating test-pkg_1.0.tar.xz validating test-pkg_1.0_amd64.buildinfo All files validated successfully. tmp/test-pkg_1.0_amd64.changes: validating test-pkg_1.0.dsc validating test-pkg_1.0.tar.xz validating test-pkg_1.0_all.deb validating test-pkg_1.0_amd64.buildinfo All files validated successfully. Also, maybe unrelated, why are you running sbuild with --source? Did you read the part in the sbuild man page about the --source option? Thanks! cheers, josch signature.asc Description: signature
Bug#977674: Corrupt changes file when built with --source-only-changes
* Johannes Schauer Marin Rodrigues: " Re: Bug#977674: Corrupt changes file when built with --source-only-changes" (Sun, 24 Jan 2021 22:53:56 +0100): Hi Josh, > Quoting Mathias Behrle (2020-12-18 16:39:44) > I'm not surprised that this bug exists. The codepath you are using doesn't get > any testing. The official buildds are not using source-only-changes and those > who do, do not sign the build result with sbuild. > > I'm tagging this with "help" and "newcomer". The fix is is likely to modify > the function close_build_log() in lib/Sbuild/Build.pm. There is a part that > calls debsign differently if SOURCE_ONLY_CHANGES is active and that part is > probably broken somehow. A merge request against > https://salsa.debian.org/debian/sbuild is greatly appreciated. Thanks for your analysis. As I am not familiar with perl at all I am not able to do this merge request. > Since you are using sbuild in a buildd context I think you can work around > this bug for now by either manually signing or manually mangling your changes > files? Probably I shall try this next time resp. I will try with an old version of sbuild. IIRC this bug is a regression introduced (probably between August and September 2020). Before the signing process used to work for changes as well as for source-only-changes. From my unexperienced view the following commit comes to my attention https://salsa.debian.org/debian/sbuild/-/commit/bafc5362e0bbf29c5604fcaa73454e0fd6d0e3fc that could have caused the impact. Cheers Mathias -- Mathias Behrle PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6 AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6
Bug#977674: Corrupt changes file when built with --source-only-changes
Control: tag -1 + help newcomer Hi, sorry for the late reply. Quoting Mathias Behrle (2020-12-18 16:39:44) > I am running a builder for the Tryton packages that is configured to provide > a changes as well as a source.changes file. The first used for the upload to > the reprepro mirror, the latter used to upload to Debian. > > The results of the last builder run caused mismatches of checksums when > importing to reprepro. Indeed the control of the checksums prooved to be > wrong. I suspect the following lines in the build logs to be the > culprit: > > """ > Signature with key 'mathi...@m9s.biz' requested: > signfile dsc > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > mathi...@m9s.biz > > fixup_buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > signfile buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > mathi...@m9s.biz > > fixup_changes dsc > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes > fixup_changes buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes > signfile changes > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes > mathi...@m9s.biz > > Successfully signed dsc, buildinfo, changes files > > ---> from here comes the problematic part from --source-only-changes > > unsignfile > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > unsignfile > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > signfile dsc > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > mathi...@m9s.biz > > fixup_buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > signfile buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > mathi...@m9s.biz > > fixup_changes dsc > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes > fixup_changes buildinfo > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo > > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes > signfile changes > /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes > mathi...@m9s.biz > > Successfully signed dsc, buildinfo, changes files > """ > > > Indeed running without --sourice-only-changes produced correct changes files. > > The options used for the sbuild run are > ['sbuild', '--quiet', '--chroot=sid-amd64-sbuild', > '--keyid=mathi...@m9s.biz', '--source', '--mailfrom="Debian Autobuilder > "', '--mail-log-to=mathi...@m9s.biz', '--sbuild-mode=user', > '--no-apt-update', '--no-apt-upgrade', '--no-apt-distupgrade', > '--lintian-opts=-i -v -I -E --pedantic', '--bd-uninstallable-explainer=apt', > '--source-only-changes', > '/home/mathiasb/bin/tryton/debian_builder/build/tryton-server'] > > I am attaching for reference two changes files, one with > --source-only-changes, one without. I'm not surprised that this bug exists. The codepath you are using doesn't get any testing. The official buildds are not using source-only-changes and those who do, do not sign the build result with sbuild. I'm tagging this with "help" and "newcomer". The fix is is likely to modify the function close_build_log() in lib/Sbuild/Build.pm. There is a part that calls debsign differently if SOURCE_ONLY_CHANGES is active and that part is probably broken somehow. A merge request against https://salsa.debian.org/debian/sbuild is greatly appreciated. Since you are using sbuild in a buildd context I think you can work around this bug for now by either manually signing or manually mangling your changes files? Thanks! cheers, josch signature.asc Description: signature
Bug#977674: Corrupt changes file when built with --source-only-changes
Package: sbuild Version: 0.80.1 Severity: important Dear Maintainer, I am running a builder for the Tryton packages that is configured to provide a changes as well as a source.changes file. The first used for the upload to the reprepro mirror, the latter used to upload to Debian. The results of the last builder run caused mismatches of checksums when importing to reprepro. Indeed the control of the checksums prooved to be wrong. I suspect the following lines in the build logs to be the culprit: """ Signature with key 'mathi...@m9s.biz' requested: signfile dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc mathi...@m9s.biz fixup_buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo signfile buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo mathi...@m9s.biz fixup_changes dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes fixup_changes buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes signfile changes /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.changes mathi...@m9s.biz Successfully signed dsc, buildinfo, changes files ---> from here comes the problematic part from --source-only-changes unsignfile /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo unsignfile /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc signfile dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc mathi...@m9s.biz fixup_buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo signfile buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo mathi...@m9s.biz fixup_changes dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1.dsc /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes fixup_changes buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_amd64.buildinfo /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes signfile changes /home/mathiasb/bin/tryton/debian_builder/results/sid/tryton-server_5.0.30-1_source.changes mathi...@m9s.biz Successfully signed dsc, buildinfo, changes files """ Indeed running without --sourice-only-changes produced correct changes files. The options used for the sbuild run are ['sbuild', '--quiet', '--chroot=sid-amd64-sbuild', '--keyid=mathi...@m9s.biz', '--source', '--mailfrom="Debian Autobuilder "', '--mail-log-to=mathi...@m9s.biz', '--sbuild-mode=user', '--no-apt-update', '--no-apt-upgrade', '--no-apt-distupgrade', '--lintian-opts=-i -v -I -E --pedantic', '--bd-uninstallable-explainer=apt', '--source-only-changes', '/home/mathiasb/bin/tryton/debian_builder/build/tryton-server'] I am attaching for reference two changes files, one with --source-only-changes, one without. -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (800, 'testing'), (700, 'unstable'), (600, 'experimental'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.9.0-4-amd64 (SMP w/4 CPU threads) Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8), LANGUAGE not set Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages sbuild depends on: ii adduser 3.118 ii libsbuild-perl 0.80.1 ii perl5.32.0-5 Versions of packages sbuild recommends: ii autopkgtest 5.15 ii debootstrap 1.0.123 ii schroot 1.6.10-11+b1 Versions of packages sbuild suggests: pn deborphan ii e2fsprogs 1.45.6-1 ii kmod 27+20200310-2 ii wget 1.20.3-1+b3 -- no debconf information -- Mathias Behrle PGP/GnuPG key availabable from any keyserver, ID: 0xD6D09BE48405BBF6 AC29 7E5C 46B9 D0B6 1C71 7681 D6D0 9BE4 8405 BBF6 tryton-server_5.0.30-1_amd64.changes_source-only-changes Description: Binary data tryton-server_5.0.30-1_amd64.changes Description: Binary data
