Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
So, I did some tests here.
Helmut, look below:
On Tue, Dec 07, 2021 at 09:20:09PM +0100, Niels Thykier wrote:
> You can either depend on it or do a runtime check for the new function
> to see if the feature is available - whatever floats your boat. :)
I went this way indeed:
--- strip-nondeterminism.orig/bin/dh_strip_nondeterminism
+++ strip-nondeterminism/bin/dh_strip_nondeterminism
@@ -91,7 +91,13 @@
find(\,$tmp);
next unless @nondeterministic_files;
- $File::StripNondeterminism::canonical_time = get_source_date_epoch();
+ if (eval "get_non_binnmu_date_epoch()") {
+ $File::StripNondeterminism::canonical_time
+ = get_non_binnmu_date_epoch();
+ $ENV{SOURCE_DATE_EPOCH} = get_non_binnmu_date_epoch()
+ } else {
+ $File::StripNondeterminism::canonical_time =
get_source_date_epoch();
+ }
verbose_print("Using $File::StripNondeterminism::canonical_time as
canonical time");
And then I built and then binNMUed p11-kit/0.24.0-2 - which was one of
those packages affected by this "bug".
I confirm that for all 2 builds, strip-nondeterminism used 1630835345 as
timestamp, so the patch works.
However, in that particular package, there is still this change:
│ │ │ ├── ./usr/share/man/man1/trust.1.gz
│ │ │ │ ├── trust.1
│ │ │ │ │ @@ -1,12 +1,12 @@
│ │ │ │ │ '\" t
│ │ │ │ │ .\" Title: trust
│ │ │ │ │ .\"Author: Stef Walter
│ │ │ │ │ .\" Generator: DocBook XSL Stylesheets vsnapshot
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Chris Lamb:
> Hi Niels,
>
>> I think it would something like this from the debhelper side:
>>
>> https://salsa.debian.org/debian/debhelper/-/merge_requests/58
>>
>> Would that work for you? :)
>
> Indeed it would. Would strip-nondeterminism then simply Depend on the
> version of debhelper this change gets released in? Or rather: should
> we do something more involved than that?
>
>
> Regards,
>
Hi,
You can either depend on it or do a runtime check for the new function
to see if the feature is available - whatever floats your boat. :)
When you concluded the feature is available, you would then call
get_non_binnmu_date_epoch instead of get_source_date_epoch to obtain the
value for SOURCE_DATE_EPOCH. You may want to do a
`$ENV{SOURCE_DATE_EPOCH} = get_non_binnmu_date_epoch()` depending on
whether you call third-party tools that use the variable.
Thanks,
~Niels
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Hi Niels, > I think it would something like this from the debhelper side: > > https://salsa.debian.org/debian/debhelper/-/merge_requests/58 > > Would that work for you? :) Indeed it would. Would strip-nondeterminism then simply Depend on the version of debhelper this change gets released in? Or rather: should we do something more involved than that? Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Mattia Rizzolo: > [...] > > Niels: how do you suspect such "infrastructure" to look like? Right now > dh_strip_nondeterminism is using get_source_date_epoch() from Dh_Lib.pm > to get the timestamp to use for everything. If you can provide a > similar function returning a similar value but coming from the last > non-binary-only changelog entry, I suppose we could just switchin > dh_strip_nondeterminism to use that. > Hi, I think it would something like this from the debhelper side: https://salsa.debian.org/debian/debhelper/-/merge_requests/58 Would that work for you? :) Thanks, ~Niels
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
On Sun, Nov 28, 2021 at 10:38:40AM +0100, Niels Thykier wrote: > As in, would it be an option for dh_strip_nondeterminism to use the > changelog date from the original upload inside the files while > debhelper/dpkg used the changelog date from the binNMU load for the > external mtimes for files? I suppose this could be fine indeed. It feels a bit overengineered but should be fine perhaps. > * Provided it solves the issue without regression and is feasible, we >use asymmetric SOURCE_DATE_EPOCH timestamps inside files vs. external >mtimes. I am fine with providing "infrastructure" code for this in >Dh_Lib.pm to support dh_strip_nondeterminism. To be honest, this to me feels the best approach now. I actually didn't think of this hybrid solution in the past months when I was chatting with Helumt et al. on the topic. Thank you for proposing it! Though, tbh, I also just realized that StripNondeterminism only ever handles file content, the stuff dealing with file metadata in the "bare" directories is within debhelper itself. So uh, I suppose most of my concern are gone now. Niels: how do you suspect such "infrastructure" to look like? Right now dh_strip_nondeterminism is using get_source_date_epoch() from Dh_Lib.pm to get the timestamp to use for everything. If you can provide a similar function returning a similar value but coming from the last non-binary-only changelog entry, I suppose we could just switchin dh_strip_nondeterminism to use that. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. More about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- signature.asc Description: PGP signature
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Control: tags -1 + patch
Hi Chris,
On Wed, Dec 01, 2021 at 01:13:03AM -, Chris Lamb wrote:
> In other words, it's not a lack of interest or an uncaring disregard
> for other parts of Debian (!), something that other mails in this
> thread might have accidentally implied.
I did not mean to imply intent. The fact is that this issue is causing
significant pain for a significant amount of time and the reproducible
builds folks have generally not reacted to it in any helpful way. I also
note that your reply is fully consistent with this observation: You
acknowledge the problem and double down on keeping the broken status
quo.
> I'd be more than happy to change dh_strip_nondeterminism now to fix
> this longstanding issue for all of us, although with the proviso that
> the solution we want to get behind can be outlined in brief; and we
> seem to be close to there.
You seem to be referring to Niels' approach here. I think he already put
it quite precise. Is there anything you find missing? Moving ahead with
that would be my preference.
> > [Option:]
> > * We phase out dh_strip_nondeterminism in the next compat level.
>
> Not the ideal place for this info but unfortunately I don't think the
> rest of the archive is in a place to drop strip-nondeterminism on the
> next compat level. From memory, this is chiefly due to some key
> toolchain packages, where the required changes are not forthcoming,
> changes are blocking on upstream, patches have not been applied in
> Debian, or the discussion has stalled for various reasons.
As much as I see the other side of the picture and as much as I value
the achievements of reproducible builds, I come to the conclusion that
it is irrelevant. dh_strip_nondeterminism was accepted on the grounds of
not breaking stuff. Those grounds no longer exist. So the question
whether it still is needed, is secondary. The question of whether it
still is affordable already has an answer: no.
The more I deal with this issue, the more I am convinced that
dh_strip_nondeterminism should be removed from debhelper as soon as
possible. It wasn't added in a compat level. It doesn't need a compat
level to be dropped.
I think we can afford about a month for finding a better solution.
Failing that, we should move forward with the one we have.
Helmut
diff --minimal -Nru debhelper-13.5.2/debian/changelog
debhelper-13.5.2+nmu1/debian/changelog
--- debhelper-13.5.2/debian/changelog 2021-09-23 18:44:25.0 +0200
+++ debhelper-13.5.2+nmu1/debian/changelog 2021-12-01 11:25:45.0
+0100
@@ -1,3 +1,10 @@
+debhelper (13.5.2+nmu1) UNRELEASED; urgency=medium
+
+ * Non-maintainer upload.
+ * Do not call dh_strip_nondeterminism. (Closes: #999665)
+
+ -- Helmut Grohne Wed, 01 Dec 2021 11:25:45 +0100
+
debhelper (13.5.2) unstable; urgency=medium
[ Niels Thykier ]
diff --minimal -Nru debhelper-13.5.2/debian/control
debhelper-13.5.2+nmu1/debian/control
--- debhelper-13.5.2/debian/control 2021-09-06 20:16:48.0 +0200
+++ debhelper-13.5.2+nmu1/debian/control2021-12-01 11:24:59.0
+0100
@@ -22,10 +22,6 @@
# Constraint can be relaxed if packages do not rely/require any of the
# above fixes
dh-autoreconf (>= 17~),
-# Version constraint is to get a version of dh-strip-nondeterminism
-# without several bugs. The constraint can be relaxed if an older
-# version is sufficient.
- dh-strip-nondeterminism (>= 0.028~),
# dbgsym support landed in 1.18.0, 1.18.2 fixes a related nasty bug.
dpkg (>= 1.18.0~),
dpkg-dev (>= 1.18.2~),
diff --minimal -Nru debhelper-13.5.2/debian/rules
debhelper-13.5.2+nmu1/debian/rules
--- debhelper-13.5.2/debian/rules 2021-08-16 19:36:28.0 +0200
+++ debhelper-13.5.2+nmu1/debian/rules 2021-12-01 11:25:26.0 +0100
@@ -15,8 +15,8 @@
./run dh $@ --without autoreconf --with build-stamp
# Disable as they are unneeded (and we can then be sure debhelper
-# builds without needing autotools-dev, dh-strip-nondeterminism etc.)
-override_dh_update_autotools_config override_dh_strip_nondeterminism:
+# builds without needing autotools-dev etc.)
+override_dh_update_autotools_config:
execute_before_dh_install:
PERLLIBDIR=$$($(PERL) -MConfig -e 'print $$Config{vendorlib}')/ ; \
diff --minimal -Nru
debhelper-13.5.2/lib/Debian/Debhelper/Sequence/root_sequence.pm
debhelper-13.5.2+nmu1/lib/Debian/Debhelper/Sequence/root_sequence.pm
--- debhelper-13.5.2/lib/Debian/Debhelper/Sequence/root_sequence.pm
2021-08-16 19:30:02.0 +0200
+++ debhelper-13.5.2+nmu1/lib/Debian/Debhelper/Sequence/root_sequence.pm
2021-12-01 11:24:40.0 +0100
@@ -76,7 +76,6 @@
dh_link
dh_installwm
dh_installxfonts
- dh_strip_nondeterminism
dh_compress
dh_fixperms
dh_missing
diff --minimal -Nru debhelper-13.5.2/t/override_target.t
debhelper-13.5.2+nmu1/t/override_target.t
--- debhelper-13.5.2/t/override_target.t
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Hi all, >> As such, I propose that debhelper automatically disables >> dh_strip_nondeterminism if and only if both relevant conditions are met. >> What do you think about this? > > If we go that route, then the conditional would belong in > dh_strip_nondeterminism to disable itself. Just as a bit of throat-clearing, if you've not heard anything from the Reproducible Builds folks (or at least from me…) then the underlying reason is more that I am just not well-versed enough with the binNMU and builddd apparatus to have an informed opinion on this fairly technical and quite wide-reaching topic. In other words, it's not a lack of interest or an uncaring disregard for other parts of Debian (!), something that other mails in this thread might have accidentally implied. I'd be more than happy to change dh_strip_nondeterminism now to fix this longstanding issue for all of us, although with the proviso that the solution we want to get behind can be outlined in brief; and we seem to be close to there. > [Option:] > * We phase out dh_strip_nondeterminism in the next compat level. Not the ideal place for this info but unfortunately I don't think the rest of the archive is in a place to drop strip-nondeterminism on the next compat level. From memory, this is chiefly due to some key toolchain packages, where the required changes are not forthcoming, changes are blocking on upstream, patches have not been applied in Debian, or the discussion has stalled for various reasons. I do very much share your desire to phase out this tool completely and I even introduced a deprecation roadmap for every change that strip-nondeterminism makes for this very reason. But we're just not there... yet. > [Option:] > * debhelper works around dh_strip_nondeterminism deficiencies. Just to be explicit, I totally agree with this. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Hi Niels, On Sun, Nov 28, 2021 at 10:38:40AM +0100, Niels Thykier wrote: > To confirm: The root cause here is that dh_strip_nondeterminism uses the > binNMU timestamp which differs between each architecture for timestamps > *inside* the files being modified. Yes. > Clarification needed: This is about the *mtime* of files. Does this also > cover mtimes inside archives (e.g., .zip/.tar.gz) or is it only about > the mtime of the file on the file system? Thank you for pointing the finger at my imprecision. The issue with backups is about the mtime of files and that's what caused dh_strip_nondeterminism to change. The issue with Multi-Arch: same very much is not about the mtime of files and only about their contents. dh_strip_nondeterminism presently makes the contents of files vary and that's the very issue here. > As in, would it be an option for dh_strip_nondeterminism to use the > changelog date from the original upload inside the files while > debhelper/dpkg used the changelog date from the binNMU load for the > external mtimes for files? I agree. > If we go that route, then the conditional would belong in > dh_strip_nondeterminism to disable itself. Fine with me. I had little success convincing the reproducible folks that this problem needs solving though. > For me, the options are: > > * Provided it solves the issue without regression and is feasible, we >use asymmetric SOURCE_DATE_EPOCH timestamps inside files vs. external >mtimes. I am fine with providing "infrastructure" code for this in >Dh_Lib.pm to support dh_strip_nondeterminism. I actually prefer this one. Thank you for proposing it. > * dh_strip_nondeterminism detects this issue itself and exits (silently >or with a warning). Works for me. > * We phase out dh_strip_nondeterminism in the next compat level. >- In this case, I can be persuaded to do the work around inside > dh logic (but it would not work for "classic" debhelper and maybe > not for overrides of dh_strip_nondeterminism). Can we maybe agree to phase out dh_strip_nondeterminism on the next compat level unless it implements one of the other options? That'd definitely give the reproducible folks sufficient time to fix their tooling and it would solve the Multi-Arch issue in a reasonable amount of time. > Options that I am not interested in: > > * debhelper works around dh_strip_nondeterminism deficiencies. Makes sense. The options that I am not interested in: * Continue breaking Multi-Arch: same on binNMUs. Helmut
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Helmut Grohne: > Package: debhelper > Version: 13.5.2 > Severity: important > X-Debbugs-Cc: debian-cr...@lists.debian.org, > rb-gene...@lists.reproducible-builds.org > > Hi, > > [...] > > The situation is that when performing a binNMU, the changelogs are > created with differing timestamps. [...] As a consequence, the > SOURCE_DATE_EPOCH differs. When dh_strip_nondeterminism fixes up files, > it actually makes them differ from one another in a coinstallationset of > Multi-Arch: same packages. Files that were previously reproducible are > made unreproducible by dh_strip_nondeterminism. To confirm: The root cause here is that dh_strip_nondeterminism uses the binNMU timestamp which differs between each architecture for timestamps *inside* the files being modified. > The behaviour used to be > different and the most recent non-binNMU date was considered, but that > happend to break backup software as it could no longer detect > modifications from a non-binNMU to binNMU upgrade. That is how we ended > up with current situation. > Clarification needed: This is about the *mtime* of files. Does this also cover mtimes inside archives (e.g., .zip/.tar.gz) or is it only about the mtime of the file on the file system? As in, would it be an option for dh_strip_nondeterminism to use the changelog date from the original upload inside the files while debhelper/dpkg used the changelog date from the binNMU load for the external mtimes for files? > [...] > > In order for a package to actually hit the bug, two conditions must be > met simultaneously: > * Some binary package must be marked Multi-Arch: same. > * The build must be a binNMU. > > [...] > > Given that there is no progress for prolonged time, we have a choice: > Either we produce reproducible packages that happen to not be > installable or we produce installable packages that happen to not be > reproducible. The trade-off appears obvious to me. Keep in mind that > dh_strip_nondeterminism was enabled on the provision of not breaking > stuff. Now it does break stuff. > I always saw dh_strip_nondeterminism was a *temporary* solution to kick-start the reproducibility process. As such I hoped we would eventually phase it out. > As such, I propose that debhelper automatically disables > dh_strip_nondeterminism if and only if both relevant conditions are met. > What do you think about this? > If we go that route, then the conditional would belong in dh_strip_nondeterminism to disable itself. > I think the reproducible builds folks favour a solution that involves > sourcefull uploads replacing binNMUs. The proposed workaround does not > interfere with the reproducible proposal in any way. > > Do you need a patch? > > Helmut > For me, the options are: * Provided it solves the issue without regression and is feasible, we use asymmetric SOURCE_DATE_EPOCH timestamps inside files vs. external mtimes. I am fine with providing "infrastructure" code for this in Dh_Lib.pm to support dh_strip_nondeterminism. * dh_strip_nondeterminism detects this issue itself and exits (silently or with a warning). * We phase out dh_strip_nondeterminism in the next compat level. - In this case, I can be persuaded to do the work around inside dh logic (but it would not work for "classic" debhelper and maybe not for overrides of dh_strip_nondeterminism). Options that I am not interested in: * debhelper works around dh_strip_nondeterminism deficiencies. Thanks, ~Niels
Bug#999665: dh_strip_nondeterminism breaks Multi-Arch: same in binNMUs
Package: debhelper
Version: 13.5.2
Severity: important
X-Debbugs-Cc: debian-cr...@lists.debian.org,
rb-gene...@lists.reproducible-builds.org
Hi,
dh_strip_nondeterminism was automatically enabled on the provision that
it wouldn't break stuff. To a large extend, that's the case.
Unfortunately, there is a corner case where it does get things wrong and
we cannot make progress as the requirements appear to conflict with one
another. The issue is reported as #950806.
The situation is that when performing a binNMU, the changelogs are
created with differing timestamps. (That's a technical limitation of our
archive implementation. In theory, binNMUs could use equal timestamps
and sbuid already has --binNMU-timestamp.) As a consequence, the
SOURCE_DATE_EPOCH differs. When dh_strip_nondeterminism fixes up files,
it actually makes them differ from one another in a coinstallationset of
Multi-Arch: same packages. Files that were previously reproducible are
made unreproducible by dh_strip_nondeterminism. The behaviour used to be
different and the most recent non-binNMU date was considered, but that
happend to break backup software as it could no longer detect
modifications from a non-binNMU to binNMU upgrade. That is how we ended
up with current situation.
So we're actually talking about a relatively small set of affected
packages. Unfortunately, libruby2.N is among them and that has a
relatively high impact. But the issue does not stop there. We've had
other high profile cases in the past. They tend to pop up at unfortunate
times.
In order for a package to actually hit the bug, two conditions must be
met simultaneously:
* Some binary package must be marked Multi-Arch: same.
* The build must be a binNMU.
We can estimate this set using grep-dctrl:
grep-dctrl '(' -r -F Version '.*+b[0-9]\+$' ')' --and '(' -F Multi-Arch -X same
')' -s Package -n $PKGFILE
For unstable, that's about 2100 binary packages built from 900 source
packages at present.
Given that there is no progress for prolonged time, we have a choice:
Either we produce reproducible packages that happen to not be
installable or we produce installable packages that happen to not be
reproducible. The trade-off appears obvious to me. Keep in mind that
dh_strip_nondeterminism was enabled on the provision of not breaking
stuff. Now it does break stuff.
As such, I propose that debhelper automatically disables
dh_strip_nondeterminism if and only if both relevant conditions are met.
What do you think about this?
I think the reproducible builds folks favour a solution that involves
sourcefull uploads replacing binNMUs. The proposed workaround does not
interfere with the reproducible proposal in any way.
Do you need a patch?
Helmut
