Bug#1030785: -ffile-prefix-map option and reproducibility
Hi! I'm end up with following thing: https://github.com/rockdrilla/debian-container/blob/808dfa15d130f833602a1486362cc551593a0d8a/image/python/template/rules#L543 Hovewer, "execute_before_dh_builddeb" recipe is for information only (to be viewed by human) and "execute_before_dh_install" recipe is for actual adjustments. ср, 8 февр. 2023 г. в 13:39, Guillem Jover : > > On Tue, 2023-02-07 at 20:00:06 +0100, Sven Joachim wrote: > > On 2023-02-07 17:50 +0100, Guillem Jover wrote: > > > On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote: > > >> I suspect this was added to improve reproducibility. Ironically, it makes > > >> packages that capture this variable non reproducible, since the build > > >> path > > >> seems to be randomized (has it always been the case? since when?). It is > > >> the > > >> case of OCaml (see #1030785), and seemingly of R as well (found by > > >> grepping > > >> in my /etc). I wouldn't be surprised other packages are affected as well. > > > > > > AFAIR this was considered at the time, yes. If the flag is effectively > > > not fixing anything for the set of packages involved, and is in fact > > > actually making them unreproducible when they would not then, you can > > > disable the fixfilepath feature in the reproducible build flags area, > > > via DEB_BUILD_MAINT_OPTIONS. > > > > This does not help for packages which capture all build flags and store > > them in some file in the package (as is the case here). With > > DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath, dpkg-buildflags falls > > back to "-fdebug-prefix-map==.", and you have the same > > problem. If you disable that as well via > > DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath, the > > -dbgsym packages will most likely end up unreproducible. > > Ah, you are absolutely right. I don't think the case of these flags > making the build unreproducible and the package not generating any > debug objects are going to be common at all. I considered mentioning > fixing the build to stop capturing, but felt it might be more effort > than requested. :) Should probably have mentioned anyway, as was done > elsewhere in the thread. > > In any case I'm thinking to add something like the attached to the man > page to try to clarify this. > > Thanks, > Guillem -- SY, Konstantin Demin
Bug#1030785: -ffile-prefix-map option and reproducibility
On 2023-02-08, Stéphane Glondu wrote: > Thank you all for your answers! > > Using: > >DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath > > makes the package unreproducible in another way that seems difficult to fix. Most likely reintroducing the things that the -ffile-prefix-map and -fdebug-prefix-map was effectively removing... We track these kinds of issues with the "records build flags" issue, which has a description of the problem and links to more information: https://tests.reproducible-builds.org/debian/issues/unstable/records_build_flags_issue.html There are some potential fixes to the issue more fundamentally, but they are currently stalled out... one of which I should probably spend some time on after bookworm release... You had earlier asked when this was enabled, this can mostly be found in the dpkg changelog: fixfilepath (a.k.a. -ffile-prefix-map) Enabled by default: dpkg (1.20.6) unstable; urgency=medium ... * dpkg-buildflags: Enable reproducible=fixfilepath by default. Thanks to Vagrant Cascadian . See https://lists.debian.org/debian-devel/2020/10/msg00222.html. Closes: #974087 ... -- Guillem Jover Fri, 08 Jan 2021 04:39:40 +0100 fixfilepath (a.k.a. -ffile-prefix-map) feature added, and enabled in reproducible builds infrastructure soon after: dpkg (1.19.1) unstable; urgency=medium ... - Dpkg::Vendor::Debian: Add fixfilepath support to reproducible feature. ... -- Guillem Jover Wed, 26 Sep 2018 15:13:22 +0200 fixdebugpath (a.k.a. -fdebug-prefix-map) enabled by default: dpkg (1.18.10) unstable; urgency=medium ... - Enable fixdebugpath build flag feature by default. Thanks to Mattia Rizzolo . Closes: #832179 ... -- Guillem Jover Sun, 31 Jul 2016 12:57:02 +0200 fixdebugpath (a.k.a. -fdebug-prefix-map) feature added, and presumably enabled in reproducible builds infrastructure soon after: dpkg (1.18.5) unstable; urgency=medium ... - Add fixdebugpath to reproducible feature in Dpkg::Vendor::Debian. Thanks to Daniel Kahn Gillmor . Closes: #819194 ... -- Guillem Jover Mon, 02 May 2016 04:14:57 +0200 Of course, this is only for packages respecting dpkg-buildflags. > Le 07/02/2023 à 19:12, Mattia Rizzolo a écrit : >> I actually propose to you to filter out the whole option from being >> saved. [...] > > I've gone this way, and managed to make the package reproducible, at > least with the build path variation. Glad that works! > I will upload the fixed ocaml package when the current batch of related > packages waiting in unstable migrates to testing, hopefully in 4 days. Thanks! live well, vagrant signature.asc Description: PGP signature
Bug#1030785: -ffile-prefix-map option and reproducibility
On Tue, 2023-02-07 at 20:00:06 +0100, Sven Joachim wrote: > On 2023-02-07 17:50 +0100, Guillem Jover wrote: > > On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote: > >> I suspect this was added to improve reproducibility. Ironically, it makes > >> packages that capture this variable non reproducible, since the build path > >> seems to be randomized (has it always been the case? since when?). It is > >> the > >> case of OCaml (see #1030785), and seemingly of R as well (found by grepping > >> in my /etc). I wouldn't be surprised other packages are affected as well. > > > > AFAIR this was considered at the time, yes. If the flag is effectively > > not fixing anything for the set of packages involved, and is in fact > > actually making them unreproducible when they would not then, you can > > disable the fixfilepath feature in the reproducible build flags area, > > via DEB_BUILD_MAINT_OPTIONS. > > This does not help for packages which capture all build flags and store > them in some file in the package (as is the case here). With > DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath, dpkg-buildflags falls > back to "-fdebug-prefix-map==.", and you have the same > problem. If you disable that as well via > DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath, the > -dbgsym packages will most likely end up unreproducible. Ah, you are absolutely right. I don't think the case of these flags making the build unreproducible and the package not generating any debug objects are going to be common at all. I considered mentioning fixing the build to stop capturing, but felt it might be more effort than requested. :) Should probably have mentioned anyway, as was done elsewhere in the thread. In any case I'm thinking to add something like the attached to the man page to try to clarify this. Thanks, Guillem From 1d94da75b63115485ad1247c44f64973ed74339f Mon Sep 17 00:00:00 2001 From: Guillem Jover Date: Wed, 8 Feb 2023 11:26:14 +0100 Subject: [PATCH] man: Add notes about reproducibility properties for path fixing build features Mention that if the build captures the build flags that will make it unreproducible, the traps with trying to disable these flags to get back to a reproducible output. And that the ideal fix is to stop capturing build flags. Prompted-by: Sven Joachim --- man/dpkg-buildflags.pod | 9 + 1 file changed, 9 insertions(+) diff --git a/man/dpkg-buildflags.pod b/man/dpkg-buildflags.pod index 6067d4923..01c14269e 100644 --- a/man/dpkg-buildflags.pod +++ b/man/dpkg-buildflags.pod @@ -622,6 +622,13 @@ This has the effect of removing the build path from any generated file. If both B and B are set, this option takes precedence, because it is a superset of the former. +B: If the build process captures the build flags into the resulting +built objects, that will make the package unreproducible. +And while disabling this option might make some of the objects reproducible +again this would also require disabling B, which might make +any generated debug symbols objects unreproducible. +The ideal fix is to stop capturing build flags. + =item B This setting (enabled by default) adds @@ -632,6 +639,8 @@ set to the top-level directory of the package being built. This has the effect of removing the build path from any generated debug symbols. +B: This feature has similar reproducible properties as B. + =back =head1 ENVIRONMENT -- 2.39.1
Bug#1030785: -ffile-prefix-map option and reproducibility
Thank you all for your answers! Using: DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath makes the package unreproducible in another way that seems difficult to fix. Le 07/02/2023 à 19:12, Mattia Rizzolo a écrit : I actually propose to you to filter out the whole option from being saved. [...] I've gone this way, and managed to make the package reproducible, at least with the build path variation. I will upload the fixed ocaml package when the current batch of related packages waiting in unstable migrates to testing, hopefully in 4 days. Cheers, -- Stéphane
Bug#1030785: -ffile-prefix-map option and reproducibility
On 07/02/2023 20:00, Sven Joachim wrote: On 2023-02-07 17:50 +0100, Guillem Jover wrote: On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote: When building packages, a -ffile-prefix-map option is automatically injected into CFLAGS. Where does it come from? Since when? This is coming from dpkg-buildflags (in this case probably indirectly via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default, and then switched to enabled by default in dpkg 1.20.6 (see #974087). I suspect this was added to improve reproducibility. Ironically, it makes packages that capture this variable non reproducible, since the build path seems to be randomized (has it always been the case? since when?). It is the case of OCaml (see #1030785), and seemingly of R as well (found by grepping in my /etc). I wouldn't be surprised other packages are affected as well. AFAIR this was considered at the time, yes. If the flag is effectively not fixing anything for the set of packages involved, and is in fact actually making them unreproducible when they would not then, you can disable the fixfilepath feature in the reproducible build flags area, via DEB_BUILD_MAINT_OPTIONS. This does not help for packages which capture all build flags and store them in some file in the package (as is the case here). What is the purpose of having the build flags in a file in the .deb? Cheers, Emilio
Bug#1030785: -ffile-prefix-map option and reproducibility
On 2023-02-07 17:50 +0100, Guillem Jover wrote: > On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote: >> When building packages, a -ffile-prefix-map option is automatically injected >> into CFLAGS. Where does it come from? Since when? > > This is coming from dpkg-buildflags (in this case probably indirectly > via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default, > and then switched to enabled by default in dpkg 1.20.6 (see #974087). > >> I suspect this was added to improve reproducibility. Ironically, it makes >> packages that capture this variable non reproducible, since the build path >> seems to be randomized (has it always been the case? since when?). It is the >> case of OCaml (see #1030785), and seemingly of R as well (found by grepping >> in my /etc). I wouldn't be surprised other packages are affected as well. > > AFAIR this was considered at the time, yes. If the flag is effectively > not fixing anything for the set of packages involved, and is in fact > actually making them unreproducible when they would not then, you can > disable the fixfilepath feature in the reproducible build flags area, > via DEB_BUILD_MAINT_OPTIONS. This does not help for packages which capture all build flags and store them in some file in the package (as is the case here). With DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath, dpkg-buildflags falls back to "-fdebug-prefix-map==.", and you have the same problem. If you disable that as well via DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath,-fixdebugpath, the -dbgsym packages will most likely end up unreproducible. Cheers, Sven
Bug#1030785: -ffile-prefix-map option and reproducibility
On Tue, Feb 07, 2023 at 04:41:47PM +0100, Stéphane Glondu wrote:
> When building packages, a -ffile-prefix-map option is automatically injected
> into CFLAGS. Where does it come from? Since when?
>
> I suspect this was added to improve reproducibility. Ironically, it makes
> packages that capture this variable non reproducible, since the build path
> seems to be randomized (has it always been the case? since when?).
The build path has always been randomized since, or at least it has been
for as long as I've been involved in Debian.
> It is the
> case of OCaml (see #1030785), and seemingly of R as well (found by grepping
> in my /etc). I wouldn't be surprised other packages are affected as well.
>
> Is there a way to not get this option? More elegant than explicitly
> filtering it out of CFLAGS in debian/rules...
Besides doing
DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath
I actually propose to you to filter out the whole option from being
saved. I've seen a similar pattern in other packages in the past, and
all of those packages already had a filtering function in place to
remove other gcc flags that make no sense being saved (just looking at:
- 8: const("camlConfig__8"="-O2 -fno-strict-aliasing -fwrapv -pthread
-fPIC -g -O2 -ffile-prefix-map=/build/ocaml-Vq2uKK/ocaml-4.13.1=.
-fstack-protector-strong -Wformat -Werror=format-security");
+ 8: const("camlConfig__8"="-O2 -fno-strict-aliasing -fwrapv -pthread
-fPIC -g -O2 -ffile-prefix-map=/build/ocaml-xz3WL7/ocaml-4.13.1=.
-fstack-protector-strong -Wformat -Werror=format-security");
makes me believe that many options have been stripped out…)
--
regards,
Mattia Rizzolo
GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`.
More about me: https://mapreri.org : :' :
Launchpad user: https://launchpad.net/~mapreri `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia `-
signature.asc
Description: PGP signature
Bug#1030785: -ffile-prefix-map option and reproducibility
Hi! On Tue, 2023-02-07 at 16:41:47 +0100, Stéphane Glondu wrote: > When building packages, a -ffile-prefix-map option is automatically injected > into CFLAGS. Where does it come from? Since when? This is coming from dpkg-buildflags (in this case probably indirectly via debhelper). AFAICS it was added in dpkg 1.19.1 disabled by default, and then switched to enabled by default in dpkg 1.20.6 (see #974087). > I suspect this was added to improve reproducibility. Ironically, it makes > packages that capture this variable non reproducible, since the build path > seems to be randomized (has it always been the case? since when?). It is the > case of OCaml (see #1030785), and seemingly of R as well (found by grepping > in my /etc). I wouldn't be surprised other packages are affected as well. AFAIR this was considered at the time, yes. If the flag is effectively not fixing anything for the set of packages involved, and is in fact actually making them unreproducible when they would not then, you can disable the fixfilepath feature in the reproducible build flags area, via DEB_BUILD_MAINT_OPTIONS. Perhaps even "globally" from a language specific packaging helper or similar? > Is there a way to not get this option? More elegant than explicitly > filtering it out of CFLAGS in debian/rules... See above. I just noticed that several of these build flag features do not have information in the man page about when they got first introduced, so I'll be adding that in dpkg 1.22.x, once development opens up again. Thanks, Guillem
Bug#1030785: -ffile-prefix-map option and reproducibility
Hi, Quoting Stéphane Glondu (2023-02-07 16:41:47) > When building packages, a -ffile-prefix-map option is automatically injected > into CFLAGS. Where does it come from? Since when? probably due to https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/commit/?id=b60c243ba99b8483202a6f6a814476275204fdff which references: https://lists.debian.org/debian-devel/2020/10/msg00222.html https://bugs.debian.org/974087 > I suspect this was added to improve reproducibility. Ironically, it makes > packages that capture this variable non reproducible, since the build path > seems to be randomized (has it always been the case? since when?). It is the > case of OCaml (see #1030785), and seemingly of R as well (found by grepping > in my /etc). I wouldn't be surprised other packages are affected as well. > > Is there a way to not get this option? More elegant than explicitly > filtering it out of CFLAGS in debian/rules... See the man page of dpkg-buildflags -- this might do what you want: export DEB_BUILD_MAINT_OPTIONS=reproducible=-fixfilepath Thanks! cheers, josch
Bug#1030785: -ffile-prefix-map option and reproducibility
Hi, When building packages, a -ffile-prefix-map option is automatically injected into CFLAGS. Where does it come from? Since when? I suspect this was added to improve reproducibility. Ironically, it makes packages that capture this variable non reproducible, since the build path seems to be randomized (has it always been the case? since when?). It is the case of OCaml (see #1030785), and seemingly of R as well (found by grepping in my /etc). I wouldn't be surprised other packages are affected as well. Is there a way to not get this option? More elegant than explicitly filtering it out of CFLAGS in debian/rules... Cheers, -- Stéphane
