Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Hi Vagrant,
>
>   https://pad.sfconservancy.org/p/-NUkgkGAezz0wc7n54el
>
> Feel free to comment or even edit directly.

Thanks. The only substantive change I made was to add a paragraph
regarding packages that filter -fdebug-prefix-map from CFLAGS but
don't do this for -ffile-prefix-map. These will, ironically, become
unreproducible when we enable +fixfilepath as they won't be stripped
from binary packages.

I've made a few aesthetic and non-normative changes (eg. adding some
structure, making it clearer what a maintainer might do, some minor
grammar improvements etc.) but they should not affect the meaning of
your original words.

The only thing that might be missing is some kind of "let us know if
you disagree, otherwise we will go ahead with this change." This is
implicit to a certain degree, but the message could alternatively be
read as a mild FYI.


Kind regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-10-26, Vagrant Cascadian wrote:
> On 2020-10-22, Chris Lamb wrote:
>> Guillem (and all the other dpkg folks), do you see any impediment to
>> raising the question of enabling fixfilepath by default with the
>> Debian community at large?
>>
>>> All of the patches I submitted were merely disable fixfilepath, though
>>> they could all use further exploration to find the root cause and a more
>>> ideal fix, but given time constraints and the small number of affected
>>> packages, I figured this was a reasonable short-term workaround.
>>
>> Thanks, Vagrant. I would agree that is a good workaround for now and
>> probably shouldn't prevent us from making this (IMHO) long-overdue
>> change.
>>
>> Do you have a timeline on when you plan to raise this on debian-devel?
>
> Hope to craft and send it tomorrow after the Reproducible Builds IRC
> meeting.

I wrote it up, but not fast enough for folks living closer to UTC any
time to review:

  https://pad.sfconservancy.org/p/-NUkgkGAezz0wc7n54el

Feel free to comment or even edit directly.

Planning to send it 2020-10-27 between 16:00-18:00 UTC unless I hear
otherwise!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-10-22, Chris Lamb wrote:
> Guillem (and all the other dpkg folks), do you see any impediment to
> raising the question of enabling fixfilepath by default with the
> Debian community at large?
>
>> All of the patches I submitted were merely disable fixfilepath, though
>> they could all use further exploration to find the root cause and a more
>> ideal fix, but given time constraints and the small number of affected
>> packages, I figured this was a reasonable short-term workaround.
>
> Thanks, Vagrant. I would agree that is a good workaround for now and
> probably shouldn't prevent us from making this (IMHO) long-overdue
> change.
>
> Do you have a timeline on when you plan to raise this on debian-devel?

Hope to craft and send it tomorrow after the Reproducible Builds IRC
meeting.

Most of the relevent bugs have been filed with workaround patches, and
will probably follow up with bugs without patches for most of the few
remaining issues.

The pieces are finally falling into place, so yay!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

[moving Lucas to BCC]

Hi Guillem,

> I don't see either an issue with whether the bugs have been filed or
> not, but I think that's just part of the conversation with
> debian-devel at large (which I doubt would be a matter of contention
> there either).

Judging by some emails flying past my inbox, I think Vagrant has filed
these bugs since my previous mail, which I suppose makes things a
little simpler.

Anyway, great, am looking forward to seeing this conversation start on
debian-devel. I agree that there is unlikely to be little contention,
if any.

(Thanks again to Lucas for running the tests. Assuming that you that
no longer need to receive these process updates, I've moved you to
BCC faute de mieux.)


Regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o

Re: Enabling -ffile-prefix-map by default

[Guillem Jover]

Hi!

On Thu, 2020-10-22 at 22:58:02 -, Chris Lamb wrote:
> Guillem (and all the other dpkg folks), do you see any impediment to
> raising the question of enabling fixfilepath by default with the
> Debian community at large?

Sorry, had pending a reply to this but was caught up with other stuff.
And no, I don't, and I don't even see any issue with the procedure
Vagrant was wondering about, as the flags are clearly not supposed to
affect build or run-time performance or memory usage (if they did I'd
consider that a bug in the compiler instead, or in very specific
package instances :). I don't see either an issue with whether the
bugs have been filed or not, but I think that's just part of the
conversation with debian-devel at large (which I doubt would be a
matter of contention there either).

Thanks,
Guillem

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Hi all,

Guillem (and all the other dpkg folks), do you see any impediment to
raising the question of enabling fixfilepath by default with the
Debian community at large?

> All of the patches I submitted were merely disable fixfilepath, though
> they could all use further exploration to find the root cause and a more
> ideal fix, but given time constraints and the small number of affected
> packages, I figured this was a reasonable short-term workaround.

Thanks, Vagrant. I would agree that is a good workaround for now and
probably shouldn't prevent us from making this (IMHO) long-overdue
change.

Do you have a timeline on when you plan to raise this on debian-devel?


Regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-10-15, Vagrant Cascadian wrote:
> On 2020-10-15, Chris Lamb wrote:
>>> At a quick glance, looks to be 33 packages, many of which were already
>>> identified in by our tags:
>>>
>>> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
>>> https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
>>>
>>> I've just added the missing ones, and will follow-up with bugs and
>>> hopefully patches in the coming weeks. It is a pretty reasonable number
>>> of packages to tackle.
>>
>> Vagrant, can you briefly elaborate on your plan of action here? In
>> particular, at what point do you plan to request enabling
>> -ffile-prefix-map distribution-wide? I can think of a number of
>> potential answers, so would be interested to learn your current
>> intentions.
>
> I need to follow-up with bug reports for the packages that aren't
> triggered by clang. Was hoping to have that done by now; it's on my
> priority list... I would guess this is in the ballpark of 10-20 more
> bugs. Of course, anyone can file those bug reports...

Ok, submitted 15 patches for those today:

  
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org=fixfilepath

Thanks for the nudge!


One package is still building (seqan2)...

Other than the clang/llvm related packages, there are still 10 packages
that I couldn't successfully build using my reprotest environment with
or without fixfilepath enabled:

  appstream
  gloo
  kalarmcal
  kcodecs
  kdeclarative
  kitemmodels
  kparts
  libkgapi
  python-kubernetes
  seqan

It might be worth filing bugs for those without patches, just in case
someone is fixing an unrelated FTBFS issue they could maybe also fix
issues around fixfilepath at the same time... thoughts?


> Once llvm-defaults updates the default clang, I think nearly half of the
> FTBFS issues will likely disappear, so don't think this is worth filing
> for those bugs, presuming llvm-defaults gets updated soon.
>
> The stupid fix for the remaining packages is to disable this feature for
> a handful of packages, e.g. in debian/rules:
>
>   DEB_BUILD_MAIN_OPTIONS=reproducible=-fixfilepath,+fixdebugpath
>
> (or maybe just -fixfilepath, depending on how the dpkg patch lands)

All of the patches I submitted were merely disable fixfilepath, though
they could all use further exploration to find the root cause and a more
ideal fix, but given time constraints and the small number of affected
packages, I figured this was a reasonable short-term workaround.


> So, looking at:
>
>   
> https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F
>
> We're nearly ready to take it to debian-devel for comments...
...
> I guess, at risk of a small breach of procedure, it would be worth
> starting the conversation on debian-devel *now* even though the
> remaining bugs aren't yet filed just to get the process started (while
> continuing to file the remaining bugs)... to hopefully make it in time
> for bullseye.

If we filed bugs for all the remaining packages other than the
llvm/clang related issues, I'd say we're in a reasonably good position
to move this forward...


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-10-15, Chris Lamb wrote:
>> At a quick glance, looks to be 33 packages, many of which were already
>> identified in by our tags:
>>
>> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
>> https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
>>
>> I've just added the missing ones, and will follow-up with bugs and
>> hopefully patches in the coming weeks. It is a pretty reasonable number
>> of packages to tackle.
>
> Vagrant, can you briefly elaborate on your plan of action here? In
> particular, at what point do you plan to request enabling
> -ffile-prefix-map distribution-wide? I can think of a number of
> potential answers, so would be interested to learn your current
> intentions.

I need to follow-up with bug reports for the packages that aren't
triggered by clang. Was hoping to have that done by now; it's on my
priority list... I would guess this is in the ballpark of 10-20 more
bugs. Of course, anyone can file those bug reports...

Once llvm-defaults updates the default clang, I think nearly half of the
FTBFS issues will likely disappear, so don't think this is worth filing
for those bugs, presuming llvm-defaults gets updated soon.

The stupid fix for the remaining packages is to disable this feature for
a handful of packages, e.g. in debian/rules:

  DEB_BUILD_MAIN_OPTIONS=reproducible=-fixfilepath,+fixdebugpath

(or maybe just -fixfilepath, depending on how the dpkg patch lands)

More ideally just disabling it in the test suites somehow, as that's the
primary use-case that would ever matter.


So, looking at:

  
https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F

We're nearly ready to take it to debian-devel for comments...

Some remaining points:

  * For non-warning flags, a comparison of the build logs to see the
memory and build time difference if these might seem relevant
(sbuild should provide those).

I haven't compared the build logs to get this data yet, but I would be
surprised if it made any significant difference. I'd like to invoke the
"these are not relevent" clause.

  * For flags that change run-time semantics, ideally an additional run
of the autopkgtest for packages that ship them (although this cannot
be deemed conclusive as our coverage is not great yet).

I don't believe it changes run-time semantics in any way that could
matter.


I guess, at risk of a small breach of procedure, it would be worth
starting the conversation on debian-devel *now* even though the
remaining bugs aren't yet filed just to get the process started (while
continuing to file the remaining bugs)... to hopefully make it in time
for bullseye.


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Hi all,

> At a quick glance, looks to be 33 packages, many of which were already
> identified in by our tags:
>
> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
> https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html
>
> I've just added the missing ones, and will follow-up with bugs and
> hopefully patches in the coming weeks. It is a pretty reasonable number
> of packages to tackle.

Vagrant, can you briefly elaborate on your plan of action here? In
particular, at what point do you plan to request enabling
-ffile-prefix-map distribution-wide? I can think of a number of
potential answers, so would be interested to learn your current
intentions.

Thanks again for running these tests, Lucas.


Kind regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-09-27, Lucas Nussbaum wrote:
> On 07/09/20 at 20:42 -0700, Vagrant Cascadian wrote:
>> On 2020-08-31, Lucas Nussbaum wrote:
>> > On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
>> >> On 2020-08-31, Lucas Nussbaum wrote:
>> >> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
>> >> >> I'm looking into proposing to enable dpkg's
>> >> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
>> >> ...
>> >> >> It would be nice to try an archive-wide rebuild with +fixfilepath
>> >> >> enabled and on failure,
...
>> Proposed merge request adding a script that does just that:
>> 
>>   https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
>
> In http://qa-logs.debian.net/2020/09/26.fixfilepath/ you can find:
>
> 00cmp.fixfilepath.only-fails-with-fixfilepath.txt: that's the list of
> packages that fail with fixfilepath, but don't fail without it.

Huge thanks!

At a quick glance, looks to be 33 packages, many of which were already
identified in by our tags:

  
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
  
https://tests.reproducible-builds.org/debian/issues/unstable/ffile_prefix_map_passed_to_clang_issue.html

I've just added the missing ones, and will follow-up with bugs and
hopefully patches in the coming weeks. It is a pretty reasonable number
of packages to tackle.


So great to have this systematically tested so I can feel more confident
of the results; it was still a bit unclear just looking at the
reproducible builds test infrastructure, since we add so many other
variables in our tests.

Thanks, thanks, and more thanks!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Lucas Nussbaum]

Hi,

On 07/09/20 at 20:42 -0700, Vagrant Cascadian wrote:
> On 2020-08-31, Lucas Nussbaum wrote:
> > On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
> >> On 2020-08-31, Lucas Nussbaum wrote:
> >> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
> >> >> I'm looking into proposing to enable dpkg's
> >> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
> >> ...
> >> >> It would be nice to try an archive-wide rebuild with +fixfilepath
> >> >> enabled and on failure,
> >> >
> >> > Could you provide a dpkg package in a private repo (or in experimental)
> >> > with that enabled by default? And ideally a script similar to
> >> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
> >> > to install that package?
> >> 
> >> I could build and upload an updated dpkg somewhere...
> >> 
> >> I see that modes/clang10 mangles some files directly, and while this
> >> makes me cringe a bit, what about instead adding a
> >> modes/dpkg-fixfilepath doing something like:
> >> 
> >>   sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
> >> /usr/share/perl5/Dpkg/Vendor/Debian.pm
> >> 
> >> Is this an acceptible way forward?
> >
> > Ah, yes, totally!
> 
> Proposed merge request adding a script that does just that:
> 
>   https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9

In http://qa-logs.debian.net/2020/09/26.fixfilepath/ you can find:

00cmp.fixfilepath.only-fails-with-fixfilepath.txt: that's the list of
packages that fail with fixfilepath, but don't fail without it.

00res.fixfilepath.txt: the full list of built packages
00cmp.fixfilepath.txt: the differences with a normal build

... and the logs for the packages listed in the first file.

Let me know if you need something else.

Lucas


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Lucas Nussbaum]

On 27/09/20 at 18:37 +0200, Lucas Nussbaum wrote:
> Hi,
> 
> On 07/09/20 at 20:42 -0700, Vagrant Cascadian wrote:
> > On 2020-08-31, Lucas Nussbaum wrote:
> > > On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
> > >> On 2020-08-31, Lucas Nussbaum wrote:
> > >> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
> > >> >> I'm looking into proposing to enable dpkg's
> > >> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
> > >> ...
> > >> >> It would be nice to try an archive-wide rebuild with +fixfilepath
> > >> >> enabled and on failure,
> > >> >
> > >> > Could you provide a dpkg package in a private repo (or in experimental)
> > >> > with that enabled by default? And ideally a script similar to
> > >> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
> > >> > to install that package?
> > >> 
> > >> I could build and upload an updated dpkg somewhere...
> > >> 
> > >> I see that modes/clang10 mangles some files directly, and while this
> > >> makes me cringe a bit, what about instead adding a
> > >> modes/dpkg-fixfilepath doing something like:
> > >> 
> > >>   sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
> > >> /usr/share/perl5/Dpkg/Vendor/Debian.pm
> > >> 
> > >> Is this an acceptible way forward?
> > >
> > > Ah, yes, totally!
> > 
> > Proposed merge request adding a script that does just that:
> > 
> >   https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
> 
> In http://qa-logs.debian.net/2020/09/26.fixfilepath/ you can find:
> 
> 00cmp.fixfilepath.only-fails-with-fixfilepath.txt: that's the list of
> packages that fail with fixfilepath, but don't fail without it.
> 
> 00res.fixfilepath.txt: the full list of built packages
> 00cmp.fixfilepath.txt: the differences with a normal build
> 
> ... and the logs for the packages listed in the first file.
> 
> Let me know if you need something else.

00res.fixfilepath.only-failures.txt: that's the list of failures in
00cmp.fixfilepath.only-fails-with-fixfilepath.txt, with an indication of
the failure. (to avoid going through the logs)

Lucas


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Lucas Nussbaum]

Hi,

On 23/09/20 at 14:39 +0100, Chris Lamb wrote:
> Vagrant Cascadian wrote:
> 
> > Proposed merge request adding a script that does just that:
> > 
> >   https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9
> 
> 2+ weeks on, any update on this? I keep coming across packages that
> would be fixed by fixfilepath.

Sorry, the start of the academic year, + my newly born second child,
hurt quite badly. I haven't been able to touch any Debian-related stuff
recently, but this is still near the top of my To-Do list when I get
time again.

Lucas

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Vagrant Cascadian wrote:

> Proposed merge request adding a script that does just that:
> 
>   https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9

2+ weeks on, any update on this? I keep coming across packages that
would be fixed by fixfilepath.


-- 
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 
⬊   ⬋
  o

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-08-31, Lucas Nussbaum wrote:
> On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
>> On 2020-08-31, Lucas Nussbaum wrote:
>> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
>> >> I'm looking into proposing to enable dpkg's
>> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
>> ...
>> >> It would be nice to try an archive-wide rebuild with +fixfilepath
>> >> enabled and on failure,
>> >
>> > Could you provide a dpkg package in a private repo (or in experimental)
>> > with that enabled by default? And ideally a script similar to
>> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
>> > to install that package?
>> 
>> I could build and upload an updated dpkg somewhere...
>> 
>> I see that modes/clang10 mangles some files directly, and while this
>> makes me cringe a bit, what about instead adding a
>> modes/dpkg-fixfilepath doing something like:
>> 
>>   sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
>> /usr/share/perl5/Dpkg/Vendor/Debian.pm
>> 
>> Is this an acceptible way forward?
>
> Ah, yes, totally!

Proposed merge request adding a script that does just that:

  https://salsa.debian.org/lucas/collab-qa-tools/-/merge_requests/9


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Holger Levsen]

On Mon, Aug 31, 2020 at 09:35:52AM -0700, Vagrant Cascadian wrote:
> > Could you provide a dpkg package in a private repo (or in experimental)
> > with that enabled by default? And ideally a script similar to
> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
> > to install that package?
> I could build and upload an updated dpkg somewhere...

I wouldn't mind doing such a test using
https://tests.reproducible-builds.org/debian/repository/debian/
as long as such a dpkg package would be provided for all four tested
archs...

I'd be happy to help with building it and uploading it there.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

Some people say that the climate crisis  is something that we all have created,
but  that is not true,  because if everyone is guilty  then no one is to blame.
And someone is to blame.  Some people, some companies,  some decision-makers in
particular, have known exactly what priceless values they have been sacrificing
to continue making unimaginable amounts of money. (Greta Thunberg)


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-08-31, Lucas Nussbaum wrote:
> On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
>> On 2020-08-31, Lucas Nussbaum wrote:
>> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
>> >> I've appreciated all the work you've been doing with archive-wide
>> >> rebuilds of Debian over the years!
>> >> 
>> >> I'm looking into proposing to enable dpkg's
>> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
>> ...
>> >> It would be nice to try an archive-wide rebuild with +fixfilepath
>> >> enabled and on failure,
>> >
>> > Could you provide a dpkg package in a private repo (or in experimental)
>> > with that enabled by default? And ideally a script similar to
>> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
>> > to install that package?
>> 
>> I could build and upload an updated dpkg somewhere...
>> 
>> I see that modes/clang10 mangles some files directly, and while this
>> makes me cringe a bit, what about instead adding a
>> modes/dpkg-fixfilepath doing something like:
>> 
>>   sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
>> /usr/share/perl5/Dpkg/Vendor/Debian.pm
>> 
>> Is this an acceptible way forward?
>
> Ah, yes, totally!

Great, that seems much simpler!

I'm not sure what else is needed to propose a merge request; haven't
quite wrapped my head around all the various parts of collab-qa-tools
that might also need changes to test this.


> Unrelated question: those rebuilds would only be for arch:any packages,
> right? I can't think of a reason to rebuild arch:all for this test, but
> maybe I'm missing something.

Mostly not, but there are some arch:all packages which use
cross-compilers to build things like firmware for foreign architectures
(e.g. u-boot-qemu), so it might be best to include arch:all builds as
well.


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Lucas Nussbaum]

On 31/08/20 at 09:35 -0700, Vagrant Cascadian wrote:
> Thanks for the quick response! More in-line below.
> 
> On 2020-08-31, Lucas Nussbaum wrote:
> > On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
> >> I've appreciated all the work you've been doing with archive-wide
> >> rebuilds of Debian over the years!
> >> 
> >> I'm looking into proposing to enable dpkg's
> >> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
> ...
> >> It would be nice to try an archive-wide rebuild with +fixfilepath
> >> enabled and on failure,
> >
> > Could you provide a dpkg package in a private repo (or in experimental)
> > with that enabled by default? And ideally a script similar to
> > https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
> > to install that package?
> 
> I could build and upload an updated dpkg somewhere...
> 
> I see that modes/clang10 mangles some files directly, and while this
> makes me cringe a bit, what about instead adding a
> modes/dpkg-fixfilepath doing something like:
> 
>   sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
> /usr/share/perl5/Dpkg/Vendor/Debian.pm
> 
> Is this an acceptible way forward?

Ah, yes, totally!

Unrelated question: those rebuilds would only be for arch:any packages,
right? I can't think of a reason to rebuild arch:all for this test, but
maybe I'm missing something.

Lucas


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

Thanks for the quick response! More in-line below.

On 2020-08-31, Lucas Nussbaum wrote:
> On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
>> I've appreciated all the work you've been doing with archive-wide
>> rebuilds of Debian over the years!
>> 
>> I'm looking into proposing to enable dpkg's
>> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default
...
>> It would be nice to try an archive-wide rebuild with +fixfilepath
>> enabled and on failure,
>
> Could you provide a dpkg package in a private repo (or in experimental)
> with that enabled by default? And ideally a script similar to
> https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
> to install that package?

I could build and upload an updated dpkg somewhere...

I see that modes/clang10 mangles some files directly, and while this
makes me cringe a bit, what about instead adding a
modes/dpkg-fixfilepath doing something like:

  sed -i -e 's,fixfilepath => 0,fixfilepath => 1,g' 
/usr/share/perl5/Dpkg/Vendor/Debian.pm

Is this an acceptible way forward? If not, I can prepare a repository
with a modifed dpkg and follow the modes/gcc10 example.


>> automatically retry with -fixfilepath
>
> That would be the current default, right? So a rebuild with vanilla
> unstable would work as well?

Yes.



live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Lucas Nussbaum]

Hi Vagrant,

On 30/08/20 at 19:15 -0700, Vagrant Cascadian wrote:
> Hi Lucas!
> 
> I've appreciated all the work you've been doing with archive-wide
> rebuilds of Debian over the years!
> 
> I'm looking into proposing to enable dpkg's
> DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default, further
> background in this thread:
> 
>   
> https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20200601/012375.html
> 
> We would like to figure out how many packages it causes to FTBFS
> (usually test suites expecting full paths); we've identified a small
> handful through tests.reproducible-builds.org:
> 
>   
> https://tests.reproducible-builds.org/debian/issues/bullseye/ftbfs_due_to_f-file-prefix-map_issue.html
>   
> https://tests.reproducible-builds.org/debian/issues/bullseye/ffile_prefix_map_passed_to_clang_issue.html
> 
> Those issues were identified through manually checking the build logs,
> and our reproducible builds infrastructure is not great for isolating
> out the effects of specific changes as each build has several variations
> tested.
> 
> It would be nice to try an archive-wide rebuild with +fixfilepath
> enabled and on failure,

Could you provide a dpkg package in a private repo (or in experimental)
with that enabled by default? And ideally a script similar to
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/gcc10
to install that package?

> automatically retry with -fixfilepath

That would be the current default, right? So a rebuild with vanilla
unstable would work as well?

> , so that
> we can flag which packages build fine without too much worry about false
> positives and in an automated fashion.
> 
> 
> I've never experimented with anything on an archive-wide scale before,
> so would appreciate any ideas, tooling or other help you might be able
> to offer. Thanks!

- Lucas


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

Hi Lucas!

I've appreciated all the work you've been doing with archive-wide
rebuilds of Debian over the years!

I'm looking into proposing to enable dpkg's
DEB_BUILD_OPTIONS=reproducible=+fixfilepath by default, further
background in this thread:

  
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20200601/012375.html

We would like to figure out how many packages it causes to FTBFS
(usually test suites expecting full paths); we've identified a small
handful through tests.reproducible-builds.org:

  
https://tests.reproducible-builds.org/debian/issues/bullseye/ftbfs_due_to_f-file-prefix-map_issue.html
  
https://tests.reproducible-builds.org/debian/issues/bullseye/ffile_prefix_map_passed_to_clang_issue.html

Those issues were identified through manually checking the build logs,
and our reproducible builds infrastructure is not great for isolating
out the effects of specific changes as each build has several variations
tested.

It would be nice to try an archive-wide rebuild with +fixfilepath
enabled and on failure, automatically retry with -fixfilepath, so that
we can flag which packages build fine without too much worry about false
positives and in an automated fashion.


I've never experimented with anything on an archive-wide scale before,
so would appreciate any ideas, tooling or other help you might be able
to offer. Thanks!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-06-03, Vagrant Cascadian wrote:
> On 2020-06-03, Benjamin Barenblat wrote:
>> Related discussion [2] concluded that the option
>> should be disabled by default and enabled after Reproducible Builds had
>> trialed it for a while. Having enabled it in some of my recent
>> packaging, I’m curious whether it may be time to switch this flag to be
>> enabled by default. What do you think?
>
> It helps maybe hundreds of packages

I can't prove it isn't fixed in some other way, but my guess is this is
in the ballpark of 500-700, based on the currently reproducible builds
from packages marked with this issue:

  
https://tests.reproducible-builds.org/debian/issues/unstable/gcc_captures_build_path_issue.html


> and we've identified a small (16 at the moment) number packages for
> which it causes issues:
>
>   
> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
>
> But most of those packages still FTBFS on bullseye and buster where we
> don't set the flag (or vary the build path)...

On closer look, I was mistaken; tests.reproducible-builds.org sets
reproducible=+all on all distributions... even though we only vary the
build path in unstable and experimental.

We'd be able to get more accurate data if we remove the flag for
bullseye and manually schedule all of those packages, at least
temporarily... anyone have concerns with doing that and waiting for the
builds to complete before bringing this up on debian-devel?


> There may also be others that fail but haven't been identified.
>
> I recall this thread where it broke test suites of some packages:
>
>   
> https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html

Many of the identified ones are named k*, so probably directly
related...


> I believe maintainers can override this flag in their packages if
> needed? If so, I'd be inclined to explore setting it by default!

Overriding the flags in 16 packages doesn't sound like a *huge* effort
to me, although though from a quick glance, many of them from KDE
related teams:

  https://udd.debian.org/dmd/?debian-qt-kde%40lists.debian.org
  https://udd.debian.org/dmd/?pkg-kde-extras%40lists.alioth.debian.org


live well,
  vagrant


signature.asc
Description: PGP signature

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Hi Guillem,

> > I recall this thread where it broke test suites of some packages:
> >
> >   
> > https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html
>
> But I assume this would have been detected while building?

It would have been detected during build, yes. (It is unlikely,
although not technically impossible, that the adjustment of __FILE__
can affect the runtime behaviour of a package yet not at build time.)

> Most of which have already been done now in the repro infra, I think? So
> it'd just mostly be a matter of someone sending a note to d-d proposing
> the change, providing the numbers, waiting a bit in case someone has
> concerns, then I'm happy to flip the default. :)

Vagrant, could you go ahead with this?


Regards,

--
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org  chris-lamb.co.uk
   `-

Re: Enabling -ffile-prefix-map by default

[Guillem Jover]

Hi!

On Wed, 2020-06-03 at 11:59:32 -0700, Vagrant Cascadian wrote:
> On 2020-06-03, Benjamin Barenblat wrote:
> > Related discussion [2] concluded that the option
> > should be disabled by default and enabled after Reproducible Builds had
> > trialed it for a while. Having enabled it in some of my recent
> > packaging, I’m curious whether it may be time to switch this flag to be
> > enabled by default. What do you think?
> 
> It helps maybe hundreds of packages, and we've identified a small (16 at
> the moment) number packages for which it causes issues:
> 
>   
> https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

Have these got bugs filed?

> But most of those packages still FTBFS on bullseye and buster where we
> don't set the flag (or vary the build path)...

Ah OK, so it does not look bad at all. :)

> There may also be others that fail but haven't been identified.
> 
> I recall this thread where it broke test suites of some packages:
> 
>   
> https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html

But I assume this would have been detected while building?

On Thu, 2020-06-04 at 22:58:05 -, Chris Lamb wrote:
> Vagrant Cascadian wrote:
> > I believe maintainers can override this flag in their packages if
> > needed? If so, I'd be inclined to explore setting it by default!

As I mentioned last time, I've got no concerns with enabling this. I
think we should just follow the usual steps described at:

  


Most of which have already been done now in the repro infra, I think? So
it'd just mostly be a matter of someone sending a note to d-d proposing
the change, providing the numbers, waiting a bit in case someone has
concerns, then I'm happy to flip the default. :)

> Regarding maintainer control over this I have two remarks to make. To
> answer Vagrant's question of whether they can override this (ie.
> disable it in the case of it becoming a default) then unless I am
> missing something that would be possible via the usual dpkg-buildflags
> mechanism.

Indeed.

> Secondly (and related to the previous remark as well as being another
> reason to enable this soon) many maintainers are explicitly enabling
> the fixfilepath feature area of dpkg-buildflags right now as we
> incentivise making their package appear reproducible on the
> Reproducible Build's testing framework.
> 
> My point here is that if were to enable this flag anyway, doing it
> before yet more packages add this boilerplate would avoid even more
> cluttering of debian/rules files. (Many packages are still parsing
> debian/changelog and manually exporting SOURCE_DATE_EPOCH.)

Yes, adding this to packages to then remove when the default change,
does not seem like effort well spent, so let's get this moving. :)

Thanks,
Guillem

Re: Enabling -ffile-prefix-map by default

[Chris Lamb]

Vagrant Cascadian wrote:

> I believe maintainers can override this flag in their packages if
> needed? If so, I'd be inclined to explore setting it by default!

I would be very much in favour of enabling this at the earliest
opportunity.

Regarding maintainer control over this I have two remarks to make. To
answer Vagrant's question of whether they can override this (ie.
disable it in the case of it becoming a default) then unless I am
missing something that would be possible via the usual dpkg-buildflags
mechanism.

Secondly (and related to the previous remark as well as being another
reason to enable this soon) many maintainers are explicitly enabling
the fixfilepath feature area of dpkg-buildflags right now as we
incentivise making their package appear reproducible on the
Reproducible Build's testing framework.

My point here is that if were to enable this flag anyway, doing it
before yet more packages add this boilerplate would avoid even more
cluttering of debian/rules files. (Many packages are still parsing
debian/changelog and manually exporting SOURCE_DATE_EPOCH.)


Regards,

--
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org  chris-lamb.co.uk
   `-

Re: Enabling -ffile-prefix-map by default

[Vagrant Cascadian]

On 2020-06-03, Benjamin Barenblat wrote:
> About two years ago, Guillem Jover added support for -ffile-prefix-map
> to dpkg [1] (thanks!).

Wow, hard to believe that was nearly two years ago!

Thanks for bringing this up!


> Related discussion [2] concluded that the option
> should be disabled by default and enabled after Reproducible Builds had
> trialed it for a while. Having enabled it in some of my recent
> packaging, I’m curious whether it may be time to switch this flag to be
> enabled by default. What do you think?

It helps maybe hundreds of packages, and we've identified a small (16 at
the moment) number packages for which it causes issues:

  
https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

But most of those packages still FTBFS on bullseye and buster where we
don't set the flag (or vary the build path)...

There may also be others that fail but haven't been identified.

I recall this thread where it broke test suites of some packages:

  
https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html


I believe maintainers can override this flag in their packages if
needed? If so, I'd be inclined to explore setting it by default!


live well,
  vagrant


signature.asc
Description: PGP signature