Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Holger Levsen]

On Sat, Nov 17, 2018 at 04:51:21PM +, Steve McIntyre wrote:
> >> The way I see this is, that building locally is still supported.
> >this is an important, unfixed bug.
> for upload to the archive, agreed. However, we still support people
> building packages for their own use, of course. And they shouldn't
> fail unexpectedly due to merged-/usr.

agreed & thanks for pointing this out.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Steve McIntyre]

[ Trimming the large CC list a bit... ]

On Sat, Nov 17, 2018 at 01:05:23PM +, Holger Levsen wrote:
>hi,
>
>while I don't really disagree with what you said... I also think:
>
>On Sat, Nov 17, 2018 at 02:00:03PM +0100, Michael Biebl wrote:
>> The way I see this is, that building locally is still supported.
>
>this is an important, unfixed bug.

for upload to the archive, agreed. However, we still support people
building packages for their own use, of course. And they shouldn't
fail unexpectedly due to merged-/usr.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"This dress doesn't reverse." -- Alden Spiess

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Holger Levsen]

hi,

while I don't really disagree with what you said... I also think:

On Sat, Nov 17, 2018 at 02:00:03PM +0100, Michael Biebl wrote:
> The way I see this is, that building locally is still supported.

this is an important, unfixed bug.

> We can binNMU those packages in both cases to have working packages again.

and this is a workaround for the above bug.

(sorry to add nothing new, but if one is spending too much time with
workarounds it might be useful to remind this once in a while...)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Michael Biebl]

Am 17.11.18 um 12:41 schrieb Adrian Bunk:

> This would be a huge regression, buildds aren't the only place where 
> packages get built.
> 
> It doesn't sounds right that we suddenly start declaring it "misbuilt" 
> when a user builds a package locally with dpkg-buildpackage as has been 
> working for the past 25 years.
> 
> I am not aware of any real benefits of merged /usr for users, so let's 
> go back from "How can we mitigate some regressions of merged /usr?" to 
> "Can we make merged /usr working in buster without causing any
>  regressions at all?".

Of course there are benefits for the users, otherwise we wouldn't be
doing it. That said, I don't think this bug report is the right place to
discuss this.

The way I see this is, that building locally is still supported.
Some few packages will be affected by this and be misbuilt a in
merged-usr environment.
That is not too different from packages being buggy because they have
been built locally and e.g. picked up a library dependency from say
experimental or some (unwanted) package features where turned on because
./configure auto-detected a locally installed libray.
We can binNMU those packages in both cases to have working packages again.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Adrian Bunk]

On Fri, Nov 09, 2018 at 12:26:15PM +, Simon McVittie wrote:
> On Fri, 09 Nov 2018 at 06:31:00 +, Niels Thykier wrote:
> > As I understand it, what this question effectively ends up being is "Do
> > we commit to supporting merged-/usr in Buster in all source packages
> > (making such bugs RC) OR do we rollback the default merged-/usr in
> > debootstrap (making them non-RC)?".
> 
> Well, there's a middle ground, which I think would be a good short
> term plan: we can ensure that the official buildds build packages in a
> non-merged-/usr environment (for example by backporting a fix for #913228
> to the packages used on those buildds), which means misbuilt binary
> packages uploaded by maintainers remain potentially RC (as systemd's
> #843433 was), but source packages that would theoretically be misbuilt
> in a merged-/usr environment (like quilt #913226) don't necessarily have
> to be treated as RC.
>...

This would be a huge regression, buildds aren't the only place where 
packages get built.

It doesn't sounds right that we suddenly start declaring it "misbuilt" 
when a user builds a package locally with dpkg-buildpackage as has been 
working for the past 25 years.

I am not aware of any real benefits of merged /usr for users, so let's 
go back from "How can we mitigate some regressions of merged /usr?" to 
"Can we make merged /usr working in buster without causing any
 regressions at all?".

Either we fully support merged /usr everywhere, or merged /usr should 
not be supported in buster and the usrmerge package removed from buster 
as has been done in #863965 for stretch.

> smcv

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Simon McVittie]

On Fri, 09 Nov 2018 at 06:31:00 +, Niels Thykier wrote:
> As I understand it, what this question effectively ends up being is "Do
> we commit to supporting merged-/usr in Buster in all source packages
> (making such bugs RC) OR do we rollback the default merged-/usr in
> debootstrap (making them non-RC)?".

Well, there's a middle ground, which I think would be a good short
term plan: we can ensure that the official buildds build packages in a
non-merged-/usr environment (for example by backporting a fix for #913228
to the packages used on those buildds), which means misbuilt binary
packages uploaded by maintainers remain potentially RC (as systemd's
#843433 was), but source packages that would theoretically be misbuilt
in a merged-/usr environment (like quilt #913226) don't necessarily have
to be treated as RC.

I've sent a proposed patch to #913228.

smcv

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Niels Thykier]

Simon McVittie:
> Package: release.debian.org
> Severity: normal
> User: m...@linux.it
> Usertags: usrmerge
> 
> debootstrap in >= buster produces merged-/usr chroots for buster and sid
> by default. Some packages are misbuilt in such chroots: the only concrete
> example I have that is currently open is 
> in quilt, in which a package built in a merged-/usr environment will
> work on merged-/usr systems but won't work (at all) on a non-merged-/usr
> system.
> 
> There are almost certainly others; [...]
> 
> If binary packages in the archive don't work on a non-merged-/usr system,
> is that a RC bug in the binary packages? [...]
> 
> smcv
> 

Hi Simon,

Thanks for bringing the issue to our attention.

TL;DR summary:

  Can we effectively/measurably answer "which packages are affected?"
  (or "when do we know that all packages work when built-in in a
   merged-/usr?")?

As I understand it, what this question effectively ends up being is "Do
we commit to supporting merged-/usr in Buster in all source packages
(making such bugs RC) OR do we rollback the default merged-/usr in
debootstrap (making them non-RC)?".

To answer such a question at this point in the release cycle, we need to
understand the number of affected packages to tell whether it is
realistic to complete this by the freeze (regardless of which "part" of
the freeze we choose as deadline for it).

In the absence of a clear way to define "when is everything fixed", I
gut feeling is that we might be opening pandora's box right before a
release.

Thanks,
~Niels

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Holger Levsen]

On Thu, Nov 08, 2018 at 05:53:59PM +, Simon McVittie wrote:
> On Thu, 08 Nov 2018 at 15:50:48 +, Holger Levsen wrote:
> > On Thu, Nov 08, 2018 at 04:31:32PM +0100, Michael Biebl wrote:
> > > Wasn't there the idea to utilise the reproducible-build effort to detect
> > > binaries that differ depending on whether they were built in merged-usr
> > > environment or not?
> 
> 
> 
> > however, I would prefer a way which would prevent such
> > buggy binary packages from entering the archive in the first place...
> 
> Do you have a way in mind?
 
source only uploads, for one.

> Are you saying that buildds should stick to unmerged /usr until we are
> more confident that merged /usr buildds will not misbuild packages, as
> I suggested? Or something else?

that's probably something which would help a bit, yes. (but its of
course not enough.)

> For maintainer uploads, there's a lintian tag that would have caught

can we have autorejects based on that?


As said, I'll happily merge a fix for #901473, I just think it's
fundamentally wrong to use a CI system to detect bugs done by manual
mistakes if such mistakes can be systematically avoided. also these bugs
still need to be manually filed.

(and because jenkins.d.n had severe issues the last days which I only
managed to address today I'm a bit reluctant to merge that fix right now
or probably even tomorrow. give us two days of nicely working jenkins
again, first, please :)


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Michael Biebl]

Am 08.11.18 um 19:03 schrieb Michael Biebl:
> I'm basically with you on this.
> I think we should stick with unmerged-usr on the buildds as the
> resulting binaries are more likely to work with both merged and unmerged
> usr setups (at least for the time being).

To be clear: I don't think sticking with unmerged-usr on the buildds is
a viable long-term strategy if we want to embrace merged-usr (which I'm
convinced we should).

This should only be a temporary measure, until we have a better idea of
the impact on the archive.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Michael Biebl]

Am 08.11.18 um 18:53 schrieb Simon McVittie:
> On Thu, 08 Nov 2018 at 15:50:48 +, Holger Levsen wrote:
>> On Thu, Nov 08, 2018 at 04:31:32PM +0100, Michael Biebl wrote:
>>> Wasn't there the idea to utilise the reproducible-build effort to detect
>>> binaries that differ depending on whether they were built in merged-usr
>>> environment or not?
> 
> 
> 
>> however, I would prefer a way which would prevent such
>> buggy binary packages from entering the archive in the first place...
> 
> Do you have a way in mind?
> 
> Are you saying that buildds should stick to unmerged /usr until we are
> more confident that merged /usr buildds will not misbuild packages, as
> I suggested? Or something else?

I'm basically with you on this.
I think we should stick with unmerged-usr on the buildds as the
resulting binaries are more likely to work with both merged and unmerged
usr setups (at least for the time being).
If we made merged-usr the default, even for upgraded systems, I wouldn't
worry as much. The current plan though afaik is to only have merged-usr
for newly deployed systems.

I'm well aware that we don't enforce source-only uploads yet, i.e.
rebuild everything. So we still might have binaries which are built
locally in a merged-usr setup. So this would only minimize the risk.

That's why I hope that we can more systematically detect any such issues
via reproducible-builds.

Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Simon McVittie]

On Thu, 08 Nov 2018 at 15:50:48 +, Holger Levsen wrote:
> On Thu, Nov 08, 2018 at 04:31:32PM +0100, Michael Biebl wrote:
> > Wasn't there the idea to utilise the reproducible-build effort to detect
> > binaries that differ depending on whether they were built in merged-usr
> > environment or not?



> however, I would prefer a way which would prevent such
> buggy binary packages from entering the archive in the first place...

Do you have a way in mind?

Are you saying that buildds should stick to unmerged /usr until we are
more confident that merged /usr buildds will not misbuild packages, as
I suggested? Or something else?

For maintainer uploads, there's a lintian tag that would have caught
the quilt bug (if maintainers run lintian before uploading, which they
should but perhaps don't), but the systemd bug was probably not feasible
to catch with static checks like that.

On Thu, 08 Nov 2018 at 16:21:07 +0100, Michael Biebl wrote:
> A more recent issue I've encountered is with meson not handling RUNPATH
> correctly on merged-usr systems:
> https://github.com/mesonbuild/meson/issues/4392

This seems like another indication that we are not yet ready for
merged-/usr buildds.

smcv

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Holger Levsen]

On Thu, Nov 08, 2018 at 04:31:32PM +0100, Michael Biebl wrote:
> Wasn't there the idea to utilise the reproducible-build effort to detect
> binaries that differ depending on whether they were built in merged-usr
> environment or not? If that is feasible, this would allow for a less
> disruptive way then to rely on finding those issues by "breaking" unstable.

(see the bug report linked in the initial mail.)

we could (*) do that, however, I would prefer a way which would prevent such
buggy binary packages from entering the archive in the first place...


(*) and should


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc
Description: PGP signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Michael Biebl]

Am 08.11.18 um 16:21 schrieb Simon McVittie:
> On Thu, 08 Nov 2018 at 14:59:29 +0100, Mattia Rizzolo wrote:
>> On Thu, Nov 08, 2018 at 01:44:14PM +, Simon McVittie wrote:
>>> I've opened sbuild-createchroot bug 
>>> suggesting that it should create non-merged-/usr chroots to sidestep
>>> this for buildd builds, but there are many other ways for maintainers
>>> and testers to build binaries.
>>
>> Yes, this is not a solution in any way.
> 
> Of course not, but it might be a good idea anyway, to mitigate the class
> of bugs that resemble #843433 and #913226 (source packages that sometimes
> produce broken binaries are obviously bad, but they're less immediately
> disruptive than the binaries in the archive actually being broken).
> 
> Or are you advocating deliberately breaking buildd-built packages that
> have bugs similar to #843433 and #913226, as a way to detect them?

Wasn't there the idea to utilise the reproducible-build effort to detect
binaries that differ depending on whether they were built in merged-usr
environment or not? If that is feasible, this would allow for a less
disruptive way then to rely on finding those issues by "breaking" unstable.

Regards,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Simon McVittie]

On Thu, 08 Nov 2018 at 14:59:29 +0100, Mattia Rizzolo wrote:
> On Thu, Nov 08, 2018 at 01:44:14PM +, Simon McVittie wrote:
> > I've opened sbuild-createchroot bug 
> > suggesting that it should create non-merged-/usr chroots to sidestep
> > this for buildd builds, but there are many other ways for maintainers
> > and testers to build binaries.
> 
> Yes, this is not a solution in any way.

Of course not, but it might be a good idea anyway, to mitigate the class
of bugs that resemble #843433 and #913226 (source packages that sometimes
produce broken binaries are obviously bad, but they're less immediately
disruptive than the binaries in the archive actually being broken).

Or are you advocating deliberately breaking buildd-built packages that
have bugs similar to #843433 and #913226, as a way to detect them?

smcv

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Michael Biebl]

Am 08.11.18 um 14:44 schrieb Simon McVittie:

> There are almost certainly others; most of them are probably similar to
> the quilt bug, where the absolute path to an executable on the build
> system ends up in the package, but to support non-merged-/usr systems
> we need to canonicalize that path to the version that would work on the
> non-merged-/usr systems. For instance, src:systemd used to have a similar
> bug .

A more recent issue I've encountered is with meson not handling RUNPATH
correctly on merged-usr systems:
https://github.com/mesonbuild/meson/issues/4392

I haven't filed a corresponding downstream bug report (yet).
Would probably make sense to have such a bug for tracking this in Debian.

Regards,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Mattia Rizzolo]

On Thu, Nov 08, 2018 at 01:44:14PM +, Simon McVittie wrote:
> debootstrap in >= buster produces merged-/usr chroots for buster and sid
> by default. Some packages are misbuilt in such chroots: the only concrete
> example I have that is currently open is 
> in quilt, in which a package built in a merged-/usr environment will
> work on merged-/usr systems but won't work (at all) on a non-merged-/usr
> system.

Just a thing: consider that yesterday Julien Cristau updated the buildd
to use debootstrap from stretch-backports.  I just so happens that the
backport is outdated right now, but this also means the buildd will use
merged-usr chroots once somebody updates the debootstrap backports.

> I've opened sbuild-createchroot bug 
> suggesting that it should create non-merged-/usr chroots to sidestep
> this for buildd builds, but there are many other ways for maintainers
> and testers to build binaries.

Yes, this is not a solution in any way.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Bug#913229: release.debian.org: RC status of merged-/usr bugs?

[Simon McVittie]

Package: release.debian.org
Severity: normal
User: m...@linux.it
Usertags: usrmerge

debootstrap in >= buster produces merged-/usr chroots for buster and sid
by default. Some packages are misbuilt in such chroots: the only concrete
example I have that is currently open is 
in quilt, in which a package built in a merged-/usr environment will
work on merged-/usr systems but won't work (at all) on a non-merged-/usr
system.

There are almost certainly others; most of them are probably similar to
the quilt bug, where the absolute path to an executable on the build
system ends up in the package, but to support non-merged-/usr systems
we need to canonicalize that path to the version that would work on the
non-merged-/usr systems. For instance, src:systemd used to have a similar
bug .

I've opened sbuild-createchroot bug 
suggesting that it should create non-merged-/usr chroots to sidestep
this for buildd builds, but there are many other ways for maintainers
and testers to build binaries.

If binary packages in the archive don't work on a non-merged-/usr system,
is that a RC bug in the binary packages? For instance, if an uploader
of quilt uploaded binaries that had been built on a merged-/usr system
without first fixing #913228, would that be RC? (I'm assuming the answer
is yes; certainly systemd's #843433 was treated as RC.)

Are bugs like #843433 and #913226, in which a package built in a
merged-/usr environment won't work in a non-merged-/usr environment,
considered to be RC bugs in the *source package* even if the binaries
in the archive happen to be OK?

See also  which proposes that
reproducible.debian.org should exercise merged vs. non-merged /usr,
as a way to detect this class of bug.

smcv