Re: Q: Use https for {deb,security}.debian.org by default

[Philipp Kern]

On 20.08.21 21:11, Russ Allbery wrote:

The way I would put it is that the security benefit of using TLS for apt
updates is primarily that it makes certain classes of attempts to mess
with the update channel more noisy and more likely to produce immediate
errors.
One thing of note is that it introduces a time dependency on the client. 
Now we seem to gravitate towards a world where you'd also fail DNS 
resolution if your time is wrong (because you cannot get at the 
DNS-over-TLS/HTTPS server), so this is probably accepted as not making 
things worse overall. I guess we could have some (somewhat insecure) 
defense in depth if we wanted to, but maybe the world just agreed that 
you need to get your clock roughly correct. ;-)


Kind regards
Philipp Kern

Re: Q: Use https for {deb,security}.debian.org by default

[Hideki Yamane]

Hi all,

 Thanks for your comments!
 It seems that no big blocker to make https default for deb.debian.org
 and security.debian.org.


On Thu, 19 Aug 2021 22:38:20 +0900
Hideki Yamane  wrote:
>  Now deb.debian.org and security.debian.org provide https access
>  but created sources.list file use http for those. Is there any
>  reason to use http instead of https for them? (traffic, policy,
>  etc...) If not, how about to change it?

 Q: Make https as default situation worse?
   A: No :)
  If the clock setting is not appropriate, d-i provides NTP setting
  through installation. 

 Q: Is there any benefit?
   A: Yes, going forward with https as default is a trend, and some
  people complain about the way for accessing our repo as http.
  We can avoid such boring discussion and wrong message to users.

 Q: How about largely deploy environment like containers?
   A: They can choose http for that, since just "https as default"

 Q: Is it perfect solution?
   A: Of course not :)

 Q: Do you intend to mandate https access for apt?
   A: No, just make https "default" for deb.debian.org and security.debian.org.
  There is a choise to use other repo and http instead.



-- 
Hideki Yamane 

Re: Q: Use https for {deb,security}.debian.org by default

[Tomas Pospisek]

On 21.08.21 09:14, Philipp Kern wrote:

On 20.08.21 21:11, Russ Allbery wrote:

The way I would put it is that the security benefit of using TLS for apt
updates is primarily that it makes certain classes of attempts to mess
with the update channel more noisy and more likely to produce immediate
errors.
One thing of note is that it introduces a time dependency on the client. 
Now we seem to gravitate towards a world where you'd also fail DNS 
resolution if your time is wrong (because you cannot get at the 
DNS-over-TLS/HTTPS server), so this is probably accepted as not making 
things worse overall. I guess we could have some (somewhat insecure) 
defense in depth if we wanted to, but maybe the world just agreed that 
you need to get your clock roughly correct. ;-)


I remember seeing apt-get refusing to update packages or the index 
because of them "having timestamps in the future" or in other words 
system time being out of sync in direction of the past.


So we already have the situation that system time **must not** be off 
into the past by some delta in order to be able to do updates **at all**.


This is out of my memory so if anybody cares about this argument it 
should maybe be confirmed more thoroughly.

*t

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Fri, Aug 20, 2021 at 11:21:55AM +0100, Luca Boccassi wrote:
> On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
> > On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> > > 
> > > I think no one likes that idea, but it's the only solution that doesn't
> > > immediately fail because it requires a dpkg update that hasn't shipped 
> > > with
> > > the current stable release, breaks local packages (kernel modules, 
> > > firmware,
> > > site-wide systemd configuration), or both.
> > 
> > This could be solved if we could somehow require dpkg to be updated
> > before any other packages during the the next update, no?
> > 
> > Breaking this constraint means that we can't make "apt-get
> > dist-update" work seemlessly --- but what if we were to change the
> > documented procedure for doing a major update?
> > 
> > That's not ideal, granted, but how does that compare against the other
> > alternatives?
> > 
> > - Ted
> > 
> > P.S.  I had a vague memory that there was some update in the long
> > distant past where we did require a manual upgrade of dpkg first.  Or
> > is my memory playing tricks on me?  I do know that a manual update of
> > dpkg is the first step in a crossgrade
> 
> An update to dpkg is not _required_. It might be very strongly
> _desired_ which is a perfectly legitimate stance to take, but it is not
> technically required, otherwise we couldn't have been shipping with
> merged-usr as default in new installations of Buster and Bullseye for
> 2+ years, we could not have been installing usrmerge in older
> installations for 2+ years, and Ubuntu would not exist anymore since
> legacy split-usr is discontinued and even older installations are being
> forcibly converted. So continuing to live with this minor ~20 years old
> dpkg bug as we've been doing for years is a valid option - one that
> some might very, very strongly dislike and argue against which is again
> perfectly legitimate, but it is de-facto an option nonetheless, because
> it's the actual status quo for 2+ years.

It bothers me that you believe "we've been doing this for a while and it
didn't cause any problems, so let's just continue doing things that way
even if the people who actually wrote the damn code say that path is
littered with minefields and they're scared of what could happen when we
finish the tranition this way" is a valid strategy. It goes against
everything I was taught to do to write reliable software.

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: Q: Use https for {deb,security}.debian.org by default

[Wouter Verhelst]

On Thu, Aug 19, 2021 at 10:11:33PM +, Jeremy Stanley wrote:
> On 2021-08-19 16:37:13 -0400 (-0400), Kyle Edwards wrote:
> > On 8/19/21 3:46 PM, Simon Richter wrote:
> > > For the most part, users would configure https if they are behind a
> > > corporate firewall that disallows http, or modifies data in-flight so
> > > signature verification fails, everyone else is better off using plain
> > > http.
> > 
> > Or they might configure https on the sheer principle of not wanting to have
> > their traffic hoovered up by their ISP or anyone else who might be
> > listening.
> 
> While this does complicate it, a snooping party can still know the
> site they're connecting to via SNI happening unencrypted,

SNI is not unencrypted if you do TLS1.3...

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: inconsistent mailgraph settings

[Tomas Pospisek]

Hi Vincent,

On 20.08.21 16:50, Vincent Lefevre wrote:

My bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734
has been closed again, with no explanations.


https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734;msg=12 claims 
that the bug was closed via 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734;msg=10 . 
However I can't see why the latter mentioned message would close the 
bugreport. Can anybody shed some light on this?


Either way: try to stay calm, try keeping a *cooperative* conversation 
with the maintainer. Your argument being right or not right won't 
guarantee you that you'll have your way. You need to work with the 
maintainer to get your issue resolved in a way that's acceptable for 
both of you. So try not to escalate things, try to keep your urges in check.


I can't see the maintainer refusing to work with you: it just seems that 
it is unclear why mailgraph wouldn't start automatically after the 
upgrade. Remember that the maintainer doesn't own anything to you. He 
doesn't *have* to fix anything.


In particular it *seems* to work for him and he doesn't have access to 
your system where things apparently went wrong so it could be really 
hard for him to know. So what you can do is to try to debug *yourself* 
why the upgrade went wrong and come forth with some analysis that shows 
to root cause of the breakage. And then you can also propose a fix. I am 
sure that would make it a lot easier for the maintainer to do "the right 
thing".



While mailgraph was started on boot in the past, this stopped
working with the upgrade to Debian 10, and I had to enable it
again. So issues with the upgrade to Debian 11, but the mailgraph
package has not changed. And who knows what will happend with
the next upgrade...

I can see in the debian/default.conf file:

# Should Mailgraph start on boot (true|false) (default: true)
BOOT_START="true"

So I don't understand what is the intent for the default settings.
To run it on boot or not???


Evidently the package is offering a choice on whether to start or not to 
start mailgraph on boot. By virtue of that option existing it seems that 
not everybody wants mailgraph to be started on boot.



Note that I was already using systemd in Debian 9, so that it is
not normal that mailgraph stopped working after the upgrade to
Debian 10.


Don't be all up in arms about the problem. Try to find out why it's 
broken and try to propose a fix and to work with the maintainer.


Note that not only you might get frustrated but the maintainer might too 
in which case he might decide to give up on the package and then you'll 
be left with less than when you started. So try to keep the conversation 
friendly, supportive and cooperative.


All the best!
*t

Re: Q: Use https for {deb,security}.debian.org by default

[Wouter Verhelst]

On Fri, Aug 20, 2021 at 07:20:22PM +, Jeremy Stanley wrote:
> Yes transparent proxies or overridden DNS lookups could be used to
> direct deb.debian.org and security.debian.org to your alternative
> location,

I've been thinking for a while that we should bake a feature in apt
whereby a network administrator can indicate somehow that there is a
local apt mirror and that apt should use that one in preference to
deb.debian.org.

This could be useful for both the "I've got a slow uplink and would like
it to not be overwhelmed at the BSP I'm hosting for my Debian friends"
type as well as the "I'm an ISP and I want to provide a mirror to Debian
users so we can reduce our uplink connection a bit" type of situations.

However, I've not been able to come up with a scheme which is simple
enough to be doable on a LAN while at the same time be usable by larger
network providers, *and* which can't also be abused by MitM attackers.

Perhaps it's just not something we would be able to do?

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

Re: inconsistent mailgraph settings

[Mattia Rizzolo]

On Sat, Aug 21, 2021 at 10:36:04AM +0200, Tomas Pospisek wrote:
> Hi Vincent,
> 
> On 20.08.21 16:50, Vincent Lefevre wrote:
> > My bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734
> > has been closed again, with no explanations.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734;msg=12 claims that
> the bug was closed via
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989734;msg=10 . However I
> can't see why the latter mentioned message would close the bugreport. Can
> anybody shed some light on this?

Likely Jörg Frings-Fürst BCCed the -close@ address.  There are some
people who do that…

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Q: Use https for {deb,security}.debian.org by default

[Simon Richter]

Hi,

On 21.08.21 10:40, Wouter Verhelst wrote:


I've been thinking for a while that we should bake a feature in apt
whereby a network administrator can indicate somehow that there is a
local apt mirror and that apt should use that one in preference to
deb.debian.org.


I've been thinking the same thing, but that would negate the remaining 
security benefit of using https, that's why I expressed a preference for 
making it visible that the connection is not encrypted and security is 
only provided by the signatures over pretending it is and then 
(silently?) allowing a proxy to intercept the connection.


   Simon



OpenPGP_signature
Description: OpenPGP digital signature

Re: Q: Use https for {deb,security}.debian.org by default

[Andrew M.A. Cater]

On Sat, Aug 21, 2021 at 12:31:26PM +0200, Simon Richter wrote:
> Hi,
> 
> On 21.08.21 10:40, Wouter Verhelst wrote:
> 
> > I've been thinking for a while that we should bake a feature in apt
> > whereby a network administrator can indicate somehow that there is a
> > local apt mirror and that apt should use that one in preference to
> > deb.debian.org.
> 
> I've been thinking the same thing, but that would negate the remaining
> security benefit of using https, that's why I expressed a preference for
> making it visible that the connection is not encrypted and security is only
> provided by the signatures over pretending it is and then (silently?)
> allowing a proxy to intercept the connection.
> 
>Simon
> 

This is effectively what you do if you specify a non-country mirror in setup
/ specify manually anyway - and that can be preseeded / set up with ansible
or whatever scripting you want to do en masse.

For myself, I'm not convinced that forcing https everywhere will give much
benefit in reality and may impose further complexity / a greater sense of
security without much more real security but if a setting is available
to be overridden when desired, it's not a deal breaker.

All the very best as ever,

Andy Cater

Re: Q: Use https for {deb,security}.debian.org by default

[Tobias Frost]

On Sat, Aug 21, 2021 at 10:40:32AM +0200, Wouter Verhelst wrote:
> On Fri, Aug 20, 2021 at 07:20:22PM +, Jeremy Stanley wrote:
> > Yes transparent proxies or overridden DNS lookups could be used to
> > direct deb.debian.org and security.debian.org to your alternative
> > location,
> 
> I've been thinking for a while that we should bake a feature in apt
> whereby a network administrator can indicate somehow that there is a
> local apt mirror and that apt should use that one in preference to
> deb.debian.org.
>
> This could be useful for both the "I've got a slow uplink and would like
> it to not be overwhelmed at the BSP I'm hosting for my Debian friends"
> type as well as the "I'm an ISP and I want to provide a mirror to Debian
> users so we can reduce our uplink connection a bit" type of situations.
> 
> However, I've not been able to come up with a scheme which is simple
> enough to be doable on a LAN while at the same time be usable by larger
> network providers, *and* which can't also be abused by MitM attackers.
> 
> Perhaps it's just not something we would be able to do?

https://tracker.debian.org/pkg/squid-deb-proxy sparks into my mind,
but I cant tell which of those use cases it could tackle, did not investiage 
enough for it.
(eg  for sure wont help at ISP level) 

-- 
tobi

Re: Q: Use https for {deb,security}.debian.org by default

[Philip Hands]

Wouter Verhelst  writes:

> On Fri, Aug 20, 2021 at 07:20:22PM +, Jeremy Stanley wrote:
>> Yes transparent proxies or overridden DNS lookups could be used to
>> direct deb.debian.org and security.debian.org to your alternative
>> location,
>
> I've been thinking for a while that we should bake a feature in apt
> whereby a network administrator can indicate somehow that there is a
> local apt mirror and that apt should use that one in preference to
> deb.debian.org.
>
> This could be useful for both the "I've got a slow uplink and would like
> it to not be overwhelmed at the BSP I'm hosting for my Debian friends"
> type as well as the "I'm an ISP and I want to provide a mirror to Debian
> users so we can reduce our uplink connection a bit" type of situations.
>
> However, I've not been able to come up with a scheme which is simple
> enough to be doable on a LAN while at the same time be usable by larger
> network providers, *and* which can't also be abused by MitM attackers.

We could declare that if one can find a TXT record in the local domain
(e.g. _DEBIAN_LOCAL_ARCHIVE.example.com) then one should use its
contents in order to configure an additional source for packages, such
that one gets the signed Release file from one's normally configured
sources, and then when getting subsequent files gives the local source a
try, falling back to the normal setup when downloads or checksums fail.

I can see that one could try a DoS of sorts by setting up the TXT record
to point at a tarpit, say, but that could be handled by setting short
timeouts, and giving up on the local server after some number of failures.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY

Re: Q: Use https for {deb,security}.debian.org by default

[Phil Morrell]

On Sat, Aug 21, 2021 at 10:40:32AM +0200, Wouter Verhelst wrote:
> On Fri, Aug 20, 2021 at 07:20:22PM +, Jeremy Stanley wrote:
> > Yes transparent proxies or overridden DNS lookups could be used to
> > direct deb.debian.org and security.debian.org to your alternative
> > location,
> 
> I've been thinking for a while that we should bake a feature in apt
> whereby a network administrator can indicate somehow that there is a
> local apt mirror and that apt should use that one in preference to
> deb.debian.org.

This already exists in the form of an avahi service announcement for
_apt_proxy._tcp, issued by both squid-deb-proxy and apt-cacher-ng.
Literally the only thing needed client-side is installation of
squid-deb-proxy-client, which is also available in udeb form implying
that d-i already uses it.

> This could be useful for both the "I've got a slow uplink and would like
> it to not be overwhelmed at the BSP I'm hosting for my Debian friends"
> type as well as the "I'm an ISP and I want to provide a mirror to Debian
> users so we can reduce our uplink connection a bit" type of situations.

It's a great solution for everyone on the same wifi network, if everyone
has squid-deb-proxy-client installed then just one person can spawn a
proxy and suddenly everyone's caching through them.

> However, I've not been able to come up with a scheme which is simple
> enough to be doable on a LAN while at the same time be usable by larger
> network providers, *and* which can't also be abused by MitM attackers.

Isn't the MitM handled by archive signatures etc., hence why http is
fine? True I haven't tested this in a large network, since usually
configuration management is in place, but apparently mDNS can even
traverse routers via Multicast BGP.


signature.asc
Description: PGP signature

Re: Q: Use https for {deb,security}.debian.org by default

[Thomas Goirand]

On 8/20/21 4:56 PM, Russ Allbery wrote:
> Jeremy Stanley  writes:
> 
>> I agree with all of the above, my point was that the current state of
>> HTTPS doesn't especially improve integrity for Debian package management
>> over the signed indices and checksums we already rely on, and trying to
>> use HTTPS for privacy/secrecy (which isn't really what it was designed
>> for) is still and perhaps even increasingly misguided. Of course lots of
>> people will continue to expect magic HTTPS fairy dust to protect them
>> and ward off evil, but the only legitimate reason I can see for Debian
>> changing the default protocol for sources.list entries is to avoid
>> having to pointlessly debate the minimal benefits of HTTPS with people
>> who drink whatever cool-aid they're told by security "experts" (HTTP
>> bad, HTTPS good, drink up!).
> 
> Do you think using HTTPS makes security worse?

Doing an update over Tor would be a much nicer improvement (so that
nobody can tell what package you didn't upgrade yet...).

> Personally, I think we should switch our default to HTTPS not because we
> have a specific security flaw in mind against which HTTPS provides some
> protection but because it's consistent with the general message that a lot
> of us (including, for example, the EFF and the IETF) are trying to send to
> average users who don't have the expertise to analyze any of this: use TLS
> by default wherever you can.  It's not a panacea, but ubiquitous, default
> use of TLS helps both your security and your privacy compared to either
> the previous default of no TLS or spending a bunch of mental energy
> picking and choosing.
> 

IMO, it's consistent with blindly trusting HTTPS which isn't helping
much here (especially compared to Tor), and if we want to promote
something, that'd be Tor, not the blindly trusted bunch of CAs...

Cheers,

Thomas Goirand (zigo)

Re: Debian 11 Bullseye Setup Problems Error Report

[Philip Hands]

Paul Wise  writes:

> On Wed, Aug 18, 2021 at 10:42 AM admin4  wrote:
>
>> is there a Debian "testing" team?
>
> That is composed of everyone who uses Debian and especially those who
> decide to report an issue they found.

While that probably accounts for the bulk of the effort, there are also
people that work on this within Debian (as mentioned debian-cd does
testing), and we also have several mechanical testing efforts of various
sorts-- e.g.:

  https://jenkins.debian.net
  https://reproducible-builds.org/
  https://openqa.debian.net/
  https://ci.debian.net/
  https://salsa.debian.org/salsa-ci-team/pipeline/

(there are bound to be more that I've forgotten)

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Q: Use https for {deb,security}.debian.org by default

[David Kalnischkies]

On Sat, Aug 21, 2021 at 12:04:32PM +0100, Phil Morrell wrote:
> On Sat, Aug 21, 2021 at 10:40:32AM +0200, Wouter Verhelst wrote:
> > On Fri, Aug 20, 2021 at 07:20:22PM +, Jeremy Stanley wrote:
> > > Yes transparent proxies or overridden DNS lookups could be used to
> > > direct deb.debian.org and security.debian.org to your alternative
> > > location,
> > 
> > I've been thinking for a while that we should bake a feature in apt
> > whereby a network administrator can indicate somehow that there is a
> > local apt mirror and that apt should use that one in preference to
> > deb.debian.org.
> 
> This already exists in the form of an avahi service announcement for
> _apt_proxy._tcp, issued by both squid-deb-proxy and apt-cacher-ng.
> Literally the only thing needed client-side is installation of
> squid-deb-proxy-client […]

That will instruct apt to use the proxy to connect to the internet, but
this is quite literal in meaning: apt will perform a CONNECT request
establishing a tunnel between itself and the remote server via the
proxy effectively by-passing any functionality the proxy would provide
if we wouldn't connect to the remote with https (as with http apt would
just issue GET requests to the proxy it could interact with).


apt can't just downgrade https to http if it knows about a proxy,
especially if that knowledge is provided by external potentially
untrusted sources. To do that we would need to at least ask the user
interactively if its okay to send the requests unencrypted to the proxy.

There is precedence with cdrom asking the user interactively to change
CDs if needed, so it isn't an entirely new concept, but libapt has no
generic question-asking code and cdrom is a cake walk compared to the
monster that is our http(s) implementation, so that is still a non-
trivial amount of code someone would need to write. Also in the libapt
front ends as you still need at least a bit of UI to actually expose the
question to the user.


Depending on how much control you have over the clients it might be
a lot easier to work with the mirror method. It can be (ab)used for
a lot more than most people give it credit for (Disclaimer: As I wrote
the current incarnation, I might be a *tiny bit* biased). That isn't
helping of course if you have no control at all over the clients as you
need some form of opt in at least. So far, that opt in was using http.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Q: Use https for {deb,security}.debian.org by default

[David Kalnischkies]

On Sat, Aug 21, 2021 at 09:45:54AM +0200, Tomas Pospisek wrote:
> On 21.08.21 09:14, Philipp Kern wrote:
> > defense in depth if we wanted to, but maybe the world just agreed that
> > you need to get your clock roughly correct. ;-)
> 
> I remember seeing apt-get refusing to update packages or the index because
> of them "having timestamps in the future" or in other words system time
> being out of sync in direction of the past.

APT requires the time to be more or less correct since ever¹ by virtue
of e.g. gpg keys (or signatures) expiring and expired keys are bad.

In recent years we became more reliant on the time to ensure
repositories are somewhat current refusing repos from too long in the
past as well as from the future. At least these can be worked around
with -o Acquire::Check-Date=false.

For gpg you will need another workaround I can't remember of the top of
my hat. There are likely more problems as it is easier to just set the
clock approximately correct than to remember all the workarounds in
"the time of need"…


Best regards

David Kalnischkies

¹ okay, ~15 years of apt-secure are not exactly ever, but close enough.


signature.asc
Description: PGP signature

Re: Q: Use https for {deb,security}.debian.org by default

[Jeremy Stanley]

On 2021-08-21 12:04:32 +0100 (+0100), Phil Morrell wrote:
> On Sat, Aug 21, 2021 at 10:40:32AM +0200, Wouter Verhelst wrote:
[...]
> > However, I've not been able to come up with a scheme which is simple
> > enough to be doable on a LAN while at the same time be usable by larger
> > network providers, *and* which can't also be abused by MitM attackers.
> 
> Isn't the MitM handled by archive signatures etc., hence why http is
> fine? True I haven't tested this in a large network, since usually
> configuration management is in place, but apparently mDNS can even
> traverse routers via Multicast BGP.

As already pointed out by others, the risk is that a MitM could
serve you outdated package indices and packages, silently blocking
you from patching some known vulnerability until the index expires,
which might provide the attacker some extra time to work on
exploiting that vulnerability. The practicality of this particular
attack isn't all that high, as there are often going to be other
avenues of compromise which involve less effort on the part of the
attacker anyway. Still, people are correct to call it out as some
form of risk.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Fri, 2021-08-20 at 23:15 +0200, Simon Richter wrote:
> Hi,
> 
> On 8/20/21 3:56 PM, Sam Hartman wrote:
> 
> > Simon's position seemed to be that we need a dpkg update  in order
> > to
> > move forward and that we cannot depend on that mid-release.
> 
> Yes, except if we give up "apt dist-upgrade" as the interface for the
> upgrade to the next stable release.

Forgive me, but you have made this statement a few times now, and each
time you have been asked for a reference to what usrmerge-specific
measures were adopted in Ubuntu's distro updater to justify it, but so
far I have not seen any (I might have simply missed it, in which case I
apologize in advance). Could you please share such reference(s) so that
we can understand what you are referring to?

> > I can see two arguments why we might need a dpkg update:
> > 
> > 1)  To fix bugs related to directory aliasing.
> > 
> > I don't think that there is a consensus those bugs need to be fixed
> > to
> > move forward.  (Put another way it's not clear the community agrees
> > they
> > are RC).
> 
> dpkg as it is now will detect only the case when a file is moved to
> the 
> usrmerge path inside a package, or when a file is moved from one
> package 
> to another (with Replaces), but not both at the same time.

Your example is essentially having /bin/foo and /usr/bin/foo at the
same time, while not being bit-by-bit identical. I was under the
impression that this was already considered RC-buggy and has been for a
long while? Am I mistaken and is that not the case? If so, how common
is it in the archive, do we have numbers?

> > IN particular, most systems are usrmerged today, and while these
> > bugs
> > are annoying, many people get along just fine.
> 
> Yes, and on all of these systems, dpkg does not have an accurate view
> of 
> what is installed, so an upgrade to bookworm that moves files between
> packages is likely to have files vanish, depending on the order in
> which 
> packages are installed.

Why has this not happened upgrading from Buster to Bullseye? Or 19.10
to 20.04? Or 20.04 to 20.10? Etc. What's different in Bookworm that
would cause these issues to suddenly appear?

> > Yes, there are bugs.
> > Yes, it would be good to get them fixed in the bookworm cycle.
> > But despite the issue being brought up, there is not strong support
> > for
> > the idea that we must block on a solution to the dpkg directory
> > aliasing
> > bugs.
> 
> I think that one of the release goals should be that any freshly 
> installed or upgraded system should have a dpkg database that is 
> consistent with reality, and I'd prioritize that higher than actually
> finishing the transition, because as long as we can have files vanish
> from under us, we can't expect that the transition will be reliable
> for 
> bullseye->bookworm updates.
> 
> Basically, we've been lucky so far.

Sorry but I for one strongly disagree that this should be a release
goal at all. It's an internal implementation detail caused by a 20
years old bug in dpkg - the maintainers of which are free to fix such
bug at any time if they like (as they were in the past 20 years since
this has been open and remained unaddressed), or to keep ignoring it.
What is very relevant is which real consequences this does bring to the
surface for users, and so far the only proven, reported and unsolved
issue I have seen is the dpkg -S inconvenience, despite multiple years
and an install base approaching 100% of a very popular distro like
Ubuntu - of course, it is entirely possible that I simply missed them,
in which case links to relevant Launchpad/bugs.d.o would be more than
welcome.

> > 2)
> > We might need a dpkg update to actually do the transition.
> 
> Yes, that was my symlink farmer idea, but that doesn't work: 
> special-case replacing a directory that only contains symlinks with a
> symlink, the same way GNU Stow tries to create its symlinks on the 
> highest possible level, but since dpkg refuses to even unpack a
> package 
> that contains a compatibility symlink and the file itself on an 
> usrmerged system, this won't work.
> 
> Since we need a mechanism to clean up the mess the usrmerge package
> left 
> us with, adding this mechanism to dpkg itself might still be a good
> idea 
> -- while that isn't the first package to be upgraded, it is among the
> first, so we can create a safe environment for later packages and
> thus 
> minimize the number of packages we have to leave in "reinstreq"
> state.
> 
>     Simon

Sorry, but so far it is not proven at all that we _need_ to clean up
anything. Some people would very strongly like to do so, and that's a
fine and legitimate opinion to hold. But it is factual that it is not
objectively _needed_, otherwise Ubuntu wouldn't ship and default-usr-
merged Debian installations since Buster wouldn't exist, and older-and-
migrated-via-usrmerge installations since Buster wouldn't exist either.

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitall

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> On Fri, Aug 20, 2021 at 11:21:55AM +0100, Luca Boccassi wrote:
> > On Thu, 2021-08-19 at 19:55 -0400, Theodore Ts'o wrote:
> > > On Thu, Aug 19, 2021 at 10:39:45PM +0200, Simon Richter wrote:
> > > > 
> > > > I think no one likes that idea, but it's the only solution that
> > > > doesn't
> > > > immediately fail because it requires a dpkg update that hasn't
> > > > shipped with
> > > > the current stable release, breaks local packages (kernel
> > > > modules, firmware,
> > > > site-wide systemd configuration), or both.
> > > 
> > > This could be solved if we could somehow require dpkg to be
> > > updated
> > > before any other packages during the the next update, no?
> > > 
> > > Breaking this constraint means that we can't make "apt-get
> > > dist-update" work seemlessly --- but what if we were to change
> > > the
> > > documented procedure for doing a major update?
> > > 
> > > That's not ideal, granted, but how does that compare against the
> > > other
> > > alternatives?
> > > 
> > > - Ted
> > > 
> > > P.S.  I had a vague memory that there was some update in the long
> > > distant past where we did require a manual upgrade of dpkg
> > > first.  Or
> > > is my memory playing tricks on me?  I do know that a manual
> > > update of
> > > dpkg is the first step in a crossgrade
> > 
> > An update to dpkg is not _required_. It might be very strongly
> > _desired_ which is a perfectly legitimate stance to take, but it is
> > not
> > technically required, otherwise we couldn't have been shipping with
> > merged-usr as default in new installations of Buster and Bullseye
> > for
> > 2+ years, we could not have been installing usrmerge in older
> > installations for 2+ years, and Ubuntu would not exist anymore
> > since
> > legacy split-usr is discontinued and even older installations are
> > being
> > forcibly converted. So continuing to live with this minor ~20 years
> > old
> > dpkg bug as we've been doing for years is a valid option - one that
> > some might very, very strongly dislike and argue against which is
> > again
> > perfectly legitimate, but it is de-facto an option nonetheless,
> > because
> > it's the actual status quo for 2+ years.
> 
> It bothers me that you believe "we've been doing this for a while and
> it
> didn't cause any problems, so let's just continue doing things that
> way
> even if the people who actually wrote the damn code say that path is
> littered with minefields and they're scared of what could happen when
> we
> finish the tranition this way" is a valid strategy. It goes against
> everything I was taught to do to write reliable software.

Many people are bothered by many things - such is life. For example, I
am very bothered that it appears impossible to do any kind of project-
wide innovation in Debian, and that we have been delegating that to
other distros since forever, and seem condemned to, at best,
frantically play catch-up, and at worst be dragged, kicking and
screaming, into what has been normal everywhere else for a decade, by
upstreams tired and frustrated of having to maintain legacy code paths
for the sole and exclusive benefit of this project. But I digress.

The main point is that of course the insights of experts are extremely
important, incredibly valuable and worth careful consideration,
especially when making decisions about an unknown future and events yet
to unfold. But in this case these are predictions about the past, a
past that already exists and is lived experience for many users here,
and for all users in Ubuntu. So there need to be _very_ convincing
explanations on why these predictions do not seem to match reality at
all, and so far these explanations have failed to materialize, I'm
afraid. We have been told everything is broken and the sky is about to
fall any minute now, and yet we have not been inundated with new bugs
since Buster made merged-usr the default, we have not been inundated
with new bugs by users upgrading from Buster to Bullseye or installing
usrmerge (despite the yet unsubstantiated claim in this thread that apt
dist-upgrade would stop working and require abandoning as a way to
upgrade the distro), and Canonical is not drowning in bug reports for
making usrmerge the mandated reality of 100% of its user base. The
usrmerge Launchpad has 2 (two) bugs [1]. The usrmerge Debian page has 4
(four) bugs, two of which are feature requests [2]. This is hardly the
stuff of nightmares. If one's expert viewpoints and predictions do not
match reality, they are not entitled to gloss over it because they are
the recognized and widely appreciated foremost expert in the field.

The reality of this industry is that reliable software is an oxymoron:
the only bug-free software is the one that doesn't exist. So the
question becomes, what is the measurable impact and magnitude of known
bugs and what is the likelihood and projected appearance rate of
unkno

Re: merged /usr vs. symlink farms

[Wouter Verhelst]

On Sat, Aug 21, 2021 at 02:40:02PM +0100, Luca Boccassi wrote:
> On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> > It bothers me that you believe "we've been doing this for a while
> > and it didn't cause any problems, so let's just continue doing
> > things that way even if the people who actually wrote the damn code
> > say that path is littered with minefields and they're scared of what
> > could happen when we finish the tranition this way" is a valid
> > strategy. It goes against everything I was taught to do to write
> > reliable software.
> 
> Many people are bothered by many things - such is life. For example, I
> am very bothered that it appears impossible to do any kind of project-
> wide innovation in Debian,

  "I don't deny the benefits.  I do think that in the current
  implementation, the drawbacks outweigh those benefits.  That's not to
  say it couldn't be done.  But if it is done, we should do it *right*.
  We're Debian.  That's what we do."

  -- Colin Walters, https://lists.debian.org/debian-devel/2003/06/msg00475.html

It's true that there are other distributions out there who go for the
quick-and-dirty solution, who want the feature before the benefits,
downsides, and risks have all been fleshed out. There's a reason why I'm
not contributing to those distributions; there's a reason why I don't
use those distributions.

"Doing it right", even if that takes time, has proven benefits.

When the RPM world implemented "multiarch", they only supported
installing 32-bit binaries on 64-bit versions of the same architecture.
They did have that feature implemented and functioning in a few months
or so, but the functionality of it was very limited -- and even today it
has problems, in that the way in which RPM checks that packages are
correct has some inherent heuristics that can make mistakes. Yes, I've
encountered those in practice on CentOS systems that my customers at the
time really really wanted to get up and running again pretty quickly.

When Debian and Ubuntu implemented multiarch (look ma, no quotes), the
time from concept to tests to implementation to public availability was
*much* longer than it was in the RPM world; and while this work was
unfinished, there was a lot of angry nagging about the lack of this
feature and why can they do it in the RPM world you guys are idiots, but
eventually it was implemented; and I think you'll agree that the dpkg
implementation of multiarch is far superior to the RPM one: it's
possible to use multiarch not just for compatibility with 32-bit
versions of your 64-bit platform, as in the RPM world, but *also* for
running arm binaries on x86 with qemu user emulation, or for
cross-compiling, or for various other features that the RPM world can
only dream of.

To get back to the point: I'm not saying we shouldn't merge /usr. We
should; the benefits of a properly merged /usr far outweigh any
disadvantages it may bring.  However, having an inconsistent dpkg
database is far more serious than just "oh dpkg -S won't work as
expected". It means dpkg isn't properly keeping track of which files
belong to which package anymore, which means you will have issues with a
package that Replaces: another, or with removing packages (especially
with security-conscious binaries), or with diversions, or with
alternatives, or with file conflicts, or with basically anything that
asks dpkg about locations of files; and just dismissing it with a
handwavy "ah well just run dpkg -S again" is so far removed from reality
that it's not even funny. I think the dpkg maintainers are 100% correct
to point out that that *is* a problem for which currently no viable
solution seems to exist, and that any way forward *must* include a
solution to that problem.

I'm not saying the solution which the dpkg maintainers are proposing is
the only valid solution, but if you go and tell them "ah the real
problems you point out are irrelevant" then You! Are! Doing! It! Wrong!

[...]
> The main point is that of course the insights of experts are extremely
> important, incredibly valuable and worth careful consideration,
> especially when making decisions about an unknown future and events yet
> to unfold. But in this case these are predictions about the past, a
> past that already exists and is lived experience for many users here,
> and for all users in Ubuntu.

What that is, is anecdotal evidence. "We've been doing X for a while and
it seems to not kill everything". Cool, great, awesome data points, but
not likely to convince me that there won't ever be any problems. You
can't prove the absense of bugs by anecdotal evidence; you can only
prove the existence of them that way.

What the dpkg maintainers are providing is analytical evidence. "There's
some corner cases here which need to be catered to". You just can't say
that corner cases don't happen because "anecdotal evidence". That's just
not how any of that works.

[...]
> The reality of this industry is that reliable software is an oxymo

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Sat, Aug 21, 2021 at 10:26:13AM +0200, Wouter Verhelst wrote:
> It bothers me that you believe "we've been doing this for a while and it
> didn't cause any problems, so let's just continue doing things that way
> even if the people who actually wrote the damn code say that path is
> littered with minefields and they're scared of what could happen when we
> finish the tranition this way" is a valid strategy. It goes against
> everything I was taught to do to write reliable software.

So as an expert, what's your recommendation about what is to be done?
Personally, I *don't* have a problem about telling people to manually
update dpkg, apt, and/or apt-get before they do the next major stable
release (maybe it's because this is something I do as a matter of
course; it's not that much extra effort, and I'm a paranoid s.o.b.,
and I know that's the most tested path given how Debian testing
works).

Other people think that is a terrible idea, to be avoided at all
costs.  I don't understand why that is such a terrible outcome, since
I do it already, but perhaps I can be educated on that point.

In any case, I believe downsides of the symlink farm alternative are
greater than the downsides of other options:

* just living with the risk of potential corner cases which
  might affect users when they do the bullseye -> bookworm
  upgrade, since aparently the sky hasn't fallen with Ubuntu, or

* advise users to upgrade dpkg/apt/apt-get first when they do
  the next release, with the strength of that advice depending
  on how likely users are to suffer from a "mine" causing
  their system to lose a limb or two.

Heck, if the minefield is that dangerous (and if so, why the *heck*
aren't Ubuntu users screaming from data loss, system instability,
etc?), perhaps you should advise the release managers that dpkg needs
to have fixes pushed out to Bullseye NOW! NOW! NOW! to eliminate the
potential imminent damage that you seem to be so fearful of our users
might get hit with.

Can you give more details about real life scenarios which is
triggering your fears, and whether there are ways we can mitigate
against those scenarios?

Best regards,

- Ted

Re: testing for rootfs vs. /usr reproducibility regressions

[Andy Smith]

Hi Tim,

On Fri, Aug 20, 2021 at 05:29:54PM -0400, Timothy M Butterworth wrote:
> I have a new-be question, what is the point of merged-usr?

I put "debian merged-usr" into my favourite search engine and the
first result was:

https://wiki.debian.org/UsrMerge

Does this page and those linked from it answer your question
sufficiently as to what the point of the effort is?

The link near to the bottom:

http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge

is quite informative.

If you read those and it still isn't clear, it might be best to ask
about it on debian-user, as debian-devel would really be for Debian
Developers (who are mostly up to speed on this already to varying
degrees) to decide about *how* to do it, not the basics of *why* or if
it even *should* be done.

It has been the default for some time for new installs and the
tech-ctte already decided that it will be the only supported
configuration for the next release after bookworm. At this point
rehashing the whys of it would be repeating conversations had over
the last several years, so I feel like it's a user documentation
issue at this point if that's necessary.

Cheers,
Andy

Re: merged /usr vs. symlink farms

[Luca Boccassi]

On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> On Sat, Aug 21, 2021 at 02:40:02PM +0100, Luca Boccassi wrote:
> > On Sat, 2021-08-21 at 10:26 +0200, Wouter Verhelst wrote:
> > > It bothers me that you believe "we've been doing this for a while
> > > and it didn't cause any problems, so let's just continue doing
> > > things that way even if the people who actually wrote the damn
> > > code
> > > say that path is littered with minefields and they're scared of
> > > what
> > > could happen when we finish the tranition this way" is a valid
> > > strategy. It goes against everything I was taught to do to write
> > > reliable software.
> > 
> > Many people are bothered by many things - such is life. For
> > example, I
> > am very bothered that it appears impossible to do any kind of
> > project-
> > wide innovation in Debian,
> 
>   "I don't deny the benefits.  I do think that in the current
>   implementation, the drawbacks outweigh those benefits.  That's not
> to
>   say it couldn't be done.  But if it is done, we should do it
> *right*.
>   We're Debian.  That's what we do."
> 
>   -- Colin Walters,  
> https://lists.debian.org/debian-devel/2003/06/msg00475.html
> 
> It's true that there are other distributions out there who go for the
> quick-and-dirty solution, who want the feature before the benefits,
> downsides, and risks have all been fleshed out. There's a reason why
> I'm
> not contributing to those distributions; there's a reason why I don't
> use those distributions.
> 
> "Doing it right", even if that takes time, has proven benefits.
> 
> When the RPM world implemented "multiarch", they only supported
> installing 32-bit binaries on 64-bit versions of the same
> architecture.
> They did have that feature implemented and functioning in a few
> months
> or so, but the functionality of it was very limited -- and even today
> it
> has problems, in that the way in which RPM checks that packages are
> correct has some inherent heuristics that can make mistakes. Yes,
> I've
> encountered those in practice on CentOS systems that my customers at
> the
> time really really wanted to get up and running again pretty quickly.
> 
> When Debian and Ubuntu implemented multiarch (look ma, no quotes),
> the
> time from concept to tests to implementation to public availability
> was
> *much* longer than it was in the RPM world; and while this work was
> unfinished, there was a lot of angry nagging about the lack of this
> feature and why can they do it in the RPM world you guys are idiots,
> but
> eventually it was implemented; and I think you'll agree that the dpkg
> implementation of multiarch is far superior to the RPM one: it's
> possible to use multiarch not just for compatibility with 32-bit
> versions of your 64-bit platform, as in the RPM world, but *also* for
> running arm binaries on x86 with qemu user emulation, or for
> cross-compiling, or for various other features that the RPM world can
> only dream of.

My recollection (which might be wrong, but a quick look at release
notes seems to support it with 11.04 having multiarch 2 years before
Wheezy) is that Canonical led the way with the multiarch effort in
Ubuntu, and Debian followed with lots of huffing, puffing and
grumbling.

> To get back to the point: I'm not saying we shouldn't merge /usr. We
> should; the benefits of a properly merged /usr far outweigh any
> disadvantages it may bring.  However, having an inconsistent dpkg
> database is far more serious than just "oh dpkg -S won't work as
> expected". It means dpkg isn't properly keeping track of which files
> belong to which package anymore, which means you will have issues
> with a
> package that Replaces: another, or with removing packages (especially
> with security-conscious binaries), or with diversions, or with
> alternatives, or with file conflicts, or with basically anything that
> asks dpkg about locations of files; and just dismissing it with a
> handwavy "ah well just run dpkg -S again" is so far removed from
> reality
> that it's not even funny. I think the dpkg maintainers are 100%
> correct
> to point out that that *is* a problem for which currently no viable
> solution seems to exist, and that any way forward *must* include a
> solution to that problem.
> 
> I'm not saying the solution which the dpkg maintainers are proposing
> is
> the only valid solution, but if you go and tell them "ah the real
> problems you point out are irrelevant" then You! Are! Doing! It!
> Wrong!

Again, if the magnitude of this dpkg bug was really that serious there
would be visible consequences after almost 3 years of deployments
across two distributions with who knows how many million instances, and
yet "having to run dpkg -S again" is all we can see. Where are the bug
reports? Where are the enraged users with unusable broken system and
lost data? Where are the reports of Canonical going out of business
because Ubuntu is unusable? The bug is real, nobody doubts that - it
has been filed on d

Re: merged /usr vs. symlink farms

[Colin Watson]

On Sat, Aug 21, 2021 at 06:47:50PM +0100, Luca Boccassi wrote:
> My recollection (which might be wrong, but a quick look at release
> notes seems to support it with 11.04 having multiarch 2 years before
> Wheezy) is that Canonical led the way with the multiarch effort in
> Ubuntu, and Debian followed with lots of huffing, puffing and
> grumbling.

As a Canonical employee who was involved in the multiarch design work at
the time, this is a pretty unfair-to-Debian version of history.  Yes,
some things took a bit longer to get organized on the Debian side for
various reasons, but the design and implementation work was done in
collaboration with key people in Debian and was definitely better for
it; Guillem and Raphaël in particular did a lot of hard work on dpkg and
dpkg-dev respectively.  (Also, several of us on the Canonical side
regarded ourselves as having one foot firmly in each camp; I certainly
didn't see it as a confrontational sort of thing where we were having to
drag Debian along with us - rather the contrary, there was a lot of
enthusiasm in Debian for it.)

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: testing for rootfs vs. /usr reproducibility regressions

[Timothy M Butterworth]

On 8/21/21, Andy Smith  wrote:
> Hi Tim,
>
> On Fri, Aug 20, 2021 at 05:29:54PM -0400, Timothy M Butterworth wrote:
>> I have a new-be question, what is the point of merged-usr?
>
> I put "debian merged-usr" into my favourite search engine and the
> first result was:
>
> https://wiki.debian.org/UsrMerge
>
> Does this page and those linked from it answer your question
> sufficiently as to what the point of the effort is?
>
> The link near to the bottom:
>
> http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge
>
> is quite informative.

Thanks for the links, I have read them and I am all caught up now.
This seems like something that Linux Standard  Base LSB should have
resolved regarding the installation directory.

> If you read those and it still isn't clear, it might be best to ask
> about it on debian-user, as debian-devel would really be for Debian
> Developers (who are mostly up to speed on this already to varying
> degrees) to decide about *how* to do it, not the basics of *why* or if
> it even *should* be done.
>
> It has been the default for some time for new installs and the
> tech-ctte already decided that it will be the only supported
> configuration for the next release after bookworm. At this point
> rehashing the whys of it would be repeating conversations had over
> the last several years, so I feel like it's a user documentation
> issue at this point if that's necessary.
>
> Cheers,
> Andy
>
>

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Sat, 2021-08-21 at 18:47:50 +0100, Luca Boccassi wrote:
> My recollection (which might be wrong, but a quick look at release
> notes seems to support it with 11.04 having multiarch 2 years before
> Wheezy) is that Canonical led the way with the multiarch effort in
> Ubuntu, and Debian followed with lots of huffing, puffing and
> grumbling.

You mean that time when Ubuntu merged an implementation based on a broken
design with broken interfaces, that was causing dpkg database damage, where
the tech-ctte also tried to force through, in all its wisdom? Right, great
example. (And let's ignore release cadences for more spectacular effect.)

  

But just wow, such a mischaracterization and deformation of the events.
Sadly, at this point I'm not surprised. This goes along comments such as
that the intersection of packages not using debhelper and shipping
split-/usr files are in the "thousands" (way less than the actual number
of packages shipping those pathnames), or the ones in the paragraphs
below about that mythical bug report, and the single failed attempt to
symlink farm, etc, etc…

> On Sat, 2021-08-21 at 16:20 +0200, Wouter Verhelst wrote:
> > I'm not saying the solution which the dpkg maintainers are proposing
> > is the only valid solution, but if you go and tell them "ah the real
> > problems you point out are irrelevant" then You! Are! Doing! It!
> > Wrong!
> 
> Again, if the magnitude of this dpkg bug was really that serious there
> would be visible consequences after almost 3 years of deployments
> across two distributions with who knows how many million instances, and
> yet "having to run dpkg -S again" is all we can see. Where are the bug
> reports? Where are the enraged users with unusable broken system and
> lost data? Where are the reports of Canonical going out of business
> because Ubuntu is unusable?

Just like no one had detected the database corruption in Ubuntu before
I spotted the problem via code review and analysis (which I guess in
your world translates to "opinion"). I'd expect the problems with
aliased directories to be that kind of insidious issue that people
have a very hard time trying to pin point, and which will be getting
worse as time passes.

> The bug is real, nobody doubts that - it has been filed on dpkg 20
> years ago.

You keep repeating this, but I have no idea what bug you refer to.

There's #148258 (from 2002), which is conffile related, and not
actionable and should probably just be closed.

There's #182747 (from 2003), which while apparently similar is
something else completely. This is about the (IMO) misfeature of
supporting a local admin to redirect (not alias) a directory using a
local symlink (mainly for space management reasons). For an explanation
see .

There's #406715 (from 2007) which is related to the above misfeature.

> What I am taking issues with is the representation of its
> actual, real effects, and thus its severity and the consequences for
> the project. There are a lot of words being spent on how terrible and
> broken and unacceptable the status quo is, and yet not a single link
> to a bug report.

What I'm appalled at is the sloppiness and dogma shown in the name of
a filesystem layout that will have very minimal benefit for final users
(in contrast to some use cases for some admins or installations that
should already know what they are doing, and can manage all potential
downsides in a controlled way through a hack like usrmerge), knowingly
in detriment of robustness and stability.

> By all means, go and fix it, make it a top priority for dpkg to sort
> out, all hands on deck, whatever needed -

To even consider the possibility to support this missing feature in
dpkg would require for it to get support for at least tracking
filesystem metadata (which is has *never* *ever* supported), which is
currently not deployable:

  

Even with that support in place, that just would not give automatic
aliased directory support. It would need no package to ship anything
inside those directories, and it would also need for some package to
ship those aliased symlinks. And then many corner cases would need to
be considered as then dpkg will need to reconcile what's on the
filesystem, on its database and on the various .deb (given that these
have been a shared resource, that it cannot possibly and safely switch
type by itself), w/o also breaking previous expectations. Not to mention
that the general aliasing issues still would not disappear.

> but to demand the entire project has to stand still,

So wait, when it suits you the "entire project" is involved and cannot
do stuff, but when it does not the "entire project" is not required to
do anything because the proposed solution magically solves stuff for
free with no effort involved… right.

> and to de-facto derail the 

Re: merged /usr vs. symlink farms

[Guillem Jover]

On Fri, 2021-08-20 at 07:56:33 -0600, Sam Hartman wrote:
> > "Theodore" == Theodore Ts'o  writes:
> Theodore> FWIW, from following the discussion, I've become more and
> Theodore> more convinced that a symlink farm is *not* the right
> Theodore> answer, regardless of whether it is done centrally or via
> Theodore> individual packages moving files and created symlinks if
> Theodore> necessary in individual maintainer scripts.

There was talk about the huge amount of symlinks required in a symlink
farm setting, but that might have been true for a scenario where those
symlinks would have been handled automatically and transparently.

The huge majority of files under /lib* (which is the actual bulk of them)
should require no symlink farms. Many of the ones under /bin and /sbin
(we are talking about around 240 packages here) might be switchable w/o
compat symlinks after careful consideration (or not…), many of the ones
that require symlinks would need these just for a period of time, and
only a handful would remain (the ones that are part of standard
interfaces, which I'd expect would be mostly shells?). I think the amount
of symlinks currently provided by f.ex. lvm2 and e2fsprogs combined would
amount to more symlinks than what we would eventually end up with TBH.

> Ted, in terms of judging consensus, I'd like to explicitly ACK your
> summary.
> I think that you have accurately described where we are in the
> discussion.
> 
> As you know, one of the ways we can see  how close we are on consensus
> is to look at what happens when someone proposes a summary like you did.
> 
> Simon Richter tried to challenge your summary effectively saying that we
> couldn't have an informed consensus because there were open technical
> issues that had not been addressed.  This was roundly rejected by
> yourself, Philip Hands and Luca Boccassi.

Well then that's a very underwhelming way to judge consensus TBH.

Philip remarked on the falling-sky theme (which I covered in a reply to
Luca elsewhere), and then used a crystal ball to predict the future,
where a particular filesystem layout would pass the test of time in
contrast to dpkg, which apparently (obviously a bit biased here) is such
a trivial detail in our distribution that it can be easily replaced w/o
requiring to scrap everything and starting a new distribution from
scratch.

Luca posted a mess of disinformation.

Ted's reply I've partially covered at the top of this mail. The rest
I've covered in the aforementioned reply to Luca. And some of the
remaining parts I've heard the "dpkg team" think are wild conjectures
on motivations and similar.

Or there's the following for example:

> IN particular, most systems are usrmerged today, and while these bugs
> are annoying, many people get along just fine.

That would imply that most people have either gone out of their way to
explicitly install and run usrmerge or they have reinstalled from
scratch. I cannot believe the first to be of any significance, the
second would put into question why we bother supporting release upgrades
(after all many other distros do not support that, we perhaps should
catch up with what the rest of the world is doing!).


I've also tried to stop replying to all misconceptions, inaccuracies
and plain wrong stuff on this thread (f.ex. people vocal in this thread
apparently do not exactly seem to know how upgrades or our package
manager works in Debian?), first to avoid dominating it with replies
(like some kind of deranged person, where I think I've probably already
surpassed my quota), and because it just all seems like a monumental
waste of time, energy and motivation. Which I'd rather spend elsewhere.



I'm personally just not seeing such consensus, despite the attempts of
some to make it pass as so. My perception is that this topic has become
such a black hole of despair, that people that take issue with it, are
simply stepping away.

Exhausted,
Guillem

Re: Q: Use https for {deb,security}.debian.org by default

[Stephan Verbücheln]

What about HTTP 304 Not Modified?

Regards

Bug#992664: ITP: ruby-parser -- Ruby parser written in pure Ruby

[Hideki Yamane]

Package: wnpp
Severity: wishlist
Owner: Hideki Yamane 
X-Debbugs-Cc: debian-devel@lists.debian.org, 
pkg-ruby-extras-maintain...@lists.alioth.debian.org

* Package name: ruby-parser
  Version : 3.0.2.0
* URL : https://github.com/whitequark/parser
* License : MIT
  Programming Lang: Ruby
  Description : Ruby parser written in pure Ruby

 Parser is a production-ready Ruby parser written in pure Ruby. It recognizes
 as much or more code than Ripper, Melbourne, JRubyParser or ruby_parser, and
 is vastly more convenient to use.

Re: merged /usr vs. symlink farms

[Simon Richter]

Hi,

On 21.08.21 19:47, Luca Boccassi wrote:


By all means, go and fix it, make it a top priority for dpkg to sort
out, all hands on deck, whatever needed - but to demand the entire
project has to stand still, and to de-facto derail the effort put in to
catch up with the rest of the world by imposing an unworkable,
demonstrably failed solution (symlinks farm) to work around a dpkg bug
instead of fixing it internally, to me does not seem acceptable in any
way, shape or form without some real, serious evidence that the sky has
indeed fallen.


There are two issues here: dpkg not handling certain corner cases, and 
the usemerge package modifying the file system, bypassing dpkg.


The latter is what brought us into a situation where it is no longer 
safe to move files between packages and between aliased directories in 
the same upgrade, and because users will be expected to upgrade in a 
single step between stable releases, that means these two types of 
changes are mutually exclusive for the entire release cycle.


This happened precisely because people "put in the effort" to implement 
this change without coordinating. This thing derailed on its own, and 
accusing the people pointing that out of ill will is not going to fix that.


The fact that the sky has not yet fallen is not a reason for inaction, 
but instead gives us the breathing room to implement a proper solution 
instead of another hack that will add yet another possible upgrade 
scenario we have to anticipate in the future.


We already have inconsistent package builds depending on whether a 
package was built on a pre- or post-transition autobuilder, with the 
exact same packages installed otherwise.


We have no piuparts test coverage for these scenarios, we have no QA 
tools to verify that installing the new packages will always lead to 
predictable outcomes no matter in which order you install them in -- 
such tools were never necessary before as dpkg resolves these problems 
in a deterministic manner provided its installation database is 
consistent with reality.


This is the situation we're in: the millions of systems out there work, 
but we cannot guarantee that they will continue to work through the next 
update because the QA infrastructure now has massive blind spots. This 
is what needs to be fixed before further progress can be made.


If the bookworm upgrade does break systems on upgrade, then the sky will 
indeed have fallen, only then it will be too late to do anything about it.



"It works for me" is anecdote, bugs count is not, it is a key metric of
this industry, I am quite surprised this needs to be specified. It's
how we decide whether a release is ready or not.


I see two problems here:

First, we're not the "industry." We're a Free Software movement. The 
industry just happens to embrace us at the moment.


Second, I wonder why it doesn't give you pause if a group of senior 
software people that uses a particular metric in one instance suddenly 
seems to be utterly unaware of its existence in another.



What you call analytical
evidence on the other hand is a fancy word for "opinion".


Analytical evidence happens when you read the dpkg source code and the 
usrmerge perl script and find a glaring obvious problem in the way they 
interact, then construct a simple adversarial example in 11 lines of 
text and two dpkg-deb calls and verify that it indeed loses files.


The scenario I posted will apply to any systemd package split between 
bullseye and bookworm. If systemd moves unit files to /usr, then it 
cannot move these to another package for the entirety of the release, or 
risk the units disappearing on upgrade. It also applies to kernel module 
and firmware packages, and several core system utilities.


We got lucky with the "which" command as that was in /usr already.


Opinions are
useful and interesting and important, but saying "everything is broken"
when there is a surprising lack of evidence of that being the case, is
not very useful or constructive.


Absence of evidence is not evidence of absence.

   Simon

Re: merged /usr vs. symlink farms

[Theodore Ts'o]

On Sun, Aug 22, 2021 at 02:15:31AM +0200, Simon Richter wrote:
> 
> The latter is what brought us into a situation where it is no longer safe to
> move files between packages and between aliased directories in the same
> upgrade, and because users will be expected to upgrade in a single step
> between stable releases, that means these two types of changes are mutually
> exclusive for the entire release cycle.

So with the goal of trying to enumerate possible solutions, it sounds
some combination of:

(a) disallowing moving problematic files between packages, with possibly some
QA tools to enforce this
(b) keeping the next release cycle *short*, say only a year
(c) requiring that dpkg be upgraded first, and having dpkg and
related tools understand the concept of usrmerge and the
fact that /{bin,lib,sbin} and /usr/{bin,lib,sbin} are identical
for usrmerged systems

might be possible paths forward.  Do you agree?  What are other
possible solutions?

- Ted