Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[David Kalnischkies]

On Sun, Sep 12, 2021 at 03:10:27AM +, Paul Wise wrote:
> On Fri, Sep 10, 2021 at 6:03 PM David Kalnischkies wrote:
> > Because this thread started with the idea to switch the default of d-i
> > and co to another URI. If you target only apt then you still need
> > a solution for d-i and a way to convert whatever d-i had into what apt
> > gets in the end (of the installation).
> 
> ISTR the future of creating new Debian installations is to move from
> debootstrap to dpkg/apt. As an interim step, debootstrap could move
> from doing its own downloads to passing the appropriate
> APT_CONFIG/DPKG_ROOT/etc to `apt download`.
> 
> https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap

The spec deals with the installation of the essential set.
APT isn't essential – it is 'only' one of the first packages installed
after the bootstrap is done, usually at least.

Moving {,c}debootstrap to use apt means you increase the system
requirements from "can execute debootstrap" all the way up to "is
a fully bootstrapped Debian-based system". At which point you could
just use mmdebstrap instead of debootstrap and be done.

I am not involved with d-i to know if they would plan such a move, but
I have at least never heard of it and it seems outside the linked spec.
You might have confused this with the pipe-dream of obsoleting
mmdebstrap at some far away in the future point by folding it into apt
directly. The spec is one (of the many) pre-requirements for that.


Even if we do, that would move the goal post only slightly as you still
have the problem that the conf used to create the system might very well
not be the conf that can be used by the created system (as a trivial
example some old apt versions do not support https). That doesn't really
change regardless of using anna, debootstrap, apt or whatever else.


Best regards

David Kalnischkies

P.S.: Having apt be involved in its own bootstrap reminds me of that
time when I saved myself from drowning in a swamp by pulling on my hair…
https://en.wikipedia.org/wiki/Baron_Munchausen#Fictional_character


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Holger Levsen]

On Sun, Sep 12, 2021 at 03:10:27AM +, Paul Wise wrote:
> ISTR the future of creating new Debian installations is to move from
> debootstrap to dpkg/apt. As an interim step, debootstrap could [...]

I've switched all my occurances of using debootstrap to mmdebstrap and
am a very happy user of it.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Paul Wise]

On Fri, Sep 10, 2021 at 6:03 PM David Kalnischkies wrote:

> Because this thread started with the idea to switch the default of d-i
> and co to another URI. If you target only apt then you still need
> a solution for d-i and a way to convert whatever d-i had into what apt
> gets in the end (of the installation).

ISTR the future of creating new Debian installations is to move from
debootstrap to dpkg/apt. As an interim step, debootstrap could move
from doing its own downloads to passing the appropriate
APT_CONFIG/DPKG_ROOT/etc to `apt download`.

https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Fri, Sep 10, 2021 at 08:02:42PM +0200, David Kalnischkies wrote:

On Fri, Sep 10, 2021 at 11:08:38AM -0400, Michael Stone wrote:

On Fri, Sep 10, 2021 at 04:33:42PM +0200, David Kalnischkies wrote:
> On Thu, Sep 09, 2021 at 08:53:21AM -0400, Michael Stone wrote:
> > The only thing I could see that would be a net gain would be to generalizes
> > sources.list more. Instead of having a user select a specific protocol and
> > path, allow the user to just select high-level objects. Make this a new
> > pseudo-protocol for backward compatibility, and introduce something like
> >   deb apt:// suite component[s]
> > so the default could be something like
> >   deb apt:// bullseye main
> >   deb apt:// bullseye/updates main
> > then the actual protocols, servers, and paths could be managed by various
> > plugins and overridden by configuration directives in apt.conf.d. For
>
> In this scheme the Debian bullseye main repo has the same 'URI' as the
> Darts bullseye main repo. So, you would need to at least include an
> additional unique identifier of the likes of Debian and Darts, but
> who is to assign those URNs?
> (Currently we are piggybacking on the domain name system for this)

I have no idea what darts is, so I don't have an answer. :)


"Darts" was just a play on "bullseye". It is not hard to imagine
a repository which has the same suite and component(s) but is not
Debian itself. As a pseudo-random [= its in an other topic here] real
example you can take Wine (https://dl.winehq.org/wine-builds/debian/).
So to what is "deb apt:// bullseye main" referring? Debian or Wine?

And to pre-empt the most common response: As an apt dev I can assure you
that we won't accept a solution involving "I am on Debian, so it means
Debian" as that is impossible to correctly guess programmatically (for
example on derivatives using a small overlay repo).


I'd considered adding a scope to the model, e.g., apt://debian.org but 
removed it for simplicity. If that was a desired feature then there'd 
either have to be some sort of well-known path or such a distribution 
would need to provide a policy plugin for that scope. Alternatively, 
they could just use the existing http/https/whatever syntax in 
sources.list for their overlay if they didn't want to bother. Same for 
third party repos. 


> > their thing, and a plugin like auto-apt-proxy can override defaults to do
> > something more complicated, using more policy-friendly .d configurations
> > rather than figuring out a way to rewrite some other package's configuration
> > file.
>
> JFTR: auto-apt-proxy has nothing to do with sources. It is true that
> apt-cacher-ng (and perhaps others) have a mode of operation in which you
> prefix the URI of your (in that case caching) proxy to the URI of the
> actual repo, but that isn't how a proxy usually operates and/or is
> configured.

I have no idea what you're saying here.


And I have no idea if you know what you are talking about.

auto-apt-proxy already uses an interface apt provides to configure the
proxy at runtime. It isn't in the business of modifying sources.list nor
has it any interest in that. So you using it as an example for a plugin
who could use your proposed scheme to modify the sources at runtime
makes no sense.


The concern I was responding to was that switching to https breaks the 
case of using auto-apt-proxy to cache the transaction. Just turning the 
proxy on and off isn't sufficient if the default sources.list uses https 
instead of http--you'd currently have to both turn the proxy on *and* 
change sources.list from http to https. Hence my musings on whether it's 
possible/desirable to separate the configuration of what to use as a 
transport from the configuration of what repository is desired.


More generally, if we're talking about changing the default way that 
people interact with debian it just seems like a good time to ponder 
whether specifying sources the same way we did in 1998 makes sense given 
how many changes there have been to expectations about how we use
internet resources. Maybe the answer is yes, and I'm not arguing that 
there has to be a change or that what I threw out as a possibility is 
the right answer, but it does seem worth considering.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[David Kalnischkies]

On Fri, Sep 10, 2021 at 11:08:38AM -0400, Michael Stone wrote:
> On Fri, Sep 10, 2021 at 04:33:42PM +0200, David Kalnischkies wrote:
> > On Thu, Sep 09, 2021 at 08:53:21AM -0400, Michael Stone wrote:
> > > The only thing I could see that would be a net gain would be to 
> > > generalizes
> > > sources.list more. Instead of having a user select a specific protocol and
> > > path, allow the user to just select high-level objects. Make this a new
> > > pseudo-protocol for backward compatibility, and introduce something like
> > >   deb apt:// suite component[s]
> > > so the default could be something like
> > >   deb apt:// bullseye main
> > >   deb apt:// bullseye/updates main
> > > then the actual protocols, servers, and paths could be managed by various
> > > plugins and overridden by configuration directives in apt.conf.d. For
> > 
> > In this scheme the Debian bullseye main repo has the same 'URI' as the
> > Darts bullseye main repo. So, you would need to at least include an
> > additional unique identifier of the likes of Debian and Darts, but
> > who is to assign those URNs?
> > (Currently we are piggybacking on the domain name system for this)
> 
> I have no idea what darts is, so I don't have an answer. :)

"Darts" was just a play on "bullseye". It is not hard to imagine
a repository which has the same suite and component(s) but is not
Debian itself. As a pseudo-random [= its in an other topic here] real
example you can take Wine (https://dl.winehq.org/wine-builds/debian/).
So to what is "deb apt:// bullseye main" referring? Debian or Wine?

And to pre-empt the most common response: As an apt dev I can assure you
that we won't accept a solution involving "I am on Debian, so it means
Debian" as that is impossible to correctly guess programmatically (for
example on derivatives using a small overlay repo).


> > Also, but just as an aside, whatever clever system you think of apt
> > could be using, you still need a rather simple system for the likes of
> > tools which come before apt like the installers/bootstrappers as they
> > are not (all) using apt, especially not in the very early stages, and
> > a mapping between them.
> 
> I'm not sure why you think I need that? The goal of my musings is to

Because this thread started with the idea to switch the default of d-i
and co to another URI. If you target only apt then you still need
a solution for d-i and a way to convert whatever d-i had into what apt
gets in the end (of the installation).

The configuration option which only works with apt tools already
exists in the form of the mirror method…


> > > their thing, and a plugin like auto-apt-proxy can override defaults to do
> > > something more complicated, using more policy-friendly .d configurations
> > > rather than figuring out a way to rewrite some other package's 
> > > configuration
> > > file.
> > 
> > JFTR: auto-apt-proxy has nothing to do with sources. It is true that
> > apt-cacher-ng (and perhaps others) have a mode of operation in which you
> > prefix the URI of your (in that case caching) proxy to the URI of the
> > actual repo, but that isn't how a proxy usually operates and/or is
> > configured.
> 
> I have no idea what you're saying here.

And I have no idea if you know what you are talking about.

auto-apt-proxy already uses an interface apt provides to configure the
proxy at runtime. It isn't in the business of modifying sources.list nor
has it any interest in that. So you using it as an example for a plugin
who could use your proposed scheme to modify the sources at runtime
makes no sense.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Fri, Sep 10, 2021 at 04:33:42PM +0200, David Kalnischkies wrote:

On Thu, Sep 09, 2021 at 08:53:21AM -0400, Michael Stone wrote:

The only thing I could see that would be a net gain would be to generalizes
sources.list more. Instead of having a user select a specific protocol and
path, allow the user to just select high-level objects. Make this a new
pseudo-protocol for backward compatibility, and introduce something like
  deb apt:// suite component[s]
so the default could be something like
  deb apt:// bullseye main
  deb apt:// bullseye/updates main
then the actual protocols, servers, and paths could be managed by various
plugins and overridden by configuration directives in apt.conf.d. For


In this scheme the Debian bullseye main repo has the same 'URI' as the
Darts bullseye main repo. So, you would need to at least include an
additional unique identifier of the likes of Debian and Darts, but
who is to assign those URNs?
(Currently we are piggybacking on the domain name system for this)


I have no idea what darts is, so I don't have an answer. :)


Also, but just as an aside, whatever clever system you think of apt
could be using, you still need a rather simple system for the likes of
tools which come before apt like the installers/bootstrappers as they
are not (all) using apt, especially not in the very early stages, and
a mapping between them.


I'm not sure why you think I need that? The goal of my musings is to 
separate the definition of what set of debian packages to use from the 
decision of how to get those packages *when using apt*, so that there 
are fewer things to consider in the common case, and to allow new 
capabilities in uncommon cases. If someone's using some other tool, why 
would anyone assume that the same configuration syntax would work? Just 
use the same configuration file you use now, and ignore all of this. If 
you want configurations to match across tools, then ignore all of this 
again and keep the same sources.list syntax you're already using that 
presumably does what you want.



If someone wants to use tor by default but fall back to https if it's
unreachable, they can do that. If someone wants to use a local proxy via
http but https if they're not on their local network, they can do that. New
installations could default to https, existing installations can keep doing


You can do most of the fallback part with the mirror method backed by
a local file. It is of no concern to apt how that file comes to be, so
you could create it out of a massive amount of options or simply by
hand. I do think if the creation is tool-based it shouldn't be apt
as I envision far too many unique snowflakes for a one-size-fits-all
approach.


The intent would be to make it easier to plug other tools into apt,
not to have the apt codebase do everything.


their thing, and a plugin like auto-apt-proxy can override defaults to do
something more complicated, using more policy-friendly .d configurations
rather than figuring out a way to rewrite some other package's configuration
file.


JFTR: auto-apt-proxy has nothing to do with sources. It is true that
apt-cacher-ng (and perhaps others) have a mode of operation in which you
prefix the URI of your (in that case caching) proxy to the URI of the
actual repo, but that isn't how a proxy usually operates and/or is
configured.


I have no idea what you're saying here. 

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[David Kalnischkies]

On Thu, Sep 09, 2021 at 08:53:21AM -0400, Michael Stone wrote:
> The only thing I could see that would be a net gain would be to generalizes
> sources.list more. Instead of having a user select a specific protocol and
> path, allow the user to just select high-level objects. Make this a new
> pseudo-protocol for backward compatibility, and introduce something like
>   deb apt:// suite component[s]
> so the default could be something like
>   deb apt:// bullseye main
>   deb apt:// bullseye/updates main
> then the actual protocols, servers, and paths could be managed by various
> plugins and overridden by configuration directives in apt.conf.d. For

In this scheme the Debian bullseye main repo has the same 'URI' as the
Darts bullseye main repo. So, you would need to at least include an
additional unique identifier of the likes of Debian and Darts, but
who is to assign those URNs?
(Currently we are piggybacking on the domain name system for this)

Also, but just as an aside, whatever clever system you think of apt
could be using, you still need a rather simple system for the likes of
tools which come before apt like the installers/bootstrappers as they
are not (all) using apt, especially not in the very early stages, and
a mapping between them.


> If someone wants to use tor by default but fall back to https if it's
> unreachable, they can do that. If someone wants to use a local proxy via
> http but https if they're not on their local network, they can do that. New
> installations could default to https, existing installations can keep doing

You can do most of the fallback part with the mirror method backed by
a local file. It is of no concern to apt how that file comes to be, so
you could create it out of a massive amount of options or simply by
hand. I do think if the creation is tool-based it shouldn't be apt
as I envision far too many unique snowflakes for a one-size-fits-all
approach.

(The Tor to https fallback can be done already if we talk onion services
 to others. You can't fall out of Tor – or redirect into it – through as
 that would allow bad actors to discover who you are/that you have an
 operational tor client installed. proxy configuration you can already
 change programmatically on the fly – a job auto-apt-proxy implements –,
 changing the mirror file with a hook from your network manager would
 be equally easy.)


> their thing, and a plugin like auto-apt-proxy can override defaults to do
> something more complicated, using more policy-friendly .d configurations
> rather than figuring out a way to rewrite some other package's configuration
> file.

JFTR: auto-apt-proxy has nothing to do with sources. It is true that
apt-cacher-ng (and perhaps others) have a mode of operation in which you
prefix the URI of your (in that case caching) proxy to the URI of the
actual repo, but that isn't how a proxy usually operates and/or is
configured.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Fri, Sep 10, 2021 at 09:33:56AM +0200, Helmut Grohne wrote:

Laptops of end-user systems are the target, but also developers. When
people gather at a place (conference, hackspace, private meetup, etc.)
downloading of .debs should just work quickly by default. Many such
sites could easily provide a local cache and a number even do. BSPs tend
to have a blackboard with information including the local mirror to use.
Seriously, how many people change their mirror when they go to a BSP? If
we installed auto-apt-proxy by default, much of the local caching would
just work.


I think you'd get a lot of pushback on installing auto-apt-proxy by 
default. If that's a proposal, make it seperately and not in this 
thread. 


The thing we seem to be disagreeing on is what is more important? https
by default or quick and efficient downloads? Some may think that our
CDN can handle the load just fine and the effects of caching are
peanuts. I can attest that those peanuts are 4TB/year (equivalent to 8
days waiting for downloads) for me.



I see that we've given up on a global network of independently-managed
mirrors and that CDNs are way easier at this time. While initially CDNs
had bad reliability issues, those have completely vanished in my
experience. On the other hand, local caching still outperforms CDNs by a
factor of 10 or so. I'd love to make it the default.


I use a cache out of habit and to be a good netizen, but my internet 
connection is fast enough these days that it's basically a noop at best 
and a slight slowdown at worst. I had to move the cache from slow/cheap 
spinning disk to reasonably fast SSD to get to that point. If you're 
downloading the same stuff over and over (e.g., for testing or somesuch) 
it can be a big win, especially for VMs on a virtualized network 
connection, or if you're managing a large infrastructure, but 
for normal use with a couple of instances? It's just not worth the 
trouble. 

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Fri, Sep 10, 2021 at 12:00:57PM +0200, Timo Röhling wrote:

* Michael Stone  [2021-09-08 19:25]:
I think the issue isn't certificate validation, it's that https 
proxy requests are made via CONNECT rather than GET. You could 
theoretically rewrite the proxy mechanism to MITM the CONNECT, but 
that wouldn't be a drop-in replacement. I suppose you could instead 
add an apt option to pass the https request to the proxy via GET 
instead of using CONNECT, but I think that also won't necessarily 
work on an existing proxy.

apt-cacher-ng has a second mode of operation where you can prefix
the source URL with the proxy URL, i.e.

deb http://proxyhost:3142/deb.debian.org/debian unstable main

Maybe we could introduce this as an "official" APT proxy mode, where
http(s)://REPO gets replaced by http://PROXY_URL/REPO (and the proxy
can decide whether or not to fetch via HTTPS as an implementation
detail)?


I'd much rather see something more like I proposed earlier (splitting the 
selection of what suites/components to install from the policy of how to 
obtain them) rather than further layering+confusing these two concepts 
within sources.list.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Timo Röhling]

* Michael Stone  [2021-09-08 19:25]:
I think the issue isn't certificate validation, it's that https proxy 
requests are made via CONNECT rather than GET. You could theoretically 
rewrite the proxy mechanism to MITM the CONNECT, but that wouldn't be 
a drop-in replacement. I suppose you could instead add an apt option 
to pass the https request to the proxy via GET instead of using 
CONNECT, but I think that also won't necessarily work on an existing 
proxy.

apt-cacher-ng has a second mode of operation where you can prefix
the source URL with the proxy URL, i.e.

deb http://proxyhost:3142/deb.debian.org/debian unstable main

Maybe we could introduce this as an "official" APT proxy mode, where
http(s)://REPO gets replaced by http://PROXY_URL/REPO (and the proxy
can decide whether or not to fetch via HTTPS as an implementation
detail)?

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Eduard Bloch]

Hallo,
* Michael Stone [Wed, Sep 08 2021, 07:25:26PM]:
> On Wed, Sep 08, 2021 at 03:56:14PM +0200, Ansgar wrote:
> > On Wed, 2021-09-08 at 15:41 +0200, Helmut Grohne wrote:
> > > On Wed, Sep 08, 2021 at 02:01:03PM +0200, Ansgar wrote:
> > > > So what do you suggest then? Tech-ctte as with merged-/usr? Or a
> > > > GR? Or
> > > > something else?
> > >
> > > I propose that the proponents pay the cost. In this case, it is a bit
> > > unclear what that means precisely (which likely is the reason they
> > > haven't done it already). At the very least though, apt install
> > > auto-apt-proxy should continue to work on a default installation in a
> > > sensible way.
> >
> > I can file a bug for auto-apt-proxy to include an apt.conf snippet
> > saying
> >
> >  Acquire::https::Verify-Peer false;
> >
> > That clearly makes it work again
>
> I think the issue isn't certificate validation, it's that https proxy
> requests are made via CONNECT rather than GET. You could theoretically
> rewrite the proxy mechanism to MITM the CONNECT, but that wouldn't be a
> drop-in replacement. I suppose you could instead add an apt option to pass
> the https request to the proxy via GET instead of using CONNECT, but I think

Precisely. Current handling of HTTPS on a caching proxy is either
impossible or PITA for the user, as long as a such mixed behavior is not
configurable.  apt-cacher-ng works around that by telling users to
disguise https URLs as HTTP with a special marker for protocol switch
(ugly, I know).

Also keep in mind that it off-loads the encryption work to the proxy,
but that might be even intentional.

> that also won't necessarily work on an existing proxy.

Speaking at least for ACNG, my assumption was that it would support that
but I was wrong. TODO created,
https://salsa.debian.org/blade/apt-cacher-ng/-/issues/11 .

> If we're imagining apt options, something like
> Acquire::https::Force-Proxy-HTTP true;
> would probably be more useful for this specific case (not that I think it's
> a great idea--too much potential for surprise).

I would make it a list of trusted proxy hosts and a special value ALL.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994032 created.

Best regards,
Eduard.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Helmut Grohne]

On Wed, Sep 08, 2021 at 07:12:18PM -0400, Michael Stone wrote:
> Why not simply automate setting it at install time using preseed? I'm
> honestly not sure who the target audience for auto-apt-proxy is--apparently
> someone who has an infrastructure including a proxy, possibly the ability to
> set dns records, etc., but can't change defaults at install time or via some
> sort of runtime configuration management?

I believe that the target audience is non-tech end-users. Most
orgnizations already optimized their way of downloading .debs via some
way (e.g. auto-apt-proxy) or another. As you point out, organizations
are easily able to deviate from defaults. They're not our primary target
with defaults.

Laptops of end-user systems are the target, but also developers. When
people gather at a place (conference, hackspace, private meetup, etc.)
downloading of .debs should just work quickly by default. Many such
sites could easily provide a local cache and a number even do. BSPs tend
to have a blackboard with information including the local mirror to use.
Seriously, how many people change their mirror when they go to a BSP? If
we installed auto-apt-proxy by default, much of the local caching would
just work.

The thing we seem to be disagreeing on is what is more important? https
by default or quick and efficient downloads? Some may think that our
CDN can handle the load just fine and the effects of caching are
peanuts. I can attest that those peanuts are 4TB/year (equivalent to 8
days waiting for downloads) for me.

I see that we've given up on a global network of independently-managed
mirrors and that CDNs are way easier at this time. While initially CDNs
had bad reliability issues, those have completely vanished in my
experience. On the other hand, local caching still outperforms CDNs by a
factor of 10 or so. I'd love to make it the default.

Helmut

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Timo Röhling]

* Michael Stone  [2021-09-09 09:05]:
Because the controversy concerning changing the default is over 
whether it's reasonable for someone using auto-apt-proxy to have to 
manage additional configuration settings.

Ah, I understand your point now and I agree.
It would be an inconvenience, yes, not but much more.

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Thu, Sep 09, 2021 at 02:54:21PM +0200, Timo Röhling wrote:

* Michael Stone  [2021-09-09 08:32]:
I'm honestly not sure who the target audience for auto-apt-proxy 
is--apparently someone who has an infrastructure including a 
proxy, possibly the ability to set dns records, etc., but can't 
change defaults at install time or via some sort of runtime 
configuration management?

The same reason you might want to deploy WPAD instead of preconfiguring
proxy settings in web browsers: flexibility. For example, we use
auto-apt-proxy for laptops which roam between different networks.
It's simple to configure, has virtually no maintenance overhead and
degrades gracefully.
None of that speaks to whether an organization that deploys such a 
thing has the ability to manage other configuration settings, even 
if for some settings there's a desire for additional flexibility.

I don't understand your point.

You asked for a target audience for auto-apt-proxy. I gave you a
legitimate (and real-world) example for such a deployment. Why
should it matter whether or not an organization has other
configuration facilities?


Because the controversy concerning changing the default is over whether 
it's reasonable for someone using auto-apt-proxy to have to manage 
additional configuration settings. If the target audience is someone who 
has the ability to manage the infrastructure required by auto-apt-proxy 
but not the ability to manage anything else then I guess it's a valid 
criticism (but I have trouble envisioning that). If the auto-apt-proxy 
target audience can manage both the proxy infrastructure *and* 
sources.list (either at install time or run time) then the criticism is 
basically academic and not generally a real-world issue.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Timo Röhling]

* Michael Stone  [2021-09-09 08:32]:
I'm honestly not sure who the target audience for auto-apt-proxy 
is--apparently someone who has an infrastructure including a 
proxy, possibly the ability to set dns records, etc., but can't 
change defaults at install time or via some sort of runtime 
configuration management?

The same reason you might want to deploy WPAD instead of preconfiguring
proxy settings in web browsers: flexibility. For example, we use
auto-apt-proxy for laptops which roam between different networks.
It's simple to configure, has virtually no maintenance overhead and
degrades gracefully.
None of that speaks to whether an organization that deploys such a 
thing has the ability to manage other configuration settings, even if 
for some settings there's a desire for additional flexibility.

I don't understand your point.

You asked for a target audience for auto-apt-proxy. I gave you a
legitimate (and real-world) example for such a deployment. Why
should it matter whether or not an organization has other
configuration facilities?

As another example, nobody says: "the target audience for DHCP is an
organization who has an infrastructure with full control over a
network, including IP address management, but apparently can't
configure IP addresses at install time or via some sort of runtime
configuration management" and concludes that DHCP is useless.

auto-apt-proxy (or DHCP for that matter) *is* the runtime
configuration management, and a quite efficient one at that.

Cheers
Timo


--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Thu, Sep 09, 2021 at 11:54:44AM +0530, Pirate Praveen wrote:

Why can't auto-apt-proxy ask this as a debconf question? I also like 
auto-apt-proxy but I agree with  this, someone needing auto-apt-proxy should be 
able to change the default as well.


I don't really see why adding another debconf question would be better 
than just preseeding the existing one.


The only thing I could see that would be a net gain would be to 
generalizes sources.list more. Instead of having a user select a 
specific protocol and path, allow the user to just select high-level 
objects. Make this a new pseudo-protocol for backward compatibility, and 
introduce something like

  deb apt:// suite component[s]
so the default could be something like
  deb apt:// bullseye main
  deb apt:// bullseye/updates main
then the actual protocols, servers, and paths could be managed by 
various plugins and overridden by configuration directives in 
apt.conf.d. For existing configurations it's a no-op but it allows more 
flexibility & new plugins/protocols in the future without having to 
micromanage sources.list. If someone wants to use tor by default but 
fall back to https if it's unreachable, they can do that. If someone 
wants to use a local proxy via http but https if they're not on their 
local network, they can do that. New installations could default to 
https, existing installations can keep doing their thing, and a plugin 
like auto-apt-proxy can override defaults to do something more 
complicated, using more policy-friendly .d configurations rather than 
figuring out a way to rewrite some other package's configuration file.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Thu, Sep 09, 2021 at 08:36:28AM +0200, Timo Röhling wrote:

* Michael Stone  [2021-09-08 19:12]:
Why not simply automate setting it at install time using preseed? 
I'm honestly not sure who the target audience for auto-apt-proxy 
is--apparently someone who has an infrastructure including a proxy, 
possibly the ability to set dns records, etc., but can't change 
defaults at install time or via some sort of runtime configuration 
management?

The same reason you might want to deploy WPAD instead of preconfiguring
proxy settings in web browsers: flexibility. For example, we use
auto-apt-proxy for laptops which roam between different networks.
It's simple to configure, has virtually no maintenance overhead and
degrades gracefully.


None of that speaks to whether an organization that deploys such a thing 
has the ability to manage other configuration settings, even if for 
some settings there's a desire for additional flexibility.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Timo Röhling]

* Michael Stone  [2021-09-08 19:12]:
Why not simply automate setting it at install time using preseed? I'm 
honestly not sure who the target audience for auto-apt-proxy 
is--apparently someone who has an infrastructure including a proxy, 
possibly the ability to set dns records, etc., but can't change 
defaults at install time or via some sort of runtime configuration 
management?

The same reason you might want to deploy WPAD instead of preconfiguring
proxy settings in web browsers: flexibility. For example, we use
auto-apt-proxy for laptops which roam between different networks.
It's simple to configure, has virtually no maintenance overhead and
degrades gracefully. 


Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Pirate Praveen]

2021, സെപ്റ്റംബർ 9 4:42:18 AM IST, Michael Stone ൽ എഴുതി
>On Wed, Sep 08, 2021 at 01:09:13PM +0200, Helmut Grohne wrote:
>>Enabling https by default quite simply breaks the simple recipe of
>>installing auto-apt-proxy. Would you agree with auto-apt-proxy's
>>postinst automatically editing your sources.list to drop the s out of
>>https? The answer repeatedly given in this thread to do so manually is
>>very unsatisfying.
>
>Why not simply automate setting it at install time using preseed? I'm 
>honestly not sure who the target audience for auto-apt-proxy 
>is--apparently someone who has an infrastructure including a proxy, 
>possibly the ability to set dns records, etc., but can't change defaults 
>at install time or via some sort of runtime configuration management?
>

Why can't auto-apt-proxy ask this as a debconf question? I also like 
auto-apt-proxy but I agree with  this, someone needing auto-apt-proxy should be 
able to change the default as well.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Timothy M Butterworth]

All,

I am against automatically setting HTTPS. Their should be an option in
the installer to set or unset HTTPS while configuring the mirror! I
like a lot of folks am on a metered internet connection with a UTM
proxy firewall. I have multiple computers that need patched and only
having to download the package once is a great convenience to both
data usage and download time as my internet is not fast 4G LTE usually
only operating at 3G speed.

Tim

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Wed, Sep 08, 2021 at 03:56:14PM +0200, Ansgar wrote:

On Wed, 2021-09-08 at 15:41 +0200, Helmut Grohne wrote:

On Wed, Sep 08, 2021 at 02:01:03PM +0200, Ansgar wrote:
> So what do you suggest then? Tech-ctte as with merged-/usr? Or a
> GR? Or
> something else?

I propose that the proponents pay the cost. In this case, it is a bit
unclear what that means precisely (which likely is the reason they
haven't done it already). At the very least though, apt install
auto-apt-proxy should continue to work on a default installation in a
sensible way.


I can file a bug for auto-apt-proxy to include an apt.conf snippet
saying

 Acquire::https::Verify-Peer false;

That clearly makes it work again


I think the issue isn't certificate validation, it's that https proxy 
requests are made via CONNECT rather than GET. You could theoretically 
rewrite the proxy mechanism to MITM the CONNECT, but that wouldn't be a 
drop-in replacement. I suppose you could instead add an apt option to 
pass the https request to the proxy via GET instead of using CONNECT, 
but I think that also won't necessarily work on an existing proxy.


If we're imagining apt options, something like 
Acquire::https::Force-Proxy-HTTP true;
would probably be more useful for this specific case (not that I think 
it's a great idea--too much potential for surprise).

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Michael Stone]

On Wed, Sep 08, 2021 at 01:09:13PM +0200, Helmut Grohne wrote:

Enabling https by default quite simply breaks the simple recipe of
installing auto-apt-proxy. Would you agree with auto-apt-proxy's
postinst automatically editing your sources.list to drop the s out of
https? The answer repeatedly given in this thread to do so manually is
very unsatisfying.


Why not simply automate setting it at install time using preseed? I'm 
honestly not sure who the target audience for auto-apt-proxy 
is--apparently someone who has an infrastructure including a proxy, 
possibly the ability to set dns records, etc., but can't change defaults 
at install time or via some sort of runtime configuration management?

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Helmut Grohne]

On Wed, Sep 08, 2021 at 02:01:03PM +0200, Ansgar wrote:
> So what do you suggest then? Tech-ctte as with merged-/usr? Or a GR? Or
> something else?

I propose that the proponents pay the cost. In this case, it is a bit
unclear what that means precisely (which likely is the reason they
haven't done it already). At the very least though, apt install
auto-apt-proxy should continue to work on a default installation in a
sensible way.

> >  * Concerns are ignored. <- This is where we are with https-default.
> 
> It's also where we are with keep-http-as-default.

I don't think https resolves any concerns. It's merely best-practice. In
the absence of reason not to use https, https should be preferred. As it
happens, we figured a reason not to use https.

> > Change has a cost. I do not want to pay the cost for either of these
> > changes.
> 
> Then we could never change anything.

Untrue. You get to choose which changes you want to pay the cost for.
For instance, I want Debian to be cross buildable and bootstrappable.
Holger, Mattia and a few others want Debian to be reproducible. You
don't get to pay the cost for those changes. Change is possible in a way
that limits cost for uninterested people. The contentious changes are
the ones where the initiators fail to pay the cost.

> To keep up with merged-/usr: keeping non-merged-/usr also has a cost.
> Nobody wants to pay the cost for it.

That is very true. With merged-/usr, I suppose most grief arises from
the way the transition was (not) planned and only a minority takes issue
with the goal.

Helmut

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Tim Woodall]

On Wed, 8 Sep 2021, Helmut Grohne wrote:


I do see the advantages of using https. I do not see how to not make it
happen without breaking relevant use cases. Same with the /usr-merge. I
do see the advantages. I've stopped counting the things that broke. Most
recent one is the uucp FTBFS. Change has a cost. I do not want to pay
the cost for either of these changes.



This is a bit tongue in cheek, but how about these sites where the .debs
are downloaded from publish their *private* key? They openly accept that
anyone can MITM them.

That way people who want to see "https" can see it. And people who want
the benefits of http can, with a bit of work, simulate that.

It also nicely addresses my concern which is that the next demand will
be to drop http (because when you visit the site with a webbrowser users
start getting a warning that the site is also available over http or
something like that because the google/firefox dream seems to be that
you cannot use http even where https doesn't add anything.)

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Helmut Grohne]

On Wed, Sep 08, 2021 at 01:37:37PM +0200, Ansgar wrote:
> Maybe we should just find out who is responsible for this decision and
> reassign the bug to them.  The installer team maintaining d-i and
> debootstrap or the mirror team seem reasonable choices?

We've already tried that approach on the /usr-merge and the resulting
transition is miserable. Let's not repeat that mistake.

It's the same pattern actually:
 * People propose a change that does have positive effects, though some
   find the positive effects unimportant.
 * Other people disagree and raise concerns.
 * Concerns are ignored. <- This is where we are with https-default.
 * Change is being implemented anyway.
 * Stuff breaks. <- This is where we are with /usr-merge.

This is frustrating.

I do see the advantages of using https. I do not see how to not make it
happen without breaking relevant use cases. Same with the /usr-merge. I
do see the advantages. I've stopped counting the things that broke. Most
recent one is the uucp FTBFS. Change has a cost. I do not want to pay
the cost for either of these changes.

Helmut

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Helmut Grohne]

Hi,

On Thu, Sep 02, 2021 at 10:22:15AM +0900, Hideki Yamane wrote:
>  Some users want proxy but they can configure their settings.
>  So just change "default setting for {deb,security}.debian.org"
>  is not so harmful, IMO. 

I fear you are putting this upside down. In reality, some sites (not
users) want their users to use their local cache (transparently or not).

>  - Users can choose other mirror than https://deb.debian.org

As far as I can tell, most users don't want to make a choice here. They
want downloading packages to just work. Preferably fast. It is the
"fast" thing that you are breaking here.

>  - Caching .debs from security.debian.org is not so huge, I guess
>(maybe except linux-image).

Not sure why security.d.o is singled out here. The switch is only
reasonable on the whole or not at all. And there the whole volume of
packages counts.

Enabling https by default quite simply breaks the simple recipe of
installing auto-apt-proxy. Would you agree with auto-apt-proxy's
postinst automatically editing your sources.list to drop the s out of
https? The answer repeatedly given in this thread to do so manually is
very unsatisfying.

So I actually argue for installing auto-apt-proxy by default and inside
d-i. That is in direct conflict with the proposed change here.

Unfortunately, I don't see consensus for this, but at the same time I
neither see consensus for enabling https by default. It's a matter that
keeps popping up and people disagreeing on over and over again. The one
thing that we have clearly understood at this point is that one size
does not fit everyone. Either way makes some people unhappy.

Helmut

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Jeremy Stanley]

On 2021-09-02 12:27:34 -0400 (-0400), Roberto C. Sánchez wrote:
[...]
> In this context, it might make sense to describe using HTTPS as
> the transport for APT operations is providing "default
> confidentiality".

Which, as similarly discussed, it really doesn't do either (because
of deterministic blob sizes for publicly served files, current SNI
implementations leaking hostnames, general state of the CA and CDN
industries...), so suggesting that may also give users a false
impression. If they really do need confidentiality of their package
installation, they're probably better off doing updates over Tor or
a some VPN which does batching/interleaving of bulk transfers with
some cover traffic, or keeping a private local package mirror.

But to a great extent the degree of confidentiality they can expect
really depends on who they're trying to hide their traffic from. If
their concern is that a company headquartered in the USA might be
tracking the packages they're downloading from deb.debian.org, then
that's already a possibility even with HTTPS: the site is currently
fronted by the Fastly CDN service which terminates TLS encryption
for those HTTPS requests in order to be able to cache them globally.
Of course, without a CDN, mirror site operators can track what
packages you're downloading from them over HTTPS as well.

More generally, what I'm saying is don't try to paint this change as
actually implementing any significant amount of new security or
privacy for Debian users, that would be disingenuous. Just say the
default is switching to HTTPS because that's what users, by and
large, expect today.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Roberto C . Sánchez]

On Thu, Sep 02, 2021 at 04:08:37PM +, Jeremy Stanley wrote:
> On 2021-09-02 10:22:15 +0900 (+0900), Hideki Yamane wrote:
> [...]
> >  Providing "default secure setting" is good message to users.
> [...]
> 
> As previously covered, I'd suggest steering clear of referring to
> this as adding "default security." That implies APT wasn't already
> effectively secure over plain HTTP, and may give a false impression
> that HTTPS is addressing gaps in the existing apt-secure design.
> 
> This change is more about recognizing HTTPS as the primary transport
> protocol for the modern Web, not sending mixed signals regarding the
> general security risks posed by plain HTTP when used for unrelated
> purposes, and no longer needing to repeatedly explain to users that
> Debian has gone to great lengths to implement package distribution
> security which doesn't really depend at all on transport layer
> encryption.

In this context, it might make sense to describe using HTTPS as the
transport for APT operations is providing "default confidentiality".

Regards,

-Roberto

-- 
Roberto C. Sánchez

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Jeremy Stanley]

On 2021-09-02 10:22:15 +0900 (+0900), Hideki Yamane wrote:
[...]
>  Providing "default secure setting" is good message to users.
[...]

As previously covered, I'd suggest steering clear of referring to
this as adding "default security." That implies APT wasn't already
effectively secure over plain HTTP, and may give a false impression
that HTTPS is addressing gaps in the existing apt-secure design.

This change is more about recognizing HTTPS as the primary transport
protocol for the modern Web, not sending mixed signals regarding the
general security risks posed by plain HTTP when used for unrelated
purposes, and no longer needing to repeatedly explain to users that
Debian has gone to great lengths to implement package distribution
security which doesn't really depend at all on transport layer
encryption.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Hideki Yamane]

Hi,

On Wed, 01 Sep 2021 07:46:07 -0700
Russ Allbery  wrote:
> >> I believe that the discussion has later identified that doing so would
> >> break squid-deb-proxy-client and auto-apt-proxy. Given that the
> >> security benefits are not strong (beyond embracing good habits), I
> >> think the reasonable thing to do is keep preferring http.
> 
> > That is an opt-in choice which likely only a small number of users use.
> > People wanting to use a caching proxy can just switch to http as part of
> > this choice; it doesn't seem a good reason to not use https by default
> > for all other users.
> 
> Completely agreed.

 Providing "default secure setting" is good message to users.


 Some users want proxy but they can configure their settings.
 So just change "default setting for {deb,security}.debian.org"
 is not so harmful, IMO. 

 - Users can choose other mirror than https://deb.debian.org
 - Caching .debs from security.debian.org is not so huge, I guess
   (maybe except linux-image).


-- 
Hideki Yamane 

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Russ Allbery]

Ansgar  writes:
> On Wed, 2021-09-01 at 11:15 +0200, Helmut Grohne wrote:

>> I believe that the discussion has later identified that doing so would
>> break squid-deb-proxy-client and auto-apt-proxy. Given that the
>> security benefits are not strong (beyond embracing good habits), I
>> think the reasonable thing to do is keep preferring http.

> That is an opt-in choice which likely only a small number of users use.
> People wanting to use a caching proxy can just switch to http as part of
> this choice; it doesn't seem a good reason to not use https by default
> for all other users.

Completely agreed.

>> Caching packages and transport level encryption are fundamentally
>> incompatible.

> No. You can explicitly configure apt to use a local caching mirror or
> use a trusted TLS certificate for the mirror the proxy impersonates.

Yes.  For example, the approach used by apt-cacher-ng works fine.
Explicitly opting in to a local cache seems desirable.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Ansgar]

On Wed, 2021-09-01 at 11:15 +0200, Helmut Grohne wrote:
> I believe that the discussion has later identified that doing so
> would
> break squid-deb-proxy-client and auto-apt-proxy. Given that the
> security
> benefits are not strong (beyond embracing good habits), I think the
> reasonable thing to do is keep preferring http.

That is an opt-in choice which likely only a small number of users use.
People wanting to use a caching proxy can just switch to http as part
of this choice; it doesn't seem a good reason to not use https by
default for all other users.

> Caching packages and transport level encryption are fundamentally
> incompatible.

No. You can explicitly configure apt to use a local caching mirror or
use a trusted TLS certificate for the mirror the proxy impersonates.


Ansgar

Processed: Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Debian Bug Tracking System]

Processing control commands:

> tags -1 + moreinfo
Bug #992692 [general] general: Use https for {deb,security}.debian.org by 
default
Added tag(s) moreinfo.

-- 
992692: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992692
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Re: Bug#992692: general: Use https for {deb,security}.debian.org by default

[Helmut Grohne]

Control: tags -1 + moreinfo

On Sun, Aug 22, 2021 at 09:56:57PM +0900, Hideki Yamane wrote:
>  As we discussed on -devel(*), it seems that we can enable https for
>  {deb,security}.debian.org by default. With this bug report, I'll
>  collect related things and fix it.

I believe that the discussion has later identified that doing so would
break squid-deb-proxy-client and auto-apt-proxy. Given that the security
benefits are not strong (beyond embracing good habits), I think the
reasonable thing to do is keep preferring http.

Caching packages and transport level encryption are fundamentally
incompatible. Possibly it would make more sense to offer users a choice
between performance and privacy on installation?

Helmut