Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 07, Bernd Zeimetz  wrote:

> There are more than enough ways to keep the entries based on dns
> records in your l3 firewalls uptodate, I can't see how this should
> warrant to keep yet another patch Jan^WMarco.
Not for the form *.domain.tld.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Bernd Zeimetz]

On Tue, 2024-04-02 at 12:04 +0200, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
> 
> > At the time, denyhosts was popular, but it was removed from Debian
> > several years ago.  I remember that, when I dealt with that on my
> > own
> > systems, fail2ban seemed like the obvious replacement, and my
> > impression
> > is that it's pretty widely used nowadays; it's very pluggable but
> > it
> > normally works by adding firewall rules.  Are there any similar
> > popular
> > systems left that rely on editing /etc/hosts.deny?
> Yes, people. I object to removing TCP wrappers support since the
> patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be
> supported by L3 firewalls.
> 

There are more than enough ways to keep the entries based on dns
records in your l3 firewalls uptodate, I can't see how this should
warrant to keep yet another patch Jan^WMarco.

-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Thu, Apr 04, 2024 at 06:42:08PM -0300, Henrique de Moraes Holschuh wrote:
> If libwrap is bringing in complex libs, maybe we could reduce the
> attack surface on libwrap itself?  It would be nice to have a variant
> that only links to the libc and that's it...

Yeah, that's https://bugs.debian.org/1068311 which I linked to elsewhere
in this thread.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Henrique de Moraes Holschuh]

On Tue, Apr 2, 2024, at 07:04, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
>
>> At the time, denyhosts was popular, but it was removed from Debian
>> several years ago.  I remember that, when I dealt with that on my own
>> systems, fail2ban seemed like the obvious replacement, and my impression
>> is that it's pretty widely used nowadays; it's very pluggable but it
>> normally works by adding firewall rules.  Are there any similar popular
>> systems left that rely on editing /etc/hosts.deny?
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

If libwrap is bringing in complex libs, maybe we could reduce the attack 
surface on libwrap itself?  It would be nice to have a variant that only links 
to the libc and that's it...

And that benefits everything that links to TCP wrappers...

I also like to have the (old-school) standard extra layer of protection that 
libwrap can provide. I'd like to find a way to keep it useful for sshd.

-- 
  Henrique de Moraes Holschuh 

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Wed, Apr 03, 2024 at 04:01:34PM -0400, Michael Stone wrote:
> To speed things up for those who really want it, perhaps make
> openssh-client/server dependency-only packages on
> openssh-client/server-nogss? People can choose the less-compatible version
> for this release if they want to, and the default can change next release.
> Pushing back the ability to install the unpatched version for a few more
> years seems suboptimal.

I worry about churn, and about inviting more bugs to do with moving an
awkward combination of conffiles and non-conffile configuration files
between packages.  But maybe.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Michael Stone]

On Tue, Apr 02, 2024 at 01:30:10AM +0100, Colin Watson wrote:

  * add dependency-only packages called something like
openssh-client-gsskex and openssh-server-gsskex, depending on their
non-gsskex alternatives
  * add NEWS.Debian entry saying that people need to install these
packages if they want to retain GSS-API key exchange support
  * add release note saying the same

* for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
  exact timings):

  * add separate openssh-gsskex source package, carrying gssapi.patch
in addition to whatever's in openssh, and whose binary packages
Conflicts/Replaces/Provides the corresponding ones from openssh
  * add some kind of regular CI to warn about openssh-gsskex being out
of date relative to openssh
  * drop gssapi.patch from openssh, except for small patches to
configuration file handling to accept the relevant options with
some kind of informative warning (compare
https://bugs.debian.org/152657)


To speed things up for those who really want it, perhaps make 
openssh-client/server dependency-only packages on 
openssh-client/server-nogss? People can choose the less-compatible 
version for this release if they want to, and the default can change 
next release. Pushing back the ability to install the unpatched version 
for a few more years seems suboptimal.

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Wed, Apr 03, 2024 at 04:38:19PM +0200, Marc Haber wrote:
> On Wed, 03 Apr 2024 14:10:37 +0100, "Jonathan Dowland"
>  wrote:
> >For you and fellow greybeards, perhaps: I'd be surprised if many people
> >younger than us have even heard of tcp wrappers. I don't think the
> >muscle memory of a diminishing set of users is a strong argument,
> >especially given it's a preference rather than a requirement, and
> >alternatives do exist.
> 
> It is possible to have that alternative not present without being
> noticed (for example, a firewall build script failing, but sshd start
> nof failing), whilea security measure built into the very daemon is
> way harder to be accidentally disabled while keeping the daemon
> running.

While I'm still not totally convinced, one possible alternative here is
https://bugs.debian.org/1068311.  That would still mean one more library
than strictly needed (once the GSS-API stuff is split out), but at least
it would be one small library rather than a big linkage chain over 30
times the size.  I could probably justify keeping it in that case.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[RL]

Colin Watson  writes:

> GSS-API key exchange
> 

> However, OpenSSH upstream has long rejected it

> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package


> How does this rough plan sound?
>
>  * for Debian trixie (current testing):
>
>* add dependency-only packages called something like
>  openssh-client-gsskex and openssh-server-gsskex, depending on their
>  non-gsskex alternatives
>* add NEWS.Debian entry saying that people need to install these
>  packages if they want to retain GSS-API key exchange support
>* add release note saying the same

happy to help on release-notes.

Think you've got two audiences:

- people who rely on gss, who may be upgrading over ssh and need to know
  how to avoid being locked out (eg: make sure to install gsskex
  recommended packages before reboot?)

- people who dont use gss, and want to remove it asap: as well as
  removing the gsskex packages would they need to edit sshd_config or
  ssh_config etc -- these can currently contain things like
  'GSSAPIAuthentication no' which would (i assume) stop working (and
  cause sshd to not start) once the gss support is removed(?)

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 12:04:26PM +0200, Marco d'Itri wrote:
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

I suspect OpenSSH upstream would also want me to point out that
DNS-based ACLs are supported by Match blocks without needing a separate
library.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 02, Colin Watson  wrote:

> You could use a drop-in unit to wrap sshd in tcpd, as suggested by the
> Fedora wiki page?  This would avoid exposing sshd's process space to
> libwrap and all the stuff it links to by default.
This would require to switch to socket activation of sshd, which is not 
the default for good reasons.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 12:04:26PM +0200, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
> > At the time, denyhosts was popular, but it was removed from Debian
> > several years ago.  I remember that, when I dealt with that on my own
> > systems, fail2ban seemed like the obvious replacement, and my impression
> > is that it's pretty widely used nowadays; it's very pluggable but it
> > normally works by adding firewall rules.  Are there any similar popular
> > systems left that rely on editing /etc/hosts.deny?
> 
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

It's not about the size of the patch.

You could use a drop-in unit to wrap sshd in tcpd, as suggested by the
Fedora wiki page?  This would avoid exposing sshd's process space to
libwrap and all the stuff it links to by default.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Christian Göttsche]

On Tue, 2 Apr 2024 at 02:30, Colin Watson  wrote:
>
> [I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
> just debian-devel and debian-ssh to avoid potentially spamming them with
> a long discussion.  If you choose to override this then that's your
> call, but please be mindful of upstream's time.]
>
> Following the xz-utils backdoor, I'm reconsidering some choices in
> Debian's OpenSSH packaging.  Please note that significant rearchitecture
> of the upstream code is out of scope for the Debian packaging, so I'm
> going to disregard comments of the form "maybe there should be a module
> loader so all these things can just be plug-ins" or other such blue-sky
> things; from my point of view this is just about configuring things a
> bit more wisely within more or less our current constraints.
>
>
> libsystemd
> ==
>
> This is the obvious thing on everyone's mind right now.  At the time I
> merged that patch, "not NIHing code that's in a perfectly good library"
> seemed like a reasonable trade-off, but we do seem to have ended up on
> the wrong side of history on this one.  There's work in progress to land
> readiness protocol notification upstream without libsystemd (thanks
> Damien and Luca!), and I expect to cherry-pick this into Debian once
> it's agreed, so we'll get rid of that linkage and reduce our patch load
> a bit.
>
> We also have a patch from Ubuntu to support the systemd socket
> activation protocol.  I've rewritten this to avoid using libsystemd, and
> I'll submit it upstream once the readiness notification work is sorted
> out.  But it's not particularly invasive once the libsystemd linkage is
> removed, so it's not the end of the world if this ends up staying in our
> patch queue.
>
>
> GSS-API key exchange
> 
>
> Way back in 2005, I merged the GSS-API key exchange patch into Debian's
> main openssh package (https://bugs.debian.org/275472).  At the time it
> seemed like a sensible overall reduction in maintenance burden (if I
> remember correctly, the openssh-krb5 package often ended up lagging a
> fair bit behind openssh).  While the patch is fairly large, it hasn't
> generally been too hard to forward-port to newer versions of OpenSSH,
> and Fedora carries it too so there's some sharing of work.
>
> However, OpenSSH upstream has long rejected it, mainly on the basis that
> they don't like adding new pre-authentication attack surface, and this
> week seems like a good one to reconsider what patches we're shipping by
> default.  gssapi.patch is the largest patch in our openssh package by an
> order of magnitude, and easily the most intrusive in terms of complexity
> and exposure, so I've somewhat regretted my choice to merge it a few
> times over the years.
>
> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package: I get enough occasional bug
> reports to convince me that it's still in use.  So, if I decide to split
> it back out, I'd want to arrange for a somewhat graceful transition.
> We've had it for nearly 20 years now, so we can take the time to do a
> proper job that at least tries not to leave users in the lurch.
>
> How does this rough plan sound?
>
>  * for Debian trixie (current testing):
>
>* add dependency-only packages called something like
>  openssh-client-gsskex and openssh-server-gsskex, depending on their
>  non-gsskex alternatives
>* add NEWS.Debian entry saying that people need to install these
>  packages if they want to retain GSS-API key exchange support
>* add release note saying the same
>
>  * for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
>exact timings):
>
>* add separate openssh-gsskex source package, carrying gssapi.patch
>  in addition to whatever's in openssh, and whose binary packages
>  Conflicts/Replaces/Provides the corresponding ones from openssh
>* add some kind of regular CI to warn about openssh-gsskex being out
>  of date relative to openssh
>* drop gssapi.patch from openssh, except for small patches to
>  configuration file handling to accept the relevant options with
>  some kind of informative warning (compare
>  https://bugs.debian.org/152657)
>
> I guess we should decide whether the separate packages are to be needed
> for GSS-API authentication as well as key exchange, because that affects
> the choice of dependency-only package names in trixie.  If we only split
> out gssapi.patch (for key exchange; sorry about the slightly misleading
> name) but kept --with-kerberos5 (which also controls authentication),
> then we'd significantly reduce our patch load but not sshd's linkage.
>
> I've seen the suggestion of using libgssglue here
> (https://fosstodon.org/@jas/112194876950058188).  That might be a good
> idea and I have no particular objection to it, though I also don't know
> much about it and it would probably be better if an expert did the work.
> 

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 02, Colin Watson  wrote:

> At the time, denyhosts was popular, but it was removed from Debian
> several years ago.  I remember that, when I dealt with that on my own
> systems, fail2ban seemed like the obvious replacement, and my impression
> is that it's pretty widely used nowadays; it's very pluggable but it
> normally works by adding firewall rules.  Are there any similar popular
> systems left that rely on editing /etc/hosts.deny?
Yes, people. I object to removing TCP wrappers support since the patch 
is tiny and it supports use cases like DNS-based ACLs which cannot be 
supported by L3 firewalls.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 03:27:30AM +0200, Christoph Anton Mitterer wrote:
> Do you think it will be possible to have still only one `ssh`, `scp`,
> etc. command and that will just use extra GSSAPI stuff if installed and
> needed by a certain connection?

It would be technically possible to retain the client parts of the
GSS-API key exchange patch in the default variant.  It would require the
build to be separated into multiple passes, since that patch touches a
number of files shared by the client and the server.

Rather than trying to construct this, though, it would be much simpler
and I think safer to just have a separate openssh-client-gsskeyex
package.  Like today's openssh-client, it would be usable both with and
without GSS-API key exchange.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Howard Chu]

Damien Miller wrote:
> Another thing we're considering in OpenSSH is changing how we integrate
> with PAM. PAM's API demands loading modules into the authenticating
> process' address space, but obviously we've just been reminded that this
> is risky.

This was a long-standing problem with pam/nss-ldap, which we solved by moving 
all of the
actual libldap invocations to a separate nslcd process, and only communicated 
to it
across a unix domain socket via stubs in the pam/nss modules. Mixing instances 
of libraries
that applications call directly, with instances loaded implicitly by 
system-level mechanisms,
was always a bad idea and caused endless problems, even without malicious 
attackers.

> I think that I would prefer to move to a model where there PAM auth and
> account modules run in a helper process, and only the session module
> runs in the unprivileged post-auth sshd process.

We could probably generalize the stub wrapping that we used for nss/pam-ldapd / 
nslcd to
be a generic interface to a standalone pamd that actually loads the pam 
modules. Should
be a simple job.

> This means that PAM auth/account modules and their transitive library
> dependencies cannot affect the sshd address space. They would still
> likely need to run with privilege, could still fail permissively in
> unwanted situations and might still be able to cause problems directly
> (e.g. opening a reverse shell from the PAM module itself), but they
> would no longer have direct access to the contents of sshd network
> traffic, signatures, etc that are extremely useful in building NOBUS
> (https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.
> 
> Where this gets challenging is that some PAM modules make assumptions
> that the auth, account and session modules all run in the same address
> space. These would break until re-architected to pass things explicitly,
> e.g. via environment variables, temp files, etc.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Sirius]

In days of yore (Tue, 02 Apr 2024), Colin Watson thus quoth: 
> TCP wrappers
> 

Not used hosts.{allow,deny} for the last 17 years (since I started my
current employment) so I am biased. Honest opinion is that firewall and
fail2ban have pretty much obsoleted TCP wrappers.

> SELinux
> ===
> 
> For the time being my inclination is to leave this be, but I've seen the
> suggestion that pam_selinux is basically all you need
> (https://infosec.exchange/@alwayscurious/112192949171400643), so maybe
> it would be an option to drop --with-selinux in favour of that?  I've
> never used SELinux, so I'd need an expert to weigh on here.

If you need an expert on SELinux, you need Dan Walsh.

I have used SELinux for the last 17 years, from when it was a monolithic
policy to what it is like today in RHEL. SELinux is - as far as I know -
not an issue and have a fail-close rather than fail-open approach. IMHO,
if it is not used and you have the time to spare to drop it, do, otherwise
it should be safe with the status-quo on this.

And should Debian pick SELinux up fully and enable a targeted policy,
well, you will want this anyway.

-- 
Kind regards,

/S

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Russ Allbery]

Christoph Anton Mitterer  writes:

> Actually I think that most sites where I "need"/use GSSAPI... only
> require the ticket for AFS, and do actually allow pubkey auth (but
> right now, one doesn't have AFS access then).

In past discussions of this patch, this has not been the case.  One of the
advantages of GSSAPI key exchange is that you can disable public keys for
all of your hosts and never manage known hosts, instead only using the
system Kerberos keytabs.  Since in a Kerberos environment you have to put
keytabs on every host *anyway*, and that *is* the host's identity in a
Kerberos environment, this reduces the number of key infrastructures you
have to manage by one, which matters to some Kerberos deployments.  This
arguably gives you better security in that specific environment because
keytabs do not rely on leap-of-faith initial authentication; the server is
always properly authenticated, even on first connect.

> Not sure if there's a simple out of the box way to just transfer that
> but without all the other GSSAPI stuff?

If you want your ticket to refresh remotely when you refresh it locally,
which is often needed for Kerberos applications like AFS, you do need key
exchange, since that's the mechanism that allows that to happen.

(I use both GSSAPI and tcpwrappers, so Colin's proposal would mean more
work for me, but given the situation, I'm willing to rework the way that I
use ssh to avoid both going forward.  More features are nice, but I can
see the merits of simplicity here.  But I no longer maintain a large
infrastructure built on Kerberos, so I'm not putting as much weight on the
GSSAPI support as I used to.)

-- 
Russ Allbery (r...@debian.org)  

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Christoph Anton Mitterer]

Hey.

On Tue, 2024-04-02 at 01:30 +0100, Colin Watson wrote:
> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package: I get enough occasional
> bug
> reports to convince me that it's still in use.

Being one of those people, and having even asked for more patches to be
merged (#1053822) O:-) ... no, there was no social engineering
intended,... your approach sounds like a reasonable way to go.

Actually I think that most sites where I "need"/use GSSAPI... only
require the ticket for AFS, and do actually allow pubkey auth (but
right now, one doesn't have AFS access then).
Not sure if there's a simple out of the box way to just transfer that
but without all the other GSSAPI stuff?

Do you think it will be possible to have still only one `ssh`, `scp`,
etc. command and that will just use extra GSSAPI stuff if installed and
needed by a certain connection?


Cheers,
Chris.

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Damien Miller]

On Tue, 2 Apr 2024, Colin Watson wrote:

[I'm not subscribed to the debian-* lists, please Cc me in replies if
you want me to see them]

> [I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
> just debian-devel and debian-ssh to avoid potentially spamming them
> with a long discussion. If you choose to override this then that's
> your call, but please be mindful of upstream's time.]

Thanks Colin for considering how to reduce dependency chains for sshd.
I just remembered that this is not the first time that sshd has been
attacked via a transitive library dependency - it has happened before,
about 10 years ago:

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

Attacks like these are impossible for sshd to defend against itself.
Instead we have to look to minimising the number of libraries that end
up in sshd's address space, especially that of the privileged sshd
process.

We are currently exploring splitting sshd into separate binaries for
the listener, privileged monitor, pre- and post-auth network-facing
processes so that each can be reduced in size and functionality to
the minimum possible. This should remove a number of dependencies from
the privileged process. There's a draft of these changes at
https://github.com/djmdjm/openssh-wip/pull/26 but it's OpenBSD-only
at this stage. We're likely to proceed with splitting the listener
process from the rest of sshd hopefully before the next release.

Another thing we're considering in OpenSSH is changing how we integrate
with PAM. PAM's API demands loading modules into the authenticating
process' address space, but obviously we've just been reminded that this
is risky.

I think that I would prefer to move to a model where there PAM auth and
account modules run in a helper process, and only the session module
runs in the unprivileged post-auth sshd process.

This means that PAM auth/account modules and their transitive library
dependencies cannot affect the sshd address space. They would still
likely need to run with privilege, could still fail permissively in
unwanted situations and might still be able to cause problems directly
(e.g. opening a reverse shell from the PAM module itself), but they
would no longer have direct access to the contents of sshd network
traffic, signatures, etc that are extremely useful in building NOBUS
(https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.

Where this gets challenging is that some PAM modules make assumptions
that the auth, account and session modules all run in the same address
space. These would break until re-architected to pass things explicitly,
e.g. via environment variables, temp files, etc.

Time permitting, I'll get a prototype of these changes made for wider
experimentation.

-d