Hi Colin, Thanks a lot for taking the time to provide very valuable information. It's helping me a lot.
On 2/17/21 12:11 PM, Colin Watson wrote: > On Wed, Feb 17, 2021 at 11:46:57AM +0100, Thomas Goirand wrote: >> On 2/17/21 10:14 AM, Colin Watson wrote: >>> Are you using ListenAddress in sshd_config? >> >> Yes, with the same IP as above, in order to make sure ssh isn't >> listening on a public IP (which would be a security concern for us). > > Oh, that's vital information for this bug I'm surprised ... > using ListenAddress changes > the constraints on sshd startup, somewhat as described in README.Debian. I've read it, and the only part of the README.Debian that talks about something related, is about the removal of the if-up hook. I don't see any startup constraint changes described there. What did I miss? The launchpad bug entry about the ifupdown hook removal is specifically discussing the fact that in my case, I'd be affected. Indeed, I am. I then wonder what's advised then... > In that case I think this is at least arguably a case of needing to keep > your configuration in sync, isn't it? You've made a change to > sshd_config, so you need to change other parts of the system to support > that change. I'm not convinced that using a custom ListenAddress implies repairing the boot process, no. By default, sshd_config has this: #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: Having these commented out is an invitation to un-comment and use them with custom values. Basically, what you wrote above is that doing this breaks sshd. Hopefully, you'll agree that this isn't what one would expect! :) If that's really the case and one should do either the systemd ordering hack I'm doing, or the net.ipv4.ip_nonlocal_bind sysctl tweak, then IMO it'd be worse either: 1/ removing ListenAddress from the example sshd_config 2/ adding comments just above the directive, explaining what we're discussing in this bug entry. >> Maybe setting-up net.ipv4.ip_nonlocal_bind=1 (in sysctl.conf) would fix >> my issue, no? > > That's the system-wide version of IP_FREEBIND. OpenSSH upstream seems > to have decided not to support IP_FREEBIND > (https://bugzilla.mindrot.org/show_bug.cgi?id=2512) If I understand well what's in this bug entry, upstream seems to suggest to do what I did: 1/ ListenAddress 2/ Add an override so that sshd starts After=network-online.target However, it's looking like you're saying one shouldn't do 2/ at all? Could you explain why? I've been using After=network-online.target in most daemons I maintain, and now I'm wondering if that's wrong... Then if I shouldn't do After=network-online.target, do you believe that the sysctl hack is better? > but the sysctl should work if you're OK with it being system-wide. I'd very much prefer if this could be a per-socket thing, but I already do this because that's how I setup haproxy that binds on a VIP (which can move from one host to another). Though now, I wonder if there's an option in HAProxy so it could use IP_FREEBIND on its own. Which would then lead me to say I would prefer to not use a system-wide thing... > I'd also recommend at least considering other approaches to implementing > your security policy that avoid the ordering complexities of > ListenAddress, since there are other ways to prevent incoming > connections on public IP addresses. Approaches I can think of include: > > * Reject connections to port 22 at the firewall level (perhaps a > firewall on the local host). Considering the interaction with OpenStack Neutron, this could potentially be hard to maintain: Neutron is doing a lot of iptables stuff on its own, and I prefer if I don't touch that at all, either on compute nodes or network nodes. So the option to use ListenAddress looked a way nicer for my use case. > * It might be worth experimenting with Match LocalAddress in > sshd_config. I haven't played with that much myself, and it's > poorly-documented, but I *think* that might allow you to reject any > connections that aren't directed to appropriate addresses. >From my experience, it's best to not expose the ssh port at all if possible, as brute-forcing the port may lead brokenness. On 2/17/21 9:42 PM, Chris Hofstaedtler wrote: > But if you do this, you'll end up delaying start of sshd for up to > 120seconds in error cases. And even then, you might not get what you > want (if you read systemd-networkd-wait-online.service(8) > carefully). This talks about networkd. Unless things have changed, I don't think Debian is using this by default (yet?). > Services that use After=network-online.target are generally broken, > please do not introduce that. Can you explain why? > As discussed already, IP_FREEBIND is a thing. As per the bug entry Collin pointed out, it looks like it isn't a thing in sshd at least... > The system-wide sysctl > is a common workaround, especially for "bgp-on-the-host" setups, for > all sorts of servers/daemons. If it can be avoided, it'd be best, IMO... Cheers, Thomas Goirand (zigo)
