Re: Should we be reverting iptables to iptables-legacy for eoan?
On Tue, Sep 10, 2019 at 8:12 PM Jamie Strandboge wrote: > > On Tue, 10 Sep 2019, Julian Andres Klode wrote: > > > Hi folks, > > > > it turns out that lxd is broken by iptables now using the nft > > based stuff, because lxd is still using the legacy one from > > inside the snap. > > > > This provides a terrible experience because networking in lxd > > is not working at all once you enable ufw. > > Is this because ufw uses whatever is on the system (ie, nft default) and > lxd is using whatever is in the snap (ie, historic iptables which with > iptables 1.8 translates to 'iptables-legacy') and this means that both > backends are in use, which results in a mixed view of things? Right, ufw goes through nft, setting up a bunch of rejects, then lxd starts up, loads its own rules which would normally allow its traffic but this doesn't work as the nft ruleset runs in its entirety ahead of running the legacy iptables rules and so a final reject decision was already made. There are in theory some ways you can have both nft and legacy working at the same time, though that ends up being a debugging/management nightmare as there are no tools that query both and give you a unified view of what's going on. > The man pages for iptables-nft/iptables-legacy don't call this out as a > problem, but README.Debian does: > > NOTE: make sure you don't mix iptables-legacy and iptables-nft > rulesets in the same machine at the same time just for sanity and to > avoid unexpected behaviours in your network. > > > I'd suggest we increase the priority of iptables-legacy for eoan, > > so that it is the default, and move the switch to xtables-nft-based > > one to next release. > > > > This will allow us to have working lxd networking, and gives > > the lxd team some breathing room. > > This makes sense to me. Besides, there are still bugs trickling in where > the nft backend isn't acting fully compatibly anyway. However, it would > be nice to use the nft backend by default by 20.04, ie, at the opening > of the cycle. Agreed. We've added an item on the LXD roadmap to deal with nft next cycle. We've had requests coming in from users of other distros already and I've done a fair amount of research around nft ahead of this work, but it will require quite a lot of changes on our side as we are using a fair amount of traditional iptables/ip6tables and then a bunch of very weird ebtables (which is what failed when run with compat wrappers in our case). > That said, this is a very real immediate problem for snaps on 19.10+, > Buster+ and Ubuntu Core (CC'ing Samuele and Michael specifically) since > the only available bases are based on xenial and bionic and snaps can > only stage-packages from xenial and bionic, so snaps that plugs > 'firewall-control' (or classic snaps) will by definition by far tend to > use 'legacy' (unless they build iptables from source). Even though the > system might have the nft backend enabled. Furthermore, the man page for > iptables-nft says that kernel >= 4.17 is required for the nft backend > (but ISTR issues in Debian with > 4.19) and its entirely possible > someone could be running a system with an older kernel with a newer user > space (eg, core20 snap with bionic kernel snap). > > I'm not sure how to fix this. The only thing that seems reasonable is > for snaps to only be able to use the correct one that matches the > host/kernel. To pull that off snapd would need to examine the > capabilities and configuration of the running system then bind mount > iptables/etc from the host into runtime of the snap (the review-tools > could flag if snaps ship these binaries). Alternatively, an iptables > snapcraft part could be created that at runtime can interrogate whether > to use the nft or legacy backends (or snapd exposes which backend is in > use to the snap so it can decide). > > I suspect that Michael and Samuele will want to move the snap-specific > parts of this discussion to the snapcraft forum, but +1 from me to > prefer 'legacy' for the time being in Ubuntu. I opened https://bugs.launchpad.net/ubuntu/+source/iptables/+bug/1843468 to track this on LP. If going with the revert, this should get closed and another bug be opened to track the work to properly switch to nft early next cycle. Stéphane -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Should we be reverting iptables to iptables-legacy for eoan?
On Tue, 10 Sep 2019, Stéphane Graber wrote: > For LXD specifically, we think it would take us about 3 weeks of > engineering work to sort this in a way that can work on all > distributions, properly detecting and supporting: > - no nft present > - nft present but old iptables used > - nft present and used I realize that LXD is an atypical snap, but this is exactly the sort of thing I would hope that snapd could help with so all snaps wouldn't be required to go through the same hoops to avoid breaking the system. The one good thing is that iptables-nft is (supposed to ;) wholly support the iptables-legacy syntax, so there is some hope of making this easier for the average snap developer. Those that want full-on netfilter would need to jump through these hoops of course. For all systems, it probably makes sense to have a small tool that can perform this detection, so admins, application developers and the like can then just focus on what tool to use (ideally upstreamed into iptables itself). -- Jamie Strandboge | http://www.canonical.com signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Should we be reverting iptables to iptables-legacy for eoan?
On Tue, 10 Sep 2019, Jamie Strandboge wrote: > (but ISTR issues in Debian with > 4.19) Sorry, meant < 4.19 (ie, 4.17 and 4.18 istr still had some issues) -- Jamie Strandboge | http://www.canonical.com signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Should we be reverting iptables to iptables-legacy for eoan?
On Tue, 10 Sep 2019, Julian Andres Klode wrote: > Hi folks, > > it turns out that lxd is broken by iptables now using the nft > based stuff, because lxd is still using the legacy one from > inside the snap. > > This provides a terrible experience because networking in lxd > is not working at all once you enable ufw. Is this because ufw uses whatever is on the system (ie, nft default) and lxd is using whatever is in the snap (ie, historic iptables which with iptables 1.8 translates to 'iptables-legacy') and this means that both backends are in use, which results in a mixed view of things? The man pages for iptables-nft/iptables-legacy don't call this out as a problem, but README.Debian does: NOTE: make sure you don't mix iptables-legacy and iptables-nft rulesets in the same machine at the same time just for sanity and to avoid unexpected behaviours in your network. > I'd suggest we increase the priority of iptables-legacy for eoan, > so that it is the default, and move the switch to xtables-nft-based > one to next release. > > This will allow us to have working lxd networking, and gives > the lxd team some breathing room. This makes sense to me. Besides, there are still bugs trickling in where the nft backend isn't acting fully compatibly anyway. However, it would be nice to use the nft backend by default by 20.04, ie, at the opening of the cycle. That said, this is a very real immediate problem for snaps on 19.10+, Buster+ and Ubuntu Core (CC'ing Samuele and Michael specifically) since the only available bases are based on xenial and bionic and snaps can only stage-packages from xenial and bionic, so snaps that plugs 'firewall-control' (or classic snaps) will by definition by far tend to use 'legacy' (unless they build iptables from source). Even though the system might have the nft backend enabled. Furthermore, the man page for iptables-nft says that kernel >= 4.17 is required for the nft backend (but ISTR issues in Debian with > 4.19) and its entirely possible someone could be running a system with an older kernel with a newer user space (eg, core20 snap with bionic kernel snap). I'm not sure how to fix this. The only thing that seems reasonable is for snaps to only be able to use the correct one that matches the host/kernel. To pull that off snapd would need to examine the capabilities and configuration of the running system then bind mount iptables/etc from the host into runtime of the snap (the review-tools could flag if snaps ship these binaries). Alternatively, an iptables snapcraft part could be created that at runtime can interrogate whether to use the nft or legacy backends (or snapd exposes which backend is in use to the snap so it can decide). I suspect that Michael and Samuele will want to move the snap-specific parts of this discussion to the snapcraft forum, but +1 from me to prefer 'legacy' for the time being in Ubuntu. -- Jamie Strandboge | http://www.canonical.com signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Should we be reverting iptables to iptables-legacy for eoan?
Hi, I'm strongly on the revert camp. This change landed in the release pocket after Feature Freeze and effectively causes the majority of software in the Ubuntu archive to rely on compatibility wrappers to function. Those wrappers aren't perfect match for the tools they replace, causing failures or worse, success but leading to non-matching rules in nft. Due to both legacy iptables and nft functioning at the same time, we may also now be in the very confusing situation of having some rules loaded into nft while some older tools load directly into iptables, causing only half the rules to be visible. I agree that nft is the future, but this needs coordination, we need to figure out what packages in main are using iptables/ip6tables/ebtables and make sure that they're either perfectly compatible with the compat wrappers OR have native nft support. And for software that grows direct nft support, they will need to detect whether nft is in use rather than just check if it's supported by the kernel, that's necessary to avoid ending up with rules in both. For LXD specifically, we think it would take us about 3 weeks of engineering work to sort this in a way that can work on all distributions, properly detecting and supporting: - no nft present - nft present but old iptables used - nft present and used The other problem we'll need to look into is that many of the nft tools appear and user guides start with doing a full clear of nft before populating it and using it, so that may cause some headaches due to ordering as LXD needs to add some rules, obviously doesn't want things reset underneath it. nft is actually more flexible than iptables for updating the ruleset, but we need to ensure it's setup properly so we don't end up with surprises. Stéphane On Tue, Sep 10, 2019 at 5:12 PM Julian Andres Klode wrote: > > Hi folks, > > it turns out that lxd is broken by iptables now using the nft > based stuff, because lxd is still using the legacy one from > inside the snap. > > This provides a terrible experience because networking in lxd > is not working at all once you enable ufw. > > I'd suggest we increase the priority of iptables-legacy for eoan, > so that it is the default, and move the switch to xtables-nft-based > one to next release. > > This will allow us to have working lxd networking, and gives > the lxd team some breathing room. > > -- > debian developer - deb.li/jak | jak-linux.org - free software dev > ubuntu core developer i speak de, en > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Stéphane -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Should we be reverting iptables to iptables-legacy for eoan?
Hi folks, it turns out that lxd is broken by iptables now using the nft based stuff, because lxd is still using the legacy one from inside the snap. This provides a terrible experience because networking in lxd is not working at all once you enable ufw. I'd suggest we increase the priority of iptables-legacy for eoan, so that it is the default, and move the switch to xtables-nft-based one to next release. This will allow us to have working lxd networking, and gives the lxd team some breathing room. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
