Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
Hello, Thanks for testing this, also for the long 13h test with disabled iscsiuio. That gives me plenty of confidence that we can just kill it in local-bottom and not care about it until the system boots up - which makes our life _so_ much easier. On 01/04/2017 06:53 PM, Andrew Patterson wrote: > On 01/03/2017 03:47 PM, Christian Seiler wrote: >> That said: your initial tests are sufficient for me to upload >> a version of open-iscsi that includes support for this to >> the 'experimental' suite, so I'll do that either tomorrow or >> on Thursday, depending on when I'll find time for that. If you >> could then test this (and write down how you set this up so >> we can add it to README.Debian), and give me feedback on the >> "iscsiuio is offline for longer" test, then I'd be fine with >> uploading this to unstable so it makes it into Stretch. >> > > I would be happy to test it. I've uploaded a version 2.0.874-2~exp1 to experimental that contains these changes. I've tested this on a VM with rootfs on iSCSI (no offloading) and that still works as before. This should hit experimental soon and I expect that the package will be available in about 2 - 3 hours after the next dinstall run. (Later on some mirrors maybe.) The changes seem mostly harmless to me, so I'm very confident that this won't break existing use cases at the very least, but an independent review and some testing would be very much appreciated before I upload this to unstable. Also, if this really does work, could you write a paragraph or so for README.Debian on how to use iscsiuio offloading in the initramfs? Many thanks! Regards, Christian PS: It's much too late in the Stretch release cycle for that, but at the beginning of the Buster release cycle I'd like to get support for ISCSI_AUTO into the Debian installer, as that is likely to be the most common way that rootfs on iSCSI is going to be set up. Plus maybe support iscsiuio in the installer. With both of these changes, offloading should then work out of the box.
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
On 01/03/2017 03:47 PM, Christian Seiler wrote: > On 01/03/2017 10:51 PM, Andrew Patterson wrote: >> On 01/03/2017 01:27 PM, Christian Seiler wrote: >>> On 01/03/2017 09:10 PM, Andrew Patterson wrote: > How does the rest of the boot process proceed then? What happens > when iscsiuio is to be started regularly at boot from the systemd > service / init script? > Is the iscsiuio from the initramfs required > to be running at all times during the iSCSI session or can it be > restarted, as long as the time during which that happens is brief > enough? Is it required to be kept around during shutdown if it's > on iSCSI? I don't believe it needs to be running to support traffic, just when doing logins. I will test to confirm. >>> >>> That would be ideal. :) >> >> I tried running a load with a data integrity check while starting and >> stopping iscsiuio numerous times. I did not see any issues either >> with performance or data integrity. > > Did you try stopping iscsiuio for longer periods of time? For > example more than 20 minutes? I would like to know what happens > if you hit ARP timeouts. I ran it overnight (13 hours) with no issues. > >>> As far as I can tell from skimming the source iscsiuio is also >>> responsible for doing ARP and responding to ICMP - but even >>> then it wouldn't be a problem for restarting in that case, as >>> long as the MAC address of the target (or a router on the way >>> to the target) doesn't change during restart. >>> >> >> It this likely to be an issue in the one or two seconds between >> transitioning between the initramfs and sysvinit/systemd? > > It's only a couple of seconds if there's no fsck running for > another partition. Then it could easily be minutes or even > longer, depending on the partition size, the number of files > and the filesystem in question. > Granted. > Does it follow systemd's process name convention to make > sure systemd doesn't kill it during shutdown? >> >> I don't see any special handling of iscsiuio in systemd shutdown. > > I meant this: > > https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/ > (The @-stuff is only relevant if the binary is _not_ started > via a systemd service.) This only applies for systemd though. It won't help us for sysv-init (assuming we need to care). > > OTOH, since iscsiuio is apparently not required to be running > while data is transferred, killing it at shutdown deosn't > seem to be that big of a deal. It seems so. > >> It looks like we can use start-stop-daemon in local-top and >> local-bottom to gracefully start and stop the daemon. > > Yes, that appears to be the case. > > Before I add the aforementioned steps 1 & 2 I'd like to be sure that > this actually works properly beyond the initramfs. So any > information related to that would be appreciated. >> >> Is my current testing enough or do we need to do more? If the later, >> what sort of test would you like me to run? > > See my request above: if after 20 minutes of iscsiuio being > offline everything is still fine, I'd be fine with it. If it's > less, it depends on how much less. It looks like we are good. At least in the normal case. > > That said: your initial tests are sufficient for me to upload > a version of open-iscsi that includes support for this to > the 'experimental' suite, so I'll do that either tomorrow or > on Thursday, depending on when I'll find time for that. If you > could then test this (and write down how you set this up so > we can add it to README.Debian), and give me feedback on the > "iscsiuio is offline for longer" test, then I'd be fine with > uploading this to unstable so it makes it into Stretch. > I would be happy to test it. > Other than #850057: does iscsistart need additional changes to > work with iscsiuio? One would probably need to pass > -P iface.transport_name=bnx2i -P iface.net_ifacename=eth0 > -P iface.hwaddress=aa:bb:cc:dd:ee:ff or similar, right? > I have been using iscsstart -b with iSCSI IBFT to bring up the interface which seems to do all the required magic (at least for the iscsi_auto case). If not using iscsi_auto, the local_top script wants iSCSI login parameters on the kernel command-line. Don't forget to define OFFLOAD_BOOT_SUPPORTED, ala: diff --git a/debian/rules b/debian/rules index 906132a..562ef8d 100755 --- a/debian/rules +++ b/debian/rules @@ -3,7 +3,7 @@ #export DH_VERBOSE=1 export DEB_BUILD_MAINT_OPTIONS = hardening=+all -export DEB_CFLAGS_MAINT_APPEND = -Wall +export DEB_CFLAGS_MAINT_APPEND = -Wall -DOFFLOAD_BOOT_SUPPORTED DPKG_EXPORT_BUILDFLAGS = 1 include /usr/share/dpkg/default.mk -- Andrew Patterson Hewlett-Packard Enterprise
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
On 01/03/2017 10:51 PM, Andrew Patterson wrote: > On 01/03/2017 01:27 PM, Christian Seiler wrote: >> On 01/03/2017 09:10 PM, Andrew Patterson wrote: How does the rest of the boot process proceed then? What happens when iscsiuio is to be started regularly at boot from the systemd service / init script? Is the iscsiuio from the initramfs required to be running at all times during the iSCSI session or can it be restarted, as long as the time during which that happens is brief enough? Is it required to be kept around during shutdown if it's on iSCSI? >>> >>> I don't believe it needs to be running to support traffic, just when >>> doing logins. I will test to confirm. >> >> That would be ideal. :) > > I tried running a load with a data integrity check while starting and > stopping iscsiuio numerous times. I did not see any issues either > with performance or data integrity. Did you try stopping iscsiuio for longer periods of time? For example more than 20 minutes? I would like to know what happens if you hit ARP timeouts. >> As far as I can tell from skimming the source iscsiuio is also >> responsible for doing ARP and responding to ICMP - but even >> then it wouldn't be a problem for restarting in that case, as >> long as the MAC address of the target (or a router on the way >> to the target) doesn't change during restart. >> > > It this likely to be an issue in the one or two seconds between > transitioning between the initramfs and sysvinit/systemd? It's only a couple of seconds if there's no fsck running for another partition. Then it could easily be minutes or even longer, depending on the partition size, the number of files and the filesystem in question. Does it follow systemd's process name convention to make sure systemd doesn't kill it during shutdown? > > I don't see any special handling of iscsiuio in systemd shutdown. I meant this: https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/ (The @-stuff is only relevant if the binary is _not_ started via a systemd service.) OTOH, since iscsiuio is apparently not required to be running while data is transferred, killing it at shutdown deosn't seem to be that big of a deal. > It looks like we can use start-stop-daemon in local-top and > local-bottom to gracefully start and stop the daemon. Yes, that appears to be the case. Before I add the aforementioned steps 1 & 2 I'd like to be sure that this actually works properly beyond the initramfs. So any information related to that would be appreciated. >>> > > Is my current testing enough or do we need to do more? If the later, > what sort of test would you like me to run? See my request above: if after 20 minutes of iscsiuio being offline everything is still fine, I'd be fine with it. If it's less, it depends on how much less. That said: your initial tests are sufficient for me to upload a version of open-iscsi that includes support for this to the 'experimental' suite, so I'll do that either tomorrow or on Thursday, depending on when I'll find time for that. If you could then test this (and write down how you set this up so we can add it to README.Debian), and give me feedback on the "iscsiuio is offline for longer" test, then I'd be fine with uploading this to unstable so it makes it into Stretch. Other than #850057: does iscsistart need additional changes to work with iscsiuio? One would probably need to pass -P iface.transport_name=bnx2i -P iface.net_ifacename=eth0 -P iface.hwaddress=aa:bb:cc:dd:ee:ff or similar, right? Regards, Christian
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
On 01/03/2017 01:27 PM, Christian Seiler wrote: > On 01/03/2017 09:10 PM, Andrew Patterson wrote: >>> How does the rest of the boot process proceed then? What happens >>> when iscsiuio is to be started regularly at boot from the systemd >>> service / init script? >>> Is the iscsiuio from the initramfs required >>> to be running at all times during the iSCSI session or can it be >>> restarted, as long as the time during which that happens is brief >>> enough? Is it required to be kept around during shutdown if it's >>> on iSCSI? >> >> I don't believe it needs to be running to support traffic, just when >> doing logins. I will test to confirm. > > That would be ideal. :) I tried running a load with a data integrity check while starting and stopping iscsiuio numerous times. I did not see any issues either with performance or data integrity. > > As far as I can tell from skimming the source iscsiuio is also > responsible for doing ARP and responding to ICMP - but even > then it wouldn't be a problem for restarting in that case, as > long as the MAC address of the target (or a router on the way > to the target) doesn't change during restart. > It this likely to be an issue in the one or two seconds between transitioning between the initramfs and sysvinit/systemd? >>> Does it follow systemd's process name convention to make >>> sure systemd doesn't kill it during shutdown? >>> I don't see any special handling of iscsiuio in systemd shutdown. Here is the service file: [Unit] Description=iSCSI userspace offloading daemon (iscsiuio) Documentation=man:iscsiuio(8) Documentation=file:///usr/share/doc/iscsiuio/README.gz Wants=network.target Before=iscsid.service After=network.target DefaultDependencies=no Conflicts=shutdown.target Before=shutdown.target [Service] Type=forking PIDFile=/run/iscsiuio.pid ExecStart=/sbin/iscsiuio [Install] WantedBy=sysinit.target >> >> I am looking into this. One thought is to start the daemon, do the >> login, and then kill it. Then the normal sysvinit/upstart/systemd >> scripts can restart it normally. > > If it's really just for logins: yes, we can do that. If it also > does ARP and such, I'd rather only kill it as close to the restart > as possible. But having the init script / systemd service (there > was never a native upstart job for it) kill the initramfs one > right before starting the own binary is easy, so if that's required > then that would also be fine. It looks like we can use start-stop-daemon in local-top and local-bottom to gracefully start and stop the daemon. > > It's going to be a lot trickier if we can't restart it at all. > >>> Before I add the aforementioned steps 1 & 2 I'd like to be sure that >>> this actually works properly beyond the initramfs. So any >>> information related to that would be appreciated. >> Is my current testing enough or do we need to do more? If the later, what sort of test would you like me to run? >> I am going to look at other package code to see how they handle daemon >> startup. NFS comes to mind. > > NFS doesn't actually need a daemon running after the initial NFS > login, at least temporarily: > NFS for the initramfs is only used in client mode, so this was a red herring. > - NFSv3 needs the to have portmap running on the client side, so >that locking works. However, you can spoof that, which is what >klibc does: > > > http://sources.debian.net/src/klibc/2.0.4-9/usr/kinit/nfsmount/README.locking/ > >Basically you don't need the portmapper to be running at all >until and actual file lock is taken. > > - NFSv4 doesn't need anything running on the client side to mount >with sec=sys with raw UIDs. For idmapping you can use the >request-key / nfsidmap kernel upcalls instead of idmapd (which >is deprecated on the client side), so nothing needs to be >running there either. > > - NFSv4 + Kerberos needs rpc.gssd running, but I've never seen >root on Kerberized NFSv4 even being advertised. (You'd also >need to have credentials with which you can get a Kerberos >ticket in the initramfs + the tools to get the ticket for >that to work.) > > Regards, > Christian > -- Andrew Patterson Hewlett-Packard Enterprise
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
On 01/03/2017 09:10 PM, Andrew Patterson wrote: >> How does the rest of the boot process proceed then? What happens >> when iscsiuio is to be started regularly at boot from the systemd >> service / init script? >> Is the iscsiuio from the initramfs required >> to be running at all times during the iSCSI session or can it be >> restarted, as long as the time during which that happens is brief >> enough? Is it required to be kept around during shutdown if it's >> on iSCSI? > > I don't believe it needs to be running to support traffic, just when > doing logins. I will test to confirm. That would be ideal. :) As far as I can tell from skimming the source iscsiuio is also responsible for doing ARP and responding to ICMP - but even then it wouldn't be a problem for restarting in that case, as long as the MAC address of the target (or a router on the way to the target) doesn't change during restart. >> Does it follow systemd's process name convention to make >> sure systemd doesn't kill it during shutdown? >> > > I am looking into this. One thought is to start the daemon, do the > login, and then kill it. Then the normal sysvinit/upstart/systemd > scripts can restart it normally. If it's really just for logins: yes, we can do that. If it also does ARP and such, I'd rather only kill it as close to the restart as possible. But having the init script / systemd service (there was never a native upstart job for it) kill the initramfs one right before starting the own binary is easy, so if that's required then that would also be fine. It's going to be a lot trickier if we can't restart it at all. >> Before I add the aforementioned steps 1 & 2 I'd like to be sure that >> this actually works properly beyond the initramfs. So any >> information related to that would be appreciated. > > I am going to look at other package code to see how they handle daemon > startup. NFS comes to mind. NFS doesn't actually need a daemon running after the initial NFS login, at least temporarily: - NFSv3 needs the to have portmap running on the client side, so that locking works. However, you can spoof that, which is what klibc does: http://sources.debian.net/src/klibc/2.0.4-9/usr/kinit/nfsmount/README.locking/ Basically you don't need the portmapper to be running at all until and actual file lock is taken. - NFSv4 doesn't need anything running on the client side to mount with sec=sys with raw UIDs. For idmapping you can use the request-key / nfsidmap kernel upcalls instead of idmapd (which is deprecated on the client side), so nothing needs to be running there either. - NFSv4 + Kerberos needs rpc.gssd running, but I've never seen root on Kerberized NFSv4 even being advertised. (You'd also need to have credentials with which you can get a Kerberos ticket in the initramfs + the tools to get the ticket for that to work.) Regards, Christian
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
On 01/03/2017 01:02 PM, Christian Seiler wrote: > Control: tags -1 + confirmed moreinfo > Control: severity -1 wishlist > > On 01/03/2017 06:02 PM, Andrew Patterson wrote: >> Version 2.0.874 of the open-iscsi package requires that the iscsiuio >> daemon to be running before iscsi hardware offload cards can login to >> a LUN. The initramfs scripts used by open-iscsi to support boot from >> iSCSI do no start up this daemon resulting in a failed boot. > > I'd really like to support this setup. My idea how this could work > would be along the lines of the following: > > 1. In the local-top hook start iscsiuio before calling iscsistart -b > or iscsistart $params if the binary is present in the initramfs. > > 2. In the iscsiuio package install a initramfs hook that copies the > iscsiuio binary (luckily it doesn't depend on any library) into > the initramfs. > > Together that should start iscsiuio in the initramfs and allow > login to the targets there - but if iscsiuio is not installed, the > user won't see a difference from today. > That was my plan. > However, there are still some questions I'd like to have answered > before proceeding - and since I don't have the corresponding hardware > I have no idea what the right answers are: > > How does the rest of the boot process proceed then? What happens > when iscsiuio is to be started regularly at boot from the systemd > service / init script? Is the iscsiuio from the initramfs required > to be running at all times during the iSCSI session or can it be > restarted, as long as the time during which that happens is brief > enough? Is it required to be kept around during shutdown if it's > on iSCSI? I don't believe it needs to be running to support traffic, just when doing logins. I will test to confirm. Does it follow systemd's process name convention to make > sure systemd doesn't kill it during shutdown? > I am looking into this. One thought is to start the daemon, do the login, and then kill it. Then the normal sysvinit/upstart/systemd scripts can restart it normally. > Before I add the aforementioned steps 1 & 2 I'd like to be sure that > this actually works properly beyond the initramfs. So any > information related to that would be appreciated. I am going to look at other package code to see how they handle daemon startup. NFS comes to mind. > > Regards, > Christian > -- Andrew Patterson Hewlett-Packard Enterprise
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
Control: tags -1 + confirmed moreinfo Control: severity -1 wishlist On 01/03/2017 06:02 PM, Andrew Patterson wrote: > Version 2.0.874 of the open-iscsi package requires that the iscsiuio > daemon to be running before iscsi hardware offload cards can login to > a LUN. The initramfs scripts used by open-iscsi to support boot from > iSCSI do no start up this daemon resulting in a failed boot. I'd really like to support this setup. My idea how this could work would be along the lines of the following: 1. In the local-top hook start iscsiuio before calling iscsistart -b or iscsistart $params if the binary is present in the initramfs. 2. In the iscsiuio package install a initramfs hook that copies the iscsiuio binary (luckily it doesn't depend on any library) into the initramfs. Together that should start iscsiuio in the initramfs and allow login to the targets there - but if iscsiuio is not installed, the user won't see a difference from today. However, there are still some questions I'd like to have answered before proceeding - and since I don't have the corresponding hardware I have no idea what the right answers are: How does the rest of the boot process proceed then? What happens when iscsiuio is to be started regularly at boot from the systemd service / init script? Is the iscsiuio from the initramfs required to be running at all times during the iSCSI session or can it be restarted, as long as the time during which that happens is brief enough? Is it required to be kept around during shutdown if it's on iSCSI? Does it follow systemd's process name convention to make sure systemd doesn't kill it during shutdown? Before I add the aforementioned steps 1 & 2 I'd like to be sure that this actually works properly beyond the initramfs. So any information related to that would be appreciated. Regards, Christian
Bug#850060: open-iscsi initramfs scripts do not start up iscsiuio deamon
Package: open-iscsi Version: 2.0.874-1 Version 2.0.874 of the open-iscsi package requires that the iscsiuio daemon to be running before iscsi hardware offload cards can login to a LUN. The initramfs scripts used by open-iscsi to support boot from iSCSI do no start up this daemon resulting in a failed boot. Some discussion for this issue can be found at: https://groups.google.com/forum/?_escaped_fragment_=topic/open-iscsi/PsT65Z4Gx3I#!topic/open-iscsi/PsT65Z4Gx3I -- Andrew Patterson Hewlett-Packard Enterprise
