Re: auto-mount NFS shares on boot
Hi, On 07/16/2015 09:04 AM, n n wrote: > trying to mount nfs-shares at boot I have exactly the problem mentioned by > Christian: > (in Message-id: <558e8105.5030...@iwakd.de>) > >> - On some systems with static IP addresses (and >> /etc/network/interfaces), I had the problem that even though the >> interface was conisdered up and ready by the kernel, the switch it >> was connected to needed 30s or so to realize that fully (and >> packets were simply dropped beforehand). Since those systems also >> needed to mount NFS... > > Christian then recommends applying a unit he calls wait-for-nfs-server. > > My problem now is that if I apply that unit (or do other tricks like a sleep > in > /etc/network/interfaces the mounts are there after booting, but I have errors > like that in journalctl: > > Jul 14 16:07:02 hhh rpcbind[657]: Starting rpcbind daemon > Jul 14 16:07:02 hhh rpc.statd[681]: Version 1.2.8 starting > Jul 14 16:07:02 hhh rpc.statd[681]: Flags: TI-RPC > Jul 14 16:07:02 hhh rpc.statd[682]: Version 1.2.8 starting > Jul 14 16:07:02 hhh rpc.statd[682]: Flags: TI-RPC > Jul 14 16:07:02 hhh rpc.statd[683]: Version 1.2.8 starting > Jul 14 16:07:02 hhh sm-notify[684]: Version 1.2.8 starting > [...] > So, rpc.statd is there, but why with the --no-notify option? > > To me it looks like rpc.statd gets started multiple times (why?) and > that confuses me and the whole nfs mount process. Ok, so in your case you're also running into the same bug [1] of this very same thread, namely that remote filesystems are not ordered relative to nfs-common. In your case it's rpc.statd and not rpc.gssd, but the cause is the same. In your case, instead of mounts just failing, the mount.nfs binary has a weird logic in it that for NFSv2 and NFSv3 mounts, if statd isn't already running, it will call /usr/sbin/start-statd. That's a script (you can look at it) and it will start rpc.statd multiple times - with the --no-notify option. (Because statd notifications should only happen at boot, but mount.nfs isn't just called at boot.) The problem you have here is the following: - nfs-common tries to start statd regularly (without any options unless you specified them explicitly via /etc/default/nfs-common) - at the same time you apparently have multiple (!) nfs mounts for which mount.nfs is started - each of those mount.nfs processes tries to run statd with the --no-notify option - then they start getting it each other's way, at the end one wins all or all but one mounts (depending on which wins) will just fail The start-statd logic of mount.nfs is something that doesn't jibe well with systemd, since it was designed back when nfs mounts were not done in parallel but rather one after the other, so that even if it were executed at boot, it wouldn't have caused this issue, because boot was much more serialized. (Also, starting statd in systemd as part of a mount unit is not a good idea.) To solve your problem: Just apply the same workaround until it's fixed in Jessie: make sure that NFS mounts only happen after nfs-common is done, then mount.nfs will not try to start statd itself (because it's already running) and boot should work out. For this, create a directory /etc/systemd/system/nfs-common.service.d and then create a file /etc/systemd/system/nfs-common.service.d/remote-fs-pre.conf with the following contents: [Unit] Before=remote-fs-pre.target Then it should work. Regards, Christian [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775542 signature.asc Description: OpenPGP digital signature
Re: auto-mount NFS shares on boot
>>> - On some systems with static IP addresses (and >>> /etc/network/interfaces), I had the problem that even though the >>> interface was conisdered up and ready by the kernel, the switch it >>> was connected to needed 30s or so to realize that fully (and >>> packets were simply dropped beforehand). Since those systems also >>> needed to mount NFS... > >Then you need to talk to your networking guy and tell him to switch STP >off on that port (or switch to RSTP). > >Grüße, >Sven. Hi Sven, thank you for your promt answer. But what you are suggesting is, that I should get rid of the 30s delay, I guess. But my question is, what to do If I have (for whatever reason) such a delay. Even more precise what is going wrong (or is it realy wrong?) if I handle the delay in the way I mentioned. All the best, Hank -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/blu170-w138bd79d23318cf9ac8f554cd...@phx.gbl
Re: auto-mount NFS shares on boot
n n wrote: > trying to mount nfs-shares at boot I have exactly the problem mentioned by > Christian: > (in Message-id: <558e8105.5030...@iwakd.de>) >> - On some systems with static IP addresses (and >> /etc/network/interfaces), I had the problem that even though the >> interface was conisdered up and ready by the kernel, the switch it >> was connected to needed 30s or so to realize that fully (and >> packets were simply dropped beforehand). Since those systems also >> needed to mount NFS... Then you need to talk to your networking guy and tell him to switch STP off on that port (or switch to RSTP). Grüße, Sven. -- Sigmentation fault. Core dumped. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/3bpaejtoo...@mids.svenhartge.de
Re: Re: auto-mount NFS shares on boot
Hi there, trying to mount nfs-shares at boot I have exactly the problem mentioned by Christian: (in Message-id: <558e8105.5030...@iwakd.de>) > - On some systems with static IP addresses (and > /etc/network/interfaces), I had the problem that even though the > interface was conisdered up and ready by the kernel, the switch it > was connected to needed 30s or so to realize that fully (and > packets were simply dropped beforehand). Since those systems also > needed to mount NFS... Christian then recommends applying a unit he calls wait-for-nfs-server. My problem now is that if I apply that unit (or do other tricks like a sleep in /etc/network/interfaces the mounts are there after booting, but I have errors like that in journalctl: Jul 14 16:07:02 hhh rpcbind[657]: Starting rpcbind daemon Jul 14 16:07:02 hhh rpc.statd[681]: Version 1.2.8 starting Jul 14 16:07:02 hhh rpc.statd[681]: Flags: TI-RPC Jul 14 16:07:02 hhh rpc.statd[682]: Version 1.2.8 starting Jul 14 16:07:02 hhh rpc.statd[682]: Flags: TI-RPC Jul 14 16:07:02 hhh rpc.statd[683]: Version 1.2.8 starting Jul 14 16:07:02 hhh sm-notify[684]: Version 1.2.8 starting Jul 14 16:07:02 hhh rpc.statd[683]: failed to create RPC listeners, exiting Jul 14 16:07:02 hhh rpc.statd[682]: failed to create RPC listeners, exiting Jul 14 16:07:02 hhh nfs-common[673]: Starting NFS common utilities: statd failed! Jul 14 16:07:02 hhh systemd[1]: nfs-common.service: control process exited, code=exited status=1 Jul 14 16:07:02 hhh systemd[1]: Failed to start LSB: NFS support files common to client and server. Jul 14 16:07:02 hhh systemd[1]: Unit nfs-common.service entered failed state. Jul 14 16:07:02 hhh kernel: FS-Cache: Loaded Jul 14 16:07:03 hhh kernel: RPC: Registered named UNIX socket transport module. Jul 14 16:07:03 hhh kernel: RPC: Registered udp transport module. Jul 14 16:07:03 hhh kernel: RPC: Registered tcp transport module. Jul 14 16:07:03 hhh kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Jul 14 16:07:03 hhh kernel: FS-Cache: Netfs 'nfs' registered for caching Why the flag and what does it mean? rpc.statd[681]: Flags: TI-RPC And the other errors: rpc.statd[683]: failed to create RPC listeners, exiting nfs-common[673]: Starting NFS common utilities: statd failed! systemd[1]: nfs-common.service: control process exited, code=exited status=1 systemd[1]: Failed to start LSB: NFS support files common to client and server. systemd[1]: Unit nfs-common.service entered failed state. and if I do a root@hhh:~# ps aux | grep rpc root 669 0.1 0.0 37168 2904 ?Ss Jul14 1:42 /sbin/rpcbind -w statd 681 0.0 0.0 37268 2876 ?Ss Jul14 0:00 rpc.statd --no-notify root 692 0.0 0.0 0 0 ?S< Jul14 0:00 [rpciod] root 907 0.0 0.0 10712 1460 ?SJul14 0:00 /usr/sbin/rpc.yppasswdd -D /var/yp/ypfiles -e chsh root 910 0.0 0.0 8520 1612 ?SJul14 0:00 /usr/sbin/rpc.ypxfrd root 2758 0.0 0.0 14192 2284 pts/0S+ 15:42 0:00 grep rpc So, rpc.statd is there, but why with the --no-notify option? To me it looks like rpc.statd gets started multiple times (why?) and that confuses me and the whole nfs mount process. Now the question is: Is that normal or should I worry about it? Do you (Christian) also get such errors in journalctl? -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/blu170-w2165db8abb61d8aacac025cd...@phx.gbl
Re: auto-mount NFS shares on boot
[ bringing pkg-systemd-maintainers into the loop here ] Am 27.06.2015 um 20:14 schrieb Christian Seiler: > On 06/27/2015 08:02 PM, Jonas Meurer wrote: >> Am 27.06.2015 um 16:07 schrieb Christian Seiler: >>> Could you try to do the following: >>> >>> 1. create a directory /etc/systemd/system/remote-fs-pre.target.d >>> 2. create a file /etc/systemd/system/remote-fs-pre.target.d/nfs.conf >>>with the following contents: >>> >>> [Unit] >>> After=nfs-common.service >>> >>> And then reboot your system? I would bet it should work then. >> >> Perfect, that solution works like a charm. nfs-common is started before >> remote-fs, thus rpc.gssd runs already when the NFS share is mounted. > > Great, glad I could solve your problem. :) > >> I suggest to add this simple fix to Jessie by uploading it to >> stable-proposed-updates. What do you think? Also, do you think that >> /etc/systemd/system/remote-fs-pre.target.d/nfs.conf belongs to systemd >> package or to nfs-common? I would say it belongs to nfs-common as that >> one provides the required tools and services to mount NFS shares on a >> client. > > So the fix I gave you is a fix that shouldn't be copied verbatim into > a Debian package. /etc/systemd is administrator territory, /lib/systemd > is package territory, so any proper fix of a package should go to /lib. > > Furthermore, my suggestion for your problem was to add a drop-in that > augments the Debian-provided unit files (see man systemd.unit for how > drop-ins work) - which is great since you only want to extend the unit, > not completely replace it. > > But in the packages themselves it is perfectly possible to modify the > unit files themselves, so a drop-in is not necessary at all for any > native unit. > > So for Stretch: > > - assuming that nfs-common will have a native unit by then, the >proper fix would be to simply add Before=remote-fs-pre.target >to that unit, and that would fix that > > For Jessie: > > - nfs-common is still an init script, so one cannot simply add >Before=remote-fs-pre.target to that. But there are two other >options: > > - just for Jessie: update systemd to change the original unit file >remote-fs-pre.target to include After=nfs-common.service > > - or alternatively, package a drop-in in /lib in the nfs-common >package, i.e. >/lib/systemd/system/nfs-common.service.d/systemd-ordering.conf: >[Unit] >Before=remote-fs-pre.target > > (IMHO at least, I'll defer to the maintainers of the respective > packages as to what they think is appropriate.) Certainly, the preferred fix is, that packages ship native service files which override/replace the sysv init scripts. In case of nfs-common/rpcbind, Ubuntu has done some extensive work to properly integrate nfs-common/rpcbind with systemd. This hasn't landed in Debian (yet) and is not something which can be backported for jessie. The drop-in snippet for nfs-common to augment the dependency information when being run under systemd is something which seems to be suitable for jessie and could be added to the package in sid to give it some testing first. Ideally, that drop-in is shipped by the package itself. This would mean a stable upload for nfs-common. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Re: auto-mount NFS shares on boot
On 06/27/2015 08:02 PM, Jonas Meurer wrote: > Am 27.06.2015 um 16:07 schrieb Christian Seiler: >> Could you try to do the following: >> >> 1. create a directory /etc/systemd/system/remote-fs-pre.target.d >> 2. create a file /etc/systemd/system/remote-fs-pre.target.d/nfs.conf >>with the following contents: >> >> [Unit] >> After=nfs-common.service >> >> And then reboot your system? I would bet it should work then. > > Perfect, that solution works like a charm. nfs-common is started before > remote-fs, thus rpc.gssd runs already when the NFS share is mounted. Great, glad I could solve your problem. :) > I suggest to add this simple fix to Jessie by uploading it to > stable-proposed-updates. What do you think? Also, do you think that > /etc/systemd/system/remote-fs-pre.target.d/nfs.conf belongs to systemd > package or to nfs-common? I would say it belongs to nfs-common as that > one provides the required tools and services to mount NFS shares on a > client. So the fix I gave you is a fix that shouldn't be copied verbatim into a Debian package. /etc/systemd is administrator territory, /lib/systemd is package territory, so any proper fix of a package should go to /lib. Furthermore, my suggestion for your problem was to add a drop-in that augments the Debian-provided unit files (see man systemd.unit for how drop-ins work) - which is great since you only want to extend the unit, not completely replace it. But in the packages themselves it is perfectly possible to modify the unit files themselves, so a drop-in is not necessary at all for any native unit. So for Stretch: - assuming that nfs-common will have a native unit by then, the proper fix would be to simply add Before=remote-fs-pre.target to that unit, and that would fix that For Jessie: - nfs-common is still an init script, so one cannot simply add Before=remote-fs-pre.target to that. But there are two other options: - just for Jessie: update systemd to change the original unit file remote-fs-pre.target to include After=nfs-common.service - or alternatively, package a drop-in in /lib in the nfs-common package, i.e. /lib/systemd/system/nfs-common.service.d/systemd-ordering.conf: [Unit] Before=remote-fs-pre.target (IMHO at least, I'll defer to the maintainers of the respective packages as to what they think is appropriate.) Christian signature.asc Description: OpenPGP digital signature
Re: auto-mount NFS shares on boot
Hi Christian, Am 27.06.2015 um 16:07 schrieb Christian Seiler: > (Ccing the bugtracker because it appears you've stumbled upon a bug > that also a few other people had, see below. Please don't reply to the > bugtracker yourself unless you feel it's relevant for the bug report.) > > Link to thread on debian-user for people reading the bug report: > https://lists.debian.org/debian-user/2015/06/msg01508.html Thanks for linking this thread to bugreport #775542. I wasn't aware of that bugreport yet. > On 06/27/2015 03:39 PM, Jonas Meurer wrote: >>> Could you run the following? >>> systemd-analyze plot > bootup.svg > > Ok, so the following is going on: > > - local-fs.target is reached, this leads to networking.service > being started > > - networking.service sets up network configuration (takes 172ms) > > - after networking.service is done, network.target is reached > > - after network.target is reached, network-online.target is > reached (since you don't have any services that wait for the > network connection like NetworkManager-wait-online.service, > but you also don't need it here, since networking.service > with a static configuration and 'auto eth0' will make sure > the network is up properly before even network.target is > reached, so that's not a problem) > > - but then immediately after that systemd tries to mount the NFS > filesystem > > - in parallel, first rpcbind and then nfs-common is started Thanks a lot for disassembling and explaining the boot procedure. I see that the problem here is that systemd doesn't wait for rpcbind.service and nfs-common.service to finish before it mounts the NFS shares. > This is a bug, that shouldn't happen. Rationale: > > The problem here is that you are using sec=krb5i type mounts, where > rpc.gssd needs to have been started (by nfs-common) BEFORE mounting > can take place. Unfortunately, there's no ordering relating > nfs-common to remote filesystems, so systemd will start them in > parallel and the mount will fail. > > I myself have never seen this because I've only used sec=sys NFSv4 > mounts with Jessie, and those don't require any service to be started > when trying to mount them - and while the idmapper may be required to > have proper permissions, that can be started later (or not at all if > you use the new nfsidmap + request-key logic instead of idmapd). > > But with sec=krb5i mounts, this is bad, because you need rpc.gssd to > mount the filesystems. > > What's missing here is an ordering dependency between > remote-fs-pre.target and nfs-common.service. > > Searching through the bugtracker, this appears to be the same bug > as #775542 [1], that's why I've copied this message to that bug > report. > > Could you try to do the following: > > 1. create a directory /etc/systemd/system/remote-fs-pre.target.d > 2. create a file /etc/systemd/system/remote-fs-pre.target.d/nfs.conf >with the following contents: > > [Unit] > After=nfs-common.service > > And then reboot your system? I would bet it should work then. Perfect, that solution works like a charm. nfs-common is started before remote-fs, thus rpc.gssd runs already when the NFS share is mounted. I suggest to add this simple fix to Jessie by uploading it to stable-proposed-updates. What do you think? Also, do you think that /etc/systemd/system/remote-fs-pre.target.d/nfs.conf belongs to systemd package or to nfs-common? I would say it belongs to nfs-common as that one provides the required tools and services to mount NFS shares on a client. What do the maintainers think? I'm happy to prepare a patch for either package and eventually make an upload to stable-proposed-updates if the maintainers don't have time to do so themselves. Just tell me if I should go ahead. Cheers, jonas -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/558ee51e.4040...@freesources.org
Re: auto-mount NFS shares on boot
(Ccing the bugtracker because it appears you've stumbled upon a bug that also a few other people had, see below. Please don't reply to the bugtracker yourself unless you feel it's relevant for the bug report.) Link to thread on debian-user for people reading the bug report: https://lists.debian.org/debian-user/2015/06/msg01508.html On 06/27/2015 03:39 PM, Jonas Meurer wrote: > Sure. My /etc/network/interfaces is pretty simple: > >> # This file describes the network interfaces available on your system >> # and how to activate them. For more information, see interfaces(5). >> > # The loopback network interface > auto lo > iface lo inet loopback > > # The primary network interface > #allow-hotplug eth0 > auto eth0 > iface eth0 inet static > address 172.18.10.34 > netmask 255.255.255.0 > network 172.18.10.0 > broadcast 172.18.10.255 > gateway 172.18.10.1 > # dns-* options are implemented by the resolvconf package, if installed > dns-nameservers XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX That looks fine, there shouldn't be any problems with that. >> Could you run the following? >> >> systemd-analyze plot > bootup.svg >> >> This will produce an SVG file that tells you when and in which order >> systemd started different units at boot. Could you attach that file? >> It may be able to tell us what happened. > > Didn't know about that feature until know. Happy to learn new things > about systemd every day :) So ... here we go. The SVG is attached. If I > understand the output, then my NFS share etc-exim4-virtual.mount is > started just before rpcbind.service and network-online.target, right? Ok, so the following is going on: - local-fs.target is reached, this leads to networking.service being started - networking.service sets up network configuration (takes 172ms) - after networking.service is done, network.target is reached - after network.target is reached, network-online.target is reached (since you don't have any services that wait for the network connection like NetworkManager-wait-online.service, but you also don't need it here, since networking.service with a static configuration and 'auto eth0' will make sure the network is up properly before even network.target is reached, so that's not a problem) - but then immediately after that systemd tries to mount the NFS filesystem - in parallel, first rpcbind and then nfs-common is started This is a bug, that shouldn't happen. Rationale: The problem here is that you are using sec=krb5i type mounts, where rpc.gssd needs to have been started (by nfs-common) BEFORE mounting can take place. Unfortunately, there's no ordering relating nfs-common to remote filesystems, so systemd will start them in parallel and the mount will fail. I myself have never seen this because I've only used sec=sys NFSv4 mounts with Jessie, and those don't require any service to be started when trying to mount them - and while the idmapper may be required to have proper permissions, that can be started later (or not at all if you use the new nfsidmap + request-key logic instead of idmapd). But with sec=krb5i mounts, this is bad, because you need rpc.gssd to mount the filesystems. Mounting it later it will work because rpc.gssd has been started by that point. What's missing here is an ordering dependency between remote-fs-pre.target and nfs-common.service. Searching through the bugtracker, this appears to be the same bug as #775542 [1], that's why I've copied this message to that bug report. Could you try to do the following: 1. create a directory /etc/systemd/system/remote-fs-pre.target.d 2. create a file /etc/systemd/system/remote-fs-pre.target.d/nfs.conf with the following contents: [Unit] After=nfs-common.service And then reboot your system? I would bet it should work then. > So maybe you were correct earlier in this thread, that rpcbind is > started to late? Is there an easy solution to fix it? Maybe by adding > rpcbind to the LSB header 'Required-Start' of nfs-common init script? > I just tried that, unfortunately it didn't help at all. You were on the right track here, but it's not rpcbind itself that's the issue. (See above.) For the bugtracker: there has to be a new ordering dependency between remote-fs-pre.target and nfs-common.service. Ideally, since this is NFS-specific, this should probably go in the nfs-common package, but since at least in Jessie nfs-common is an init script and not a systemd service file, it might be better to explicitly add the After= dependency in the systemd package - whereas for Stretch there probably will be a native systemd unit file, so that's where the Before=remote-fs-pre.target dependency could be added. (IMHO.) Christian [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775542 signature.asc Description: OpenPGP digital signature
Re: auto-mount NFS shares on boot
On 06/27/2015 03:12 PM, Jonas Meurer wrote: > So I'm pretty sure that NFS is setup correctly both on server and > client. My problem is about mounting the NFS share automatically during > the boot process. Looks like it, yes. >> However, since you need the interface at boot anyway, you don't >> need allow-hotplug, you can just change it to auto, e.g.: >> allow-hotplug eth0 -> auto eth0 in /etc/network/interfaces >> >> Then /etc/init.d/networking (which is used to set up the >> interfaces in Debian from /etc/network/interfaces) will wait until >> the interface is up before continuing > > I changed 'allow-hotplug eth0' to 'auto eth0' in /etc/network/interfaces > now. Unfortunately it didn't have any effect. Still the same error > message during boot process, still no mounted NFS shares after boot. What kind of network configuration are we looking at? Could you post the contents of /etc/network/interfaces (you can replace the IP addresses with XXX or so if you don't want to disclose internal information, but beyond specific IPs the contents might be relevant). > I'm still pretty confident that systemd for some reason tries to mount > the NFS shares before the network connection is established. The boot > log to console shows the failed mount attempts of NFS shares immediately > after the rootfs has been mounted, and at least before nfs-common has > been started: Could you run the following? systemd-analyze plot > bootup.svg This will produce an SVG file that tells you when and in which order systemd started different units at boot. Could you attach that file? It may be able to tell us what happened. Christian signature.asc Description: OpenPGP digital signature
Re: auto-mount NFS shares on boot
Hi Christian, thanks for the comprehensive reply. It helps a lot to better understand the details of systemd NFS mount handling. Am 27.06.2015 um 12:55 schrieb Christian Seiler: > On 06/26/2015 07:44 PM, Arno Schuring wrote: >>> On 2015-06-26 18:38 +0200, Jonas Meurer wrote: >>> mount.nfs4 prints this not very enlightening message if the mount >>> syscall fails with EINVAL. >> >> If I've understood the mount scripts correctly, the error is correct: >> the _netdev mount option is for mount scripts only, and should /not/ >> be passed to the mount command. > > Not quite. mount(8) and mount.nfs4(8) do understand and ignore the > _netdev option (at least in Jessie's version [1]), so they don't need > to be filtered. That's what I already assumed. Thanks for providing the links to confirm this. > Also: > > - _netdev is not required for nfs4/nfs filesystems, systemd has a >whitelist of filesystems it implicitly considers network >filesystems, and nfs4 is among them [2] > >Also, if systemd hadn't detected it to be a network filesystem, it >would have assumed it to be a local filesystem - and for failing >local filesystems systemd drops you in an emergency shell, so the >systemd wouldn't even have booted then. > >_netdev does have its uses [3], but not for a typical NFS mount. > > - the same error also occurs in the original poster's message when >_netdev was not specified, so it doesn't have to do anything with >this option Ok, so the bottom line is that I don't need the '_netdev' option for NFS mounts at all. I removed it from my fstab. >> I would guess that this is another case where systemd breaks >> backwards compatibility. Maybe it's mentioned in the release notes? > > No, systemd is not at fault here. (See above.) > > With respect to the original problem: > > Q1. Does it mount manually after boot? > > What does mount /var/vmail say? As already mentioned, mounting the filesystems manually after the boot process finished works as expected. I can mount the share by starting 'systemctl start var-vmail.mount'. Afterwards the NFS share is mounted and 'systemctl status var-vmail.mount' says: # systemctl status var-vmail.mount ● var-vmail.mount - /var/vmail Loaded: loaded (/etc/fstab) Active: active (mounted) since Sat 2015-06-27 12:55:55 UTC; 1s ago Where: /var/vmail What: nfs-server:/vmail Docs: man:fstab(5) man:systemd-fstab-generator(8) Process: 982 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail -t nfs4 -o sec=krb5i (code=exited, status=0/SUCCESS) So I'm pretty sure that NFS is setup correctly both on server and client. My problem is about mounting the NFS share automatically during the boot process. > [From here on assuming that the manual mount works:] > > Q5. What kind of network configuration do you use? It may be the case > that you didn't tell it to make systemd wait for it to be online > before attempting to mount stuff.[4] Scenarios: The NFS client in question is a Xen DomU/VM server system. It doesn't have NetworkManager installed and I didn't configure anything systemd-networkd specific. My ethernet devices are configured in /etc/network/interfaces. >- you are using /etc/network/interfaces and have allow-hotplug eth0 > (or whatever your interface is called) in there > > Unfortunately, /etc/network/interfaces is Debian-specific and they > don't have a hook for allow-hotplug interfaces w.r.t. the > network-online.target logic of systemd Very interesting. Indeed, debian-installer by default adds network devices as 'allow-hotplug' to /etc/network/interfaces. > However, since you need the interface at boot anyway, you don't > need allow-hotplug, you can just change it to auto, e.g.: > allow-hotplug eth0 -> auto eth0 in /etc/network/interfaces > > Then /etc/init.d/networking (which is used to set up the > interfaces in Debian from /etc/network/interfaces) will wait until > the interface is up before continuing I changed 'allow-hotplug eth0' to 'auto eth0' in /etc/network/interfaces now. Unfortunately it didn't have any effect. Still the same error message during boot process, still no mounted NFS shares after boot. >- On some systems with static IP addresses (and > /etc/network/interfaces), I had the problem that even though the > interface was conisdered up and ready by the kernel, the switch it > was connected to needed 30s or so to realize that fully (and > packets were simply dropped beforehand). Since those systems also > needed to mount NFS, on Jessie systems I use the following unit > file to try to ping the NFS server before systemd should continue > with NFS mounts: I don't believe that this is a problem here. The NFS client is inside a private network that is routed on the Xen server, so no need to wait for a switch to detect an 'up' link. I'm still pretty confident that systemd for some re
Re: auto-mount NFS shares on boot
On 06/26/2015 07:44 PM, Arno Schuring wrote: >> From: svenj...@gmx.de >> Date: Fri, 26 Jun 2015 19:28:37 +0200 >> On 2015-06-26 18:38 +0200, Jonas Meurer wrote: Process: 352 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail -t nfs4 -o sec=krb5i,_netdev (code=exited, status=32) Jun 26 16:29:02 clt mount[352]: mount.nfs4: an incorrect mount option was specified >> >> mount.nfs4 prints this not very enlightening message if the mount >> syscall fails with EINVAL. > > If I've understood the mount scripts correctly, the error is correct: > the _netdev mount option is for mount scripts only, and should /not/ > be passed to the mount command. Not quite. mount(8) and mount.nfs4(8) do understand and ignore the _netdev option (at least in Jessie's version [1]), so they don't need to be filtered. Also: - _netdev is not required for nfs4/nfs filesystems, systemd has a whitelist of filesystems it implicitly considers network filesystems, and nfs4 is among them [2] Also, if systemd hadn't detected it to be a network filesystem, it would have assumed it to be a local filesystem - and for failing local filesystems systemd drops you in an emergency shell, so the systemd wouldn't even have booted then. _netdev does have its uses [3], but not for a typical NFS mount. - the same error also occurs in the original poster's message when _netdev was not specified, so it doesn't have to do anything with this option > I would guess that this is another case where systemd breaks > backwards compatibility. Maybe it's mentioned in the release notes? No, systemd is not at fault here. (See above.) With respect to the original problem: Q1. Does it mount manually after boot? What does mount /var/vmail say? If it works, skip to Q5, if not, check the following: Q2. You have a krb5 mount, do you have NEED_GSSD=1 in /etc/default/nfs-common? Also, while that should'nt affect the mount itself, sec=krb5 mounts also need the idmapper to work properly, so do you have NEED_IDMAPD=yes in there? Q3. Is your /etc/krb5.keytab set up properly? What does klist -k /etc/krb5.keytab say? Q4. What do the NFS server's logs say? Is there any indication there about the failed mount? [From here on assuming that the manual mount works:] Q5. What kind of network configuration do you use? It may be the case that you didn't tell it to make systemd wait for it to be online before attempting to mount stuff.[4] Scenarios: - you are using NetworkManager. In that case, you need to enable the service that tells systemd to wait for the networking to come up: systemctl enable NetworkManager-wait-online.service - you are using systemd-networkd. Same logic, different service: systemctl enable systemd-networkd-wait-online.service - you are using /etc/network/interfaces and have allow-hotplug eth0 (or whatever your interface is called) in there Unfortunately, /etc/network/interfaces is Debian-specific and they don't have a hook for allow-hotplug interfaces w.r.t. the network-online.target logic of systemd However, since you need the interface at boot anyway, you don't need allow-hotplug, you can just change it to auto, e.g.: allow-hotplug eth0 -> auto eth0 in /etc/network/interfaces Then /etc/init.d/networking (which is used to set up the interfaces in Debian from /etc/network/interfaces) will wait until the interface is up before continuing - On some systems with static IP addresses (and /etc/network/interfaces), I had the problem that even though the interface was conisdered up and ready by the kernel, the switch it was connected to needed 30s or so to realize that fully (and packets were simply dropped beforehand). Since those systems also needed to mount NFS, on Jessie systems I use the following unit file to try to ping the NFS server before systemd should continue with NFS mounts: # /etc/systemd/system/wait-for-nfs-server.service [Unit] Description=Waiting for NFS server DefaultDependencies=no Conflicts=shutdown.target Wants=network-online.target Before=network-online.target shutdown.target After=network.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/sh -c "while ! ping -c 1 HOST_NAME_OF_NFS_SERVER >/dev/null; do sleep 1; done" # Modify this timeout to your heart's content. TimeoutStartSec=60 [Install] WantedBy=network.target (And after that: systemctl enable wait-for-nfs-server.service) Hope that helps. Christian [1] Here's the source code for that and you can see that it understands the option: http://sources.debian.net/src/nfs-utils/1:1.2.8-9/utils/mount/mount.c/#L112 [2] Source code: http://sources.debian.net/src/systemd/215-17%2Bdeb8u1/src/shared/util.c/#L1543 [3] _netdev is useful in 3 use cases: - you have a netwo
Re: auto-mount NFS shares on boot
Hi Brian, Am 27.06.2015 um 07:29 schrieb bri...@aracnet.com: > On Fri, 26 Jun 2015 19:28:37 +0200 > Sven Joachim wrote: > > >> Maybe rpcbind was started too late, see bug #763315[1] on that topic. >> What is the output of "systemctl status rpcbind.service"? > > i have a similar warning/error. > > interestingly my nfs mount seems to work for some reason even > though it claims that there was a problem. You mean that your systemd reports a ordering cycle in dmesg? And your NFS shares are automatically mounted during boot process nevertheless? Would you mind to share the NFS share entries from your /etc/fstab? Cheers, jonas -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/558e7e85.1050...@freesources.org
Re: auto-mount NFS shares on boot
Hi Sven, Am 26.06.2015 um 19:28 schrieb Sven Joachim: > On 2015-06-26 18:38 +0200, Jonas Meurer wrote: >>> # grep nfs /etc/fstab >>> nfs-server:/vmail /var/vmail nfs4 _netdev,sec=krb5i,bg 0 0 >> >> results in unmounted NFS shares after reboot and: >> >>> >>> Jun 26 16:29:02 clt mount[352]: mount.nfs4: an incorrect mount >>> option was specified > > mount.nfs4 prints this not very enlightening message if the mount > syscall fails with EINVAL. Very helpful ;) Thanks for the hint. At least it's good to know that my fstab options are not the problem here. >> Maybe systemd uses an own impementation of mount.nfs4 and that doesn't >> support the 'sec=krb5i' option? > > No, it uses the tools from nfs-common. Ah, thanks for clarifying. But at least the post if-up.d hook from /etc/network/if-up.d/mountnfs exits without any action in case of a running systemd: # Skip the mountnfs hook when being triggered by the networking SysV init # script and instead use the systemd built-in mechanisms to mount remote # file systems. # This avoids a deadlock caused by the rpcbind SysV init script depending # on $network and the $network LSB facility being provided by the networking # SysV init script. if [ -d /run/systemd/system ]; then systemctl list-jobs | grep -q network.target && exit 0 fi >> Executing the exact ExecMount command >> line from above manually after the boot process works as expected. > > Maybe rpcbind was started too late, see bug #763315[1] on that topic. > What is the output of "systemctl status rpcbind.service"? Mh, my first impression is that the bugreport is unrelated. I don't find any systemd messages regarding ordering cycles in my dmesg. Also, rpcbind is running at least after the boot process: # systemctl status rpcbind.service ● rpcbind.service - LSB: RPC portmapper replacement Loaded: loaded (/etc/init.d/rpcbind) Drop-In: /run/systemd/generator/rpcbind.service.d └─50-rpcbind-$portmap.conf Active: active (running) since Sat 2015-06-27 10:21:10 UTC; 9min ago CGroup: /system.slice/rpcbind.service └─368 /sbin/rpcbind -w Jun 27 10:21:10 clt rpcbind[350]: Starting rpcbind daemon My impression is, that systemd tries to mount the NFS shares directly after the root fs is mounted. At this time it fails because the network is not setup yet. Afterwards, nothing picks up the missing NFS mounts and as a result the boot process ends without NFS shares being mounted. I don't know much about systemd and its changes to the boot process, but I suspect that (some part of) systemd is meant to mount the NFS shares once the network connection is established. Seems like that doesn't happen. /etc/init.d/mountnfs.sh doesn't do anything on systemd systems as it merely invokes /etc/network/if-up.d/mountnfs - which in turn doesn't do anything on systemd systems (see above). Cheers, jonas -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/558e7c63.4080...@freesources.org
Re: auto-mount NFS shares on boot
On 15-06-26 5:42 PM, Jonas Meurer wrote: > Problem seems to be that the NFS mounts are mounted directly after > the root fs, before the network is set up properly. Also, the mountnfs > if-up.d hook from /etc/network/if-up.d/mountnfs doesn't work for > systemd systems. > > Here's the relevant output from the boot console: > >> [FAILED] Failed to mount /var/vmail. >> See 'systemctl status var-vmail.mount' for details. >> [DEPEND] Dependency failed for Remote File Systems. >> Starting Trigger Flushing of Journal to Persistent Storage... >> Starting LSB: Prepare console... > > And 'systemctl status var-vmail.mount' reveils: > >> Process: 352 ExecMount=/bin/mount -n nfs1.freesources.org:/vmail >> /var/vmail -t nfs4 -o sec=krb5i,bg (code=exited, status=32) I just wanted to say that I noticed the same here. That's all. I ignored them, as much later all the shares mounted. But I'd been thinking of finding a solution. Glad to see you started this thread, which I will monitor. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/558e48bc.8080...@gmx.de
Re: auto-mount NFS shares on boot
On Fri, 26 Jun 2015 19:28:37 +0200 Sven Joachim wrote: > Maybe rpcbind was started too late, see bug #763315[1] on that topic. > What is the output of "systemctl status rpcbind.service"? > > Cheers, >Sven > > > 1. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763315 > i have a similar warning/error. interestingly my nfs mount seems to work for some reason even though it claims that there was a problem. short of waiting for some sort of change as the bug report talks about, is there a good way to handle automounting nfs partitions at boot ? Brian -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150626222904.141c8...@cedar.deldotd.com
RE: auto-mount NFS shares on boot
> From: svenj...@gmx.de > Date: Fri, 26 Jun 2015 19:28:37 +0200 > > On 2015-06-26 18:38 +0200, Jonas Meurer wrote: >>> >>> # systemctl status var-vmail.mount >>> ● var-vmail.mount - /var/vmail >>> Loaded: loaded (/etc/fstab) >>> Active: failed (Result: exit-code) since Fri 2015-06-26 16:29:02 >>> UTC; 6s ago >>> Where: /var/vmail >>> What: nfs-server:/vmail >>> Docs: man:fstab(5) >>> man:systemd-fstab-generator(8) >>> Process: 352 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail >>> -t >>> nfs4 -o sec=krb5i,_netdev (code=exited, status=32) >>> >>> Jun 26 16:29:02 clt mount[352]: mount.nfs4: an incorrect mount >>> option was specified > > mount.nfs4 prints this not very enlightening message if the mount > syscall fails with EINVAL. If I've understood the mount scripts correctly, the error is correct: the _netdev mount option is for mount scripts only, and should /not/ be passed to the mount command. I would guess that this is another case where systemd breaks backwards compatibility. Maybe it's mentioned in the release notes? Regards, Arno
Re: auto-mount NFS shares on boot
On 2015-06-26 18:38 +0200, Jonas Meurer wrote: > Oops, I searched only in fstab(5) and nfs(5) manpages. Still, the > option doesn't work: > >> # grep nfs /etc/fstab >> nfs-server:/vmail /var/vmail nfs4 _netdev,sec=krb5i,bg 0 0 > > results in unmounted NFS shares after reboot and: > >> [FAILED] Failed to mount /var/vmail. >> See 'systemctl status var-vmail.mount' for details. >> [DEPEND] Dependency failed for Remote File Systems. >> Starting Trigger Flushing of Journal to Persistent Storage... >> Starting LSB: Prepare console... >> >> # systemctl status var-vmail.mount >> ● var-vmail.mount - /var/vmail >>Loaded: loaded (/etc/fstab) >>Active: failed (Result: exit-code) since Fri 2015-06-26 16:29:02 >> UTC; 6s ago >> Where: /var/vmail >> What: nfs-server:/vmail >> Docs: man:fstab(5) >>man:systemd-fstab-generator(8) >> Process: 352 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail >> -t >> nfs4 -o sec=krb5i,_netdev (code=exited, status=32) >> >> Jun 26 16:29:02 clt mount[352]: mount.nfs4: an incorrect mount >> option was specified mount.nfs4 prints this not very enlightening message if the mount syscall fails with EINVAL. > Maybe systemd uses an own impementation of mount.nfs4 and that doesn't > support the 'sec=krb5i' option? No, it uses the tools from nfs-common. > Executing the exact ExecMount command > line from above manually after the boot process works as expected. Maybe rpcbind was started too late, see bug #763315[1] on that topic. What is the output of "systemctl status rpcbind.service"? Cheers, Sven 1. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763315 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/878ub6v0oq@turtle.gmx.de
Re: auto-mount NFS shares on boot
Hi again, Am 2015-06-26 18:15, schrieb Sven Joachim: On 2015-06-26 17:42 +0200, Jonas Meurer wrote: While searching the web I learned that Ubuntu and Redhat based distros seem to support the '_netdev' option to entries in /etc/fstab for such cases, but it seems like Debian doesn't support this option, right? Huh, what makes you think so? The option is mentioned in mount(8) and has been present since util-linux version 2.11x, released 2002. Oops, I searched only in fstab(5) and nfs(5) manpages. Still, the option doesn't work: # grep nfs /etc/fstab nfs-server:/vmail /var/vmail nfs4 _netdev,sec=krb5i,bg 0 0 results in unmounted NFS shares after reboot and: [FAILED] Failed to mount /var/vmail. See 'systemctl status var-vmail.mount' for details. [DEPEND] Dependency failed for Remote File Systems. Starting Trigger Flushing of Journal to Persistent Storage... Starting LSB: Prepare console... # systemctl status var-vmail.mount ● var-vmail.mount - /var/vmail Loaded: loaded (/etc/fstab) Active: failed (Result: exit-code) since Fri 2015-06-26 16:29:02 UTC; 6s ago Where: /var/vmail What: nfs-server:/vmail Docs: man:fstab(5) man:systemd-fstab-generator(8) Process: 352 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail -t nfs4 -o sec=krb5i,_netdev (code=exited, status=32) Jun 26 16:29:02 clt mount[352]: mount.nfs4: an incorrect mount option was specified Jun 26 16:29:02 clt systemd[1]: var-vmail.mount mount process exited, code=exited status=32 Jun 26 16:29:02 clt systemd[1]: Failed to mount /var/vmail. Jun 26 16:29:02 clt systemd[1]: Unit var-vmail.mount entered failed state. Am 2015-06-26 18:01, schrieb Michael Beck: does the NFS-"soft"-mount-option work for your NFS-Client? nope, doesn't help either. Actually, even with only 'sec=krb5i' as single option, 'systemctl status var-vmail.mount' reports afterwards: [...] Process: 351 ExecMount=/bin/mount -n nfs-server:/vmail /var/vmail -t nfs4 -o sec=krb5i (code=exited, status=32) Jun 26 16:34:08 clt mount[351]: mount.nfs4: an incorrect mount option was specified Jun 26 16:34:08 clt systemd[1]: var-vmail.mount mount process exited, code=exited status=32 Jun 26 16:34:08 clt systemd[1]: Failed to mount /var/vmail. Jun 26 16:34:08 clt systemd[1]: Unit var-vmail.mount entered failed state. Maybe systemd uses an own impementation of mount.nfs4 and that doesn't support the 'sec=krb5i' option? Executing the exact ExecMount command line from above manually after the boot process works as expected. Cheers, jonas -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/f5926f1eef088ed9bbe2815296bd7...@imap.steindlberger.de
Re: auto-mount NFS shares on boot
On 2015-06-26 17:42 +0200, Jonas Meurer wrote: > While searching the web I learned that Ubuntu and Redhat based distros > seem to support the '_netdev' option to entries in /etc/fstab for such > cases, but it seems like Debian doesn't support this option, right? Huh, what makes you think so? The option is mentioned in mount(8) and has been present since util-linux version 2.11x, released 2002. Cheers, Sven -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/87egkyv426@turtle.gmx.de
Re: auto-mount NFS shares on boot
> Am 26.06.2015 um 17:42 schrieb Jonas Meurer : > > Hello, > > I fail to configure my Debian/Jessie server system in a way that the > NFS mounts are mounted automatically on boot. > > Problem seems to be that the NFS mounts are mounted directly after > the root fs, before the network is set up properly. Also, the mountnfs > if-up.d hook from /etc/network/if-up.d/mountnfs doesn't work for > systemd systems. > > Here's the relevant output from the boot console: > >> [FAILED] Failed to mount /var/vmail. >> See 'systemctl status var-vmail.mount' for details. >> [DEPEND] Dependency failed for Remote File Systems. >> Starting Trigger Flushing of Journal to Persistent Storage... >> Starting LSB: Prepare console... > > And 'systemctl status var-vmail.mount' reveils: > >> Process: 352 ExecMount=/bin/mount -n nfs1.freesources.org:/vmail >>/var/vmail -t nfs4 -o sec=krb5i,bg (code=exited, status=32) > > While searching the web I learned that Ubuntu and Redhat based distros > seem to support the '_netdev' option to entries in /etc/fstab for such > cases, but it seems like Debian doesn't support this option, right? > > Is there any other solution to mount NFS shares during boot process on > Debian Jessie systems? Mounting the manually (or by rc.local) later is > not an option for me. Several service daemons that start later in the > boot process depend on the NFS shares being mounted. > > Cheers, > jonas > > > -- > To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject > of "unsubscribe". Trouble? Contact listmas...@lists.debian.org > Archive: > https://lists.debian.org/ca566086919f8e97d53bf5bca7d4e...@imap.steindlberger.de > Hi there, does the NFS-"soft"-mount-option work for your NFS-Client? Gruß, Michael Beck -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/929f19b8-9e99-45e9-bf2e-7c01b2b9b...@gmail.com
