[Touch-packages] [Bug 1864708] Re: Mount unit fails but systemctl status shows success
> I don't know why, but this doesn't result in the "Mountpoint not empty" error (or any error/warning for that matter). I thought systemd should be able to figure out the order of nested mounts, not sure why it's not working here. because systemd doesn't mount zfs mounts; zfs is a "magical" filesystem that mounts itself because it's just so smart and special. All zfs mounts are performed by 'zfs-mount.service' and systemd only creates corresponding mount units after zfs performs the mounts (i.e. the systemd mount units for zfs mounts only indicate the mount state, they don't actually perform the mounting). ** Changed in: systemd (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1864708 Title: Mount unit fails but systemctl status shows success Status in systemd package in Ubuntu: Invalid Bug description: I have this mount unit # cat /etc/systemd/system/srv-cctvstore.mount [Unit] Description=Drive for CCTVstore [Mount] What=/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 Where=/srv/cctvstore Type=ext4 Options=defaults [Install] WantedBy=multi-user.target On boot systemd thinks it is mounted: $ systemctl status srv-cctvstore.mount ● srv-cctvstore.mount - Drive for CCTVstore Loaded: loaded (/etc/systemd/system/srv-cctvstore.mount; enabled; vendor preset: enabled) Active: active (mounted) since Tue 2020-02-25 22:37:16 SAST; 48s ago Where: /srv/cctvstore What: /dev/sda1 Process: 1294 ExecMount=/bin/mount /dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 /srv/cctvstore -t ext4 -o defaults (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4915) CGroup: /system.slice/srv-cctvstore.mount Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore... Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore. and the mount command seems to agree $ mount | grep cctv /dev/sda1 on /srv/cctvstore type ext4 (rw,relatime) but the mountpoint is empty: $ ls -al /srv/cctvstore/ total 1 drwxr-xr-x 2 root root 2 Dec 7 22:51 . drwxr-xr-x 4 root root 4 Dec 7 22:51 .. and umount thinks it is not mounted: $ sudo umount /srv/cctvstore [sudo] password for johann: umount: /srv/cctvstore: not mounted. journalctl doesn't report any funnies: $ journalctl -u srv-cctvstore.mount -- Logs begin at Tue 2020-02-25 22:37:16 SAST, end at Tue 2020-02-25 22:41:43 SAST. -- Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore... Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore. but mounting by hand works: $ sudo mount /dev/sda1 /srv/cctvstore johann@kluis:~$ ls -al /srv/cctvstore total 3157 drwxrwxrwx 7 root root 4096 Dec 9 22:27 . drwxr-xr-x 4 root root 4 Dec 7 22:51 .. drwxr-xr-x 14 165569 1655694096 Dec 10 23:10 events drwxr-xr-x 2 165569 1655694096 Dec 7 22:59 images drwxr-xr-x 2 165569 1655694096 Dec 7 23:01 logs drwx-- 2 root root 16384 Dec 7 22:47 lost+found drwxrwxr-x 2 165569 165569 3194880 Feb 25 22:25 mlevents I am unsure where to go next, any help appreciated. ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: systemd 237-3ubuntu10.39 ProcVersionSignature: Ubuntu 5.3.0-40.32~18.04.1-generic 5.3.18 Uname: Linux 5.3.0-40-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.9-0ubuntu7.11 Architecture: amd64 Date: Tue Feb 25 22:27:59 2020 Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M. ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcKernelCmdLine: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-5.3.0-40-generic root=ZFS=rpool/ROOT/ubuntu ro SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 03/07/2018 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: P2.50 dmi.board.name: E3C232D2I dmi.board.vendor: ASRockRack dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 3 dmi.chassis.vendor: To Be Filled By O.E.M. dmi.chassis.version: To Be Filled By O.E.M. dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP2.50:bd03/07/2018:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRockRack:rnE3C232D2I:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.: dmi.product.family: To Be Filled By O.E.M. dmi.product.name: To Be Filled By O.E.M. dmi.product.sku: To Be Filled By O.E.M. dmi.product.version: To Be Filled By O.E.M. dmi.sys.vendor: To Be Filled By O.E.M. modified.conffile..etc.systemd.resolved.conf: [modified] mtime.conffile..etc.systemd.resolved.conf:
[Touch-packages] [Bug 1864708] Re: Mount unit fails but systemctl status shows success
So I spent a bunch more time on this tonight, and you can close this, it's not a systemd bug. Basically the problem was an ordering issue between this mount unit and a zfs mount unit responsible for providing /srv. What was happening was the /srv/cctvstore was getting mounted first and then /srv was getting mounted on top, obscuring the /srv/cctvstore mount underneath. I don't know why, but this doesn't result in the "Mountpoint not empty" error (or any error/warning for that matter). I thought systemd should be able to figure out the order of nested mounts, not sure why it's not working here. Apologies for wasting your time, and thanks for looking into this. If anyone runs into this in the future, this is what my mount unit ended up looking like: [Unit] Description=Drive for CCTVstore After=zfs.target Requires=srv.mount [Mount] What=/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 Where=/srv/cctvstore Type=ext4 Options=defaults [Install] WantedBy=lxd.service - $ systemctl list-units -t mount UNIT LOAD ACTIVE SUB DESCRIPTION -.mount loaded active mounted Root Mount dev-hugepages.mount loaded active mounted Huge Pages File System dev-mqueue.mount loaded active mounted POSIX Message Queue File System home-johann.mount loaded active mounted /home/johann home.mount loaded active mounted /home proc-sys-fs-binfmt_misc.mount loaded active mounted Arbitrary Executable File Formats File System run-user-1000.mount loaded active mounted /run/user/1000 srv-cctvstore.mount loaded active mounted Drive for CCTVstore srv.mount loaded active mounted /srv sys-fs-fuse-connections.mount loaded active mounted FUSE Control File System sys-kernel-config.mount loaded active mounted Kernel Configuration File System sys-kernel-debug.mount loaded active mounted Kernel Debug File System var-cache.mount loaded active mounted /var/cache var-games.mount loaded active mounted /var/games var-lib-lxcfs.mount loaded active mounted /var/lib/lxcfs var-lib-lxd-devices-storj-disk.aadisable.sys\x2dmodule\x2dapparmor\x2dparameters\x2denabled.mount loaded active mounted /var/lib/lxd/devices/storj/disk.aadisable.sys-module-apparmor-parameters-enabled var-lib-lxd-devices-zm-disk.disk.srv\x2dcctvstore.mount loaded active mounted /var/lib/lxd/devices/zm/disk.disk.srv-cctvstore var-lib-lxd-devlxd.mount loaded active mounted /var/lib/lxd/devlxd var-lib-lxd-shmounts.mount loaded active
[Touch-packages] [Bug 1864708] Re: Mount unit fails but systemctl status shows success
can you attach/paste the output from these commands (after booting, before manually mounting): $ journalctl -b --no-pager $ systemctl list-units -t mount $ mount $ cat /etc/fstab ** Changed in: systemd (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1864708 Title: Mount unit fails but systemctl status shows success Status in systemd package in Ubuntu: Incomplete Bug description: I have this mount unit # cat /etc/systemd/system/srv-cctvstore.mount [Unit] Description=Drive for CCTVstore [Mount] What=/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 Where=/srv/cctvstore Type=ext4 Options=defaults [Install] WantedBy=multi-user.target On boot systemd thinks it is mounted: $ systemctl status srv-cctvstore.mount ● srv-cctvstore.mount - Drive for CCTVstore Loaded: loaded (/etc/systemd/system/srv-cctvstore.mount; enabled; vendor preset: enabled) Active: active (mounted) since Tue 2020-02-25 22:37:16 SAST; 48s ago Where: /srv/cctvstore What: /dev/sda1 Process: 1294 ExecMount=/bin/mount /dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 /srv/cctvstore -t ext4 -o defaults (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4915) CGroup: /system.slice/srv-cctvstore.mount Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore... Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore. and the mount command seems to agree $ mount | grep cctv /dev/sda1 on /srv/cctvstore type ext4 (rw,relatime) but the mountpoint is empty: $ ls -al /srv/cctvstore/ total 1 drwxr-xr-x 2 root root 2 Dec 7 22:51 . drwxr-xr-x 4 root root 4 Dec 7 22:51 .. and umount thinks it is not mounted: $ sudo umount /srv/cctvstore [sudo] password for johann: umount: /srv/cctvstore: not mounted. journalctl doesn't report any funnies: $ journalctl -u srv-cctvstore.mount -- Logs begin at Tue 2020-02-25 22:37:16 SAST, end at Tue 2020-02-25 22:41:43 SAST. -- Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore... Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore. but mounting by hand works: $ sudo mount /dev/sda1 /srv/cctvstore johann@kluis:~$ ls -al /srv/cctvstore total 3157 drwxrwxrwx 7 root root 4096 Dec 9 22:27 . drwxr-xr-x 4 root root 4 Dec 7 22:51 .. drwxr-xr-x 14 165569 1655694096 Dec 10 23:10 events drwxr-xr-x 2 165569 1655694096 Dec 7 22:59 images drwxr-xr-x 2 165569 1655694096 Dec 7 23:01 logs drwx-- 2 root root 16384 Dec 7 22:47 lost+found drwxrwxr-x 2 165569 165569 3194880 Feb 25 22:25 mlevents I am unsure where to go next, any help appreciated. ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: systemd 237-3ubuntu10.39 ProcVersionSignature: Ubuntu 5.3.0-40.32~18.04.1-generic 5.3.18 Uname: Linux 5.3.0-40-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.9-0ubuntu7.11 Architecture: amd64 Date: Tue Feb 25 22:27:59 2020 Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M. ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcKernelCmdLine: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-5.3.0-40-generic root=ZFS=rpool/ROOT/ubuntu ro SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 03/07/2018 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: P2.50 dmi.board.name: E3C232D2I dmi.board.vendor: ASRockRack dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 3 dmi.chassis.vendor: To Be Filled By O.E.M. dmi.chassis.version: To Be Filled By O.E.M. dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP2.50:bd03/07/2018:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRockRack:rnE3C232D2I:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.: dmi.product.family: To Be Filled By O.E.M. dmi.product.name: To Be Filled By O.E.M. dmi.product.sku: To Be Filled By O.E.M. dmi.product.version: To Be Filled By O.E.M. dmi.sys.vendor: To Be Filled By O.E.M. modified.conffile..etc.systemd.resolved.conf: [modified] mtime.conffile..etc.systemd.resolved.conf: 2020-01-08T12:29:15.304369 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1864708/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
