So I spent a bunch more time on this tonight, and you can close this,
it's not a systemd bug.
Basically the problem was an ordering issue between this mount unit and
a zfs mount unit responsible for providing /srv.
What was happening was the /srv/cctvstore was getting mounted first and
then /srv was getting mounted on top, obscuring the /srv/cctvstore mount
underneath. I don't know why, but this doesn't result in the "Mountpoint
not empty" error (or any error/warning for that matter). I thought
systemd should be able to figure out the order of nested mounts, not
sure why it's not working here.
Apologies for wasting your time, and thanks for looking into this.
If anyone runs into this in the future, this is what my mount unit ended
up looking like:
[Unit]
Description=Drive for CCTVstore
After=zfs.target
Requires=srv.mount
[Mount]
What=/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1
Where=/srv/cctvstore
Type=ext4
Options=defaults
[Install]
WantedBy=lxd.service
---------
$ systemctl list-units -t mount
UNIT
LOAD ACTIVE SUB DESCRIPTION
-.mount
loaded active mounted Root Mount
dev-hugepages.mount
loaded active mounted Huge Pages File System
dev-mqueue.mount
loaded active mounted POSIX Message Queue File System
home-johann.mount
loaded active mounted /home/johann
home.mount
loaded active mounted /home
proc-sys-fs-binfmt_misc.mount
loaded active mounted Arbitrary Executable File Formats File
System
run-user-1000.mount
loaded active mounted /run/user/1000
srv-cctvstore.mount
loaded active mounted Drive for CCTVstore
srv.mount
loaded active mounted /srv
sys-fs-fuse-connections.mount
loaded active mounted FUSE Control File System
sys-kernel-config.mount
loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount
loaded active mounted Kernel Debug File System
var-cache.mount
loaded active mounted /var/cache
var-games.mount
loaded active mounted /var/games
var-lib-lxcfs.mount
loaded active mounted /var/lib/lxcfs
var-lib-lxd-devices-storj-disk.aadisable.sys\x2dmodule\x2dapparmor\x2dparameters\x2denabled.mount
loaded active mounted
/var/lib/lxd/devices/storj/disk.aadisable.sys-module-apparmor-parameters-enabled
var-lib-lxd-devices-zm-disk.disk.srv\x2dcctvstore.mount
loaded active mounted
/var/lib/lxd/devices/zm/disk.disk.srv-cctvstore
var-lib-lxd-devlxd.mount
loaded active mounted /var/lib/lxd/devlxd
var-lib-lxd-shmounts.mount
loaded active mounted /var/lib/lxd/shmounts
var-lib-lxd-storage\x2dpools-lxd-containers-commandfusion.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/commandfusion
var-lib-lxd-storage\x2dpools-lxd-containers-emby.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/emby
var-lib-lxd-storage\x2dpools-lxd-containers-j1backup.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/j1backup
var-lib-lxd-storage\x2dpools-lxd-containers-johann.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/johann
var-lib-lxd-storage\x2dpools-lxd-containers-kaarte.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/kaarte
var-lib-lxd-storage\x2dpools-lxd-containers-nextcloud.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/nextcloud
var-lib-lxd-storage\x2dpools-lxd-containers-smb.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/smb
var-lib-lxd-storage\x2dpools-lxd-containers-storj.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/storj
var-lib-lxd-storage\x2dpools-lxd-containers-transnet.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/transnet
var-lib-lxd-storage\x2dpools-lxd-containers-wgtunnel.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/wgtunnel
var-lib-lxd-storage\x2dpools-lxd-containers-zm.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/zm
var-lib-lxd-storage\x2dpools-lxd-containers-zmnbuild.mount
loaded active mounted
/var/lib/lxd/storage-pools/lxd/containers/zmnbuild
var-lib-lxd-storage\x2dpools-ssdlxd-containers-librenms.mount
loaded active mounted
/var/lib/lxd/storage-pools/ssdlxd/containers/librenms
var-lib-lxd-storage\x2dpools-ssdlxd-containers-ns.mount
loaded active mounted
/var/lib/lxd/storage-pools/ssdlxd/containers/ns
var-lib-lxd-storage\x2dpools-ssdlxd-containers-pihole.mount
loaded active mounted
/var/lib/lxd/storage-pools/ssdlxd/containers/pihole
var-lib-lxd-storage\x2dpools-ssdlxd-containers-unifi.mount
loaded active mounted
/var/lib/lxd/storage-pools/ssdlxd/containers/unifi
var-log.mount
loaded active mounted /var/log
var-mail.mount
loaded active mounted /var/mail
var-spool.mount
loaded active mounted /var/spool
var-tmp.mount
loaded active mounted /var/tmp
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
39 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
---------------
$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs
(rw,nosuid,relatime,size=16384920k,nr_inodes=4096230,mode=755)
devpts on /dev/pts type devpts
(rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3284068k,mode=755)
rpool/ROOT/ubuntu on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/hugetlb type cgroup
(rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2594)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
rpool/var/log on /var/log type zfs (rw,relatime,xattr,posixacl)
rpool/var/tmp on /var/tmp type zfs (rw,relatime,xattr,noacl)
rpool/var/mail on /var/mail type zfs
(rw,nosuid,nodev,noexec,noatime,xattr,noacl)
rpool/var/cache on /var/cache type zfs
(rw,nosuid,nodev,noexec,noatime,xattr,noacl)
rpool/home on /home type zfs (rw,nosuid,nodev,noatime,xattr,noacl)
rpool/var/games on /var/games type zfs
(rw,nosuid,nodev,noexec,noatime,xattr,noacl)
rpool/var/spool on /var/spool type zfs
(rw,nosuid,nodev,noexec,noatime,xattr,noacl)
rpool/home/johann on /home/johann type zfs (rw,nosuid,nodev,noatime,xattr,noacl)
srv on /srv type zfs (rw,noatime,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs
(rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/sda1 on /srv/cctvstore type ext4 (rw,relatime)
tmpfs on /var/lib/lxd/shmounts type tmpfs (rw,relatime,size=100k,mode=711)
tmpfs on /var/lib/lxd/devlxd type tmpfs (rw,relatime,size=100k,mode=755)
rpool/lxd/containers/ns on /var/lib/lxd/storage-pools/ssdlxd/containers/ns type
zfs (rw,relatime,xattr,posixacl)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
srv/lxd/containers/commandfusion on
/var/lib/lxd/storage-pools/lxd/containers/commandfusion type zfs
(rw,relatime,xattr,noacl)
tmpfs on /run/user/1000 type tmpfs
(rw,nosuid,nodev,relatime,size=3284064k,mode=700,uid=1000,gid=1000)
srv/lxd/containers/emby on /var/lib/lxd/storage-pools/lxd/containers/emby type
zfs (rw,relatime,xattr,noacl)
srv/lxd/containers/j1backup on
/var/lib/lxd/storage-pools/lxd/containers/j1backup type zfs
(rw,relatime,xattr,noacl)
srv/lxd/containers/johann on /var/lib/lxd/storage-pools/lxd/containers/johann
type zfs (rw,relatime,xattr,noacl)
srv/lxd/containers/kaarte on /var/lib/lxd/storage-pools/lxd/containers/kaarte
type zfs (rw,relatime,xattr,noacl)
rpool/lxd/containers/librenms on
/var/lib/lxd/storage-pools/ssdlxd/containers/librenms type zfs
(rw,relatime,xattr,posixacl)
srv/lxd/containers/nextcloud on
/var/lib/lxd/storage-pools/lxd/containers/nextcloud type zfs
(rw,relatime,xattr,noacl)
rpool/lxd/containers/pihole on
/var/lib/lxd/storage-pools/ssdlxd/containers/pihole type zfs
(rw,relatime,xattr,posixacl)
srv/lxd/containers/smb on /var/lib/lxd/storage-pools/lxd/containers/smb type
zfs (rw,relatime,xattr,noacl)
udev on
/var/lib/lxd/devices/storj/disk.aadisable.sys-module-apparmor-parameters-enabled
type devtmpfs (rw,nosuid,relatime,size=16384920k,nr_inodes=4096230,mode=755)
srv/lxd/containers/storj on /var/lib/lxd/storage-pools/lxd/containers/storj
type zfs (rw,relatime,xattr,noacl)
srv/lxd/containers/transnet on
/var/lib/lxd/storage-pools/lxd/containers/transnet type zfs
(rw,relatime,xattr,noacl)
rpool/lxd/containers/unifi on
/var/lib/lxd/storage-pools/ssdlxd/containers/unifi type zfs
(rw,relatime,xattr,posixacl)
srv/lxd/containers/wgtunnel on
/var/lib/lxd/storage-pools/lxd/containers/wgtunnel type zfs
(rw,relatime,xattr,noacl)
/dev/sda1 on /var/lib/lxd/devices/zm/disk.disk.srv-cctvstore type ext4
(rw,relatime)
srv/lxd/containers/zm on /var/lib/lxd/storage-pools/lxd/containers/zm type zfs
(rw,relatime,xattr,noacl)
srv/lxd/containers/zmnbuild on
/var/lib/lxd/storage-pools/lxd/containers/zmnbuild type zfs
(rw,relatime,xattr,noacl)
---------------------
$ cat /etc/fstab
rpool/var/log /var/log zfs defaults 0 0
rpool/var/tmp /var/tmp zfs defaults 0 0
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1864708
Title:
Mount unit fails but systemctl status shows success
Status in systemd package in Ubuntu:
Incomplete
Bug description:
I have this mount unit
# cat /etc/systemd/system/srv-cctvstore.mount
[Unit]
Description=Drive for CCTVstore
[Mount]
What=/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1
Where=/srv/cctvstore
Type=ext4
Options=defaults
[Install]
WantedBy=multi-user.target
On boot systemd thinks it is mounted:
$ systemctl status srv-cctvstore.mount
● srv-cctvstore.mount - Drive for CCTVstore
Loaded: loaded (/etc/systemd/system/srv-cctvstore.mount; enabled; vendor
preset: enabled)
Active: active (mounted) since Tue 2020-02-25 22:37:16 SAST; 48s ago
Where: /srv/cctvstore
What: /dev/sda1
Process: 1294 ExecMount=/bin/mount
/dev/disk/by-id/ata-ST2000NM0011_Z1P21NQ7-part1 /srv/cctvstore -t ext4 -o
defaults (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/srv-cctvstore.mount
Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore...
Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore.
and the mount command seems to agree
$ mount | grep cctv
/dev/sda1 on /srv/cctvstore type ext4 (rw,relatime)
but the mountpoint is empty:
$ ls -al /srv/cctvstore/
total 1
drwxr-xr-x 2 root root 2 Dec 7 22:51 .
drwxr-xr-x 4 root root 4 Dec 7 22:51 ..
and umount thinks it is not mounted:
$ sudo umount /srv/cctvstore
[sudo] password for johann:
umount: /srv/cctvstore: not mounted.
journalctl doesn't report any funnies:
$ journalctl -u srv-cctvstore.mount
-- Logs begin at Tue 2020-02-25 22:37:16 SAST, end at Tue 2020-02-25 22:41:43
SAST. --
Feb 25 22:37:16 kluis systemd[1]: Mounting Drive for CCTVstore...
Feb 25 22:37:16 kluis systemd[1]: Mounted Drive for CCTVstore.
but mounting by hand works:
$ sudo mount /dev/sda1 /srv/cctvstore
johann@kluis:~$ ls -al /srv/cctvstore
total 3157
drwxrwxrwx 7 root root 4096 Dec 9 22:27 .
drwxr-xr-x 4 root root 4 Dec 7 22:51 ..
drwxr-xr-x 14 165569 165569 4096 Dec 10 23:10 events
drwxr-xr-x 2 165569 165569 4096 Dec 7 22:59 images
drwxr-xr-x 2 165569 165569 4096 Dec 7 23:01 logs
drwx------ 2 root root 16384 Dec 7 22:47 lost+found
drwxrwxr-x 2 165569 165569 3194880 Feb 25 22:25 mlevents
I am unsure where to go next, any help appreciated.
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.39
ProcVersionSignature: Ubuntu 5.3.0-40.32~18.04.1-generic 5.3.18
Uname: Linux 5.3.0-40-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
Date: Tue Feb 25 22:27:59 2020
Lsusb:
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/ROOT/ubuntu@/boot/vmlinuz-5.3.0-40-generic
root=ZFS=rpool/ROOT/ubuntu ro
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/07/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: P2.50
dmi.board.name: E3C232D2I
dmi.board.vendor: ASRockRack
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias:
dmi:bvnAmericanMegatrendsInc.:bvrP2.50:bd03/07/2018:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRockRack:rnE3C232D2I:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: To Be Filled By O.E.M.
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.
modified.conffile..etc.systemd.resolved.conf: [modified]
mtime.conffile..etc.systemd.resolved.conf: 2020-01-08T12:29:15.304369
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1864708/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
