Package: util-linux
Version: 2.27.1-1
Architecture: i386
(libmount1 and libblkid1 are also 2.27.1.)
My most recent reboot (and it's been a while, so it may be some dependent
library or new 4.4 kernel or something) failed in checkfs.sh.
Upon debugging, it appeared that the invocation of "fsck -C -M -A -a"
was, despite the -M flag, trying to check /dev/sda2, my root file
system.
e2fsck of course complained about this, leading to a recovery shell
and a need for manual intervention.
$ cat /var/log/fsck/checkfs
Log of fsck -C -M -A -a
Sun Jan 24 20:36:30 2016
fsck from util-linux 2.27.1
/dev/sda2 is mounted.
e2fsck: Cannot continue, aborting.
fsck exited with status code 8
Sun Jan 24 20:36:30 2016
----------------
Running fsck under strace shows the following interesting parts
(boring bits like memory allocation elided):
open("/etc/fstab", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, ...)
close(3) = 0
lstat64("/dev", {st_mode=S_IFDIR|0755, st_size=3420, ...}) = 0
lstat64("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2), ...}) = 0
stat64("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2), ...}) = 0
# NFclue why it needs a stat() after lstat() says it's not a symlink
stat64("/sbin/fsck.ext4", ...) = 0 # Path search
open("/proc/self/mountinfo", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "13 0 8:2 / / rw,relatime - ext4 "..., 1024) = 638
# That's the right root file system
readlink("/proc/self", "2690", 4095) = 4
lstat64("/proc/2690/mountinfo", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
# Not sure what's being reverified here
open("/proc/cmdline", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
read(5, "auto BOOT_IMAGE=Linux ro root=802 "..., 1024) = 155
close(5) = 0
getcwd("$HOME", 4096)
# No, not literally, I've substituted
lstat64("$HOME/802", ...) = -1 ENOENT
# This looks wrong
read(3, "", 1024) = 0
close(3) = 0
... then it clone()s and tries to run fsck.ext4.
I'd think the process of elimination would go as follows:
- For each line in /etc/fstab
- If it's a real file system for which we have an fsck, stat
the device node.
- Search /proc/self/mountinfo for the same device mode. If it's
mounted already (rw or ro), skip it.
Poking around the code, I also found some debug options.
Trying LIBMOUNT_DEBUG=all LIBBLKID_DEBUG=all fsck -C -M -A
produces:
7320: libmount: INIT: library debug mask: 0xffff
7320: libmount: INIT: library version: 2.27.0
7320: libmount: INIT: feature: selinux
7320: libmount: INIT: feature: assert
7320: libmount: INIT: feature: debug
Available "LIBMOUNT_DEBUG=<name>[,...]|<mask>" debug masks:
all [0xffff] : info about all subsystems
cache [0x0004] : paths and tags cache
cxt [0x0200] : library context (handler)
diff [0x0400] : mountinfo changes tracking
fs [0x0040] : FS abstraction
help [0x0001] : this help
locks [0x0010] : mtab and utab locking
options [0x0008] : mount options parsing
tab [0x0020] : fstab, mtab, mounninfo routines
update [0x0080] : mtab, utab updates
utils [0x0100] : misc library utils
monitor [0x0800] : mount tables monitor
7320: libmount: CACHE: [0x9480048]: alloc
fsck from util-linux 2.27.1
7320: libmount: TAB: [0x9480068]: alloc
7320: libmount: TAB: [0x9480068]: /etc/fstab: start parsing [entries=0,
filter=not]
7320: libmount: TAB: [0x9480068]: add entry: /dev/sda2 /
7320: libmount: TAB: [0x9480068]: add entry: UUID=[swap device] none
7320: libmount: TAB: [0x9480068]: add entry: proc /proc
7320: libmount: TAB: [0x9480068]: add entry: sys /sys
7320: libmount: TAB: [0x9480068]: add entry: dev /dev
7320: libmount: TAB: [0x9480068]: add entry: devpts /dev/pts
7320: libmount: TAB: [0x9480068]: add entry: [network mount redacted]
7320: libmount: TAB: [0x9480068]: add entry: /dev/sr0 /media/cdrom0
7320: libmount: TAB: [0x9480068]: add entry: /dev/mmcblk0p1 /mnt
7320: libmount: FS: [0x9480b30]: free [refcount=0]
7320: libmount: TAB: [0x9480068]: /etc/fstab: stop parsing (9 entries)
7320: libmount: TAB: [0x9480068]: parsing done [filename=/etc/fstab, rc=0]
7320: libmount: CACHE: [0x9480048]: canonicalize path /dev/sda2
7320: libmount: CACHE: [0x9480048]: add entry [ 1] (path): /dev/sda2:
/dev/sda2
7320: libmount: TAB: [0x9480068]: lookup TARGET: '/'
7320: libmount: TAB: [0x9480130]: alloc
7320: libmount: TAB: [0x9480130]: mtab parse: ignore mtab
7320: libmount: TAB: [0x9480130]: mtab parse: #1 read mountinfo
7320: libmount: TAB: [0x9480130]: /proc/self/mountinfo: start parsing
[entries=0, filter=not]
7320: libmount: TAB: [0x9480130]: add entry: /dev/root /
7320: libmount: CACHE: canonicalize path /proc/self/mountinfo
7320: libmount: TAB: TID for /proc/self/mountinfo is 7320
7320: libmount: TAB: [0x9480130]: root FS: 802
7320: libmount: CACHE: [0x9480048]: canonicalize path 802
7320: libmount: CACHE: [0x9480048]: add entry [ 2] (path): 802: 802
7320: libmount: TAB: [0x9480130]: canonical root FS: 802
7320: libmount: TAB: [0x9480130]: add entry: sysfs /sys
7320: libmount: TAB: [0x9480130]: add entry: tmpfs /run
7320: libmount: TAB: [0x9480130]: add entry: tmpfs /run/lock
7320: libmount: TAB: [0x9480130]: add entry: proc /proc
7320: libmount: TAB: [0x9480130]: add entry: devtmpfs /dev
7320: libmount: TAB: [0x9480130]: add entry: tmpfs /run/shm
7320: libmount: TAB: [0x9480130]: add entry: devpts /dev/pts
7320: libmount: FS: [0x9482ca8]: free [refcount=0]
7320: libmount: TAB: [0x9480130]: /proc/self/mountinfo: stop parsing (8
entries)
7320: libmount: TAB: [0x9480130]: parsing done
[filename=/proc/self/mountinfo, rc=0]
7320: libmount: TAB: [0x9480130]: mtab parse: #2 read utab
7320: libmount: TAB: [0x9480130]: lookup SOURCE: '/dev/sda2'
7320: libmount: TAB: [0x9480130]: lookup SRCPATH: '/dev/sda2'
7320: libmount: TAB: [0x9480130]: lookup canonical SRCPATH: '/dev/sda2'
7321: libblkid: INIT: library debug mask: 0xffff
7321: libblkid: INIT: library version: 2.27.0 [02-Nov-2015]
Available "LIBBLKID_DEBUG=<name>[,...]|<mask>" debug masks:
all [0xffff] : info about all subsystems
cache [0x0004] : blkid tags cache
config [0x0008] : config file utils
dev [0x0010] : device utils
devname [0x0020] : /proc/partitions evaluation
devno [0x0040] : convertions to device name
evaluate [0x0080] : tags resolving
help [0x0001] : this help
lowprobe [0x0100] : superblock/raids/partitions probing
probe [0x0200] : devices verification
read [0x0400] : cache parsing
save [0x0800] : cache writing
tag [0x1000] : tags utils
7321: libblkid: CACHE: creating blkid cache (using default cache)
7321: libblkid: CONFIG: reading config file: /etc/blkid.conf.
7321: libblkid: CONFIG: /etc/blkid.conf: does not exist, using built-in
default
7321: libblkid: TAG: looking for /dev/sda2 in cache
e2fsck 1.43-WIP (18-May-2015)
/dev/sda2 is mounted.
e2fsck: Cannot continue, aborting.
There are a lot of wrappers in libmount (not to meantion a dearth of
useful contents in libmount/docs/) which make it hard for me to be sure
what things *should* be doing, so I haven't pinpointed where it's going
wrong, but hopefully this information is useful.
