Public bug reported:
[ Impact ]
Sometimes on a system that is mounting and unmounting filesystems frequently,
for example running lots of docker containers, the size of /proc/1/mountinfo,
can become very large -- 100s, to 1000s of entries or more -- with the vast
majority being a single entry duplicated many times.
This causes other problems on the system, due to systemd parsing the mount table
whenever it changes, and eating up a lot of memory, for example [0]. Waiting
long enough there are rare events where the length of mountinfo can go into the
millions of lines and lead to OOM and kernel panics.
[Fix]
Christian Brauner submitted a patch on the mailing list[1] (now merged upstream
[2])
which fixes the issue by
a) using the unique mount ID as the pos for iterating the mounts
b) updating to the pos of the iterator before returning
[ Test Plan ]
To test whether this issue occurs, it is sufficient to rapidly mount and unmount
tmpfs rapidly and poll for duplicates in `/proc/1/mountinfo` this can be done
for example by the following pair of scripts:
repro.sh
--------
#!/bin/bash
counter=0
while true; do
echo -n "."
unique_name="tmpfs_$$_$counter"
mkdir -p "/tmp/$unique_name"
sudo mount -t tmpfs "$unique_name" "/tmp/$unique_name"
sudo umount "/tmp/$unique_name"
rmdir "/tmp/$unique_name"
((counter++))
sleep 0.1
done
-------
has-bug.sh
--------
#!/bin/bash
THRESHOLD=100
WAIT_MIN=30
WAIT_SECONDS=$((WAIT_MIN * 60))
SECONDS=0
while ((SECONDS < WAIT_SECONDS)); do
# Get mountinfo entries and count total
mountinfo="$(cat /proc/1/mountinfo)"
mountinfo_count=$(echo "$mountinfo" | wc -l)
if ((mountinfo_count > THRESHOLD)); then
echo "$(date): Mount count ($mountinfo_count) exceeds threshold
($THRESHOLD)"
# Find and log duplicate mount points with their counts
duplicates=$(echo "$mountinfo" | sort | uniq -cd)
if [[ -n "$duplicates" ]]; then
echo "Duplicate mounts :"
echo "$duplicates" | while read -r count mountpoint; do
echo " $mountpoint: $count occurrences"
done
fi
exit 0
fi
sleep 0.1
done
exit 1
------
In my testing, about 5 minutes is sufficient time for the bug to occur, so I've
set the timeout to 30min in the test plan out of an abundance of caution.
If the bug is present in the running kernel, `has-bug.sh` will print a message
about duplicate entries and exit success, otherwise it will time out and exit
with return code 1.
The high level test plan is:
1. Run `has-bug.sh` and `repro.sh` on unpatched kernel.
2. Observe that bug is present
3. Upgrade to patched kernel and restart
4. Run `has-bug.sh` and `repro.sh` on unpatched kernel.
5. Observe that bug is no longer present
[ Where problems could occur ]
* The patch to fix this modifies how iterator position is tracked when iterating
mounts. It therefore potentially affects anything that iterates through the
mounts.
[ Other Info ]
[0]: https://github.com/systemd/systemd/issues/37939
[1]:
https://lore.kernel.org/lkml/20260129-geleckt-treuhand-4bb940acacd9@brauner/
[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4a403d7aa9074f527f064ef0806aaab38d14b07c
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Affects: linux (Ubuntu Noble)
Importance: Undecided
Status: New
** Affects: linux (Ubuntu Questing)
Importance: Undecided
Status: New
** Affects: linux (Ubuntu Resolute)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Questing)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Resolute)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Noble)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2143083
Title:
[SRU] Duplicated entries in /proc/<pid>/mountinfo
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2143083/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
