Public bug reported:
Apologies if this is overly verbose, I had Claude help write up this
report. Nevertheless the performance difference is real.
[Impact]
LP #2022098 ("Severe NFS performance degradation after LP #2003053") was fixed
in the
Ubuntu 5.15 kernel by:
SAUCE: Make NFS file-access stale cache behaviour opt-in
which added the `nfs_fasc` module parameter (bool, 0644, default false), gating
the
per-login NFS access-cache invalidation backported for LP #2003053:
0eb43812c027 NFS: Clear the file access cache upon login
(v6.2-rc1)
029085b8949f NFS: Judge the file access cache's timestamp in rcu path
(v6.2-rc3)
5e9a7b9c2ea1 NFS: Fix up a sparse warning
(v6.2-rc3)
That SAUCE patch was not carried forward. Ubuntu 5.19 and later retain the
invalidation with no opt-in — `fs/nfs/dir.c` in linux-aws-5.19
5.19.0-1029.30~22.04.1
contains no reference to `nfs_fasc` at all — so the regression closed by LP
#2022098
has returned for everyone on a newer kernel.
Why this hurts: the invalidation derives a "login time" by walking
`real_parent` while
credentials match and taking that ancestor's `start_time`. sshd forks per
connection
and switches credentials to the connecting user; the parent is the sshd monitor
running
as root, so the walk stops at the worker itself and "login time" becomes the
worker's
own `start_time`. Every access-cache entry left by any previous connection
therefore
predates "login" and is rejected. For any fork-per-connection daemon reading
files from
NFS — sshd with AuthorizedKeysFile on an NFS home is the obvious case — the
cache is
never reused across connections.
Measured on one host, same kernel, userspace byte-identical, 20
sequential SFTP logins:
nfs_fasc=N nfs_fasc=Y
ACCESS RPCs 13 280
login median @1 143.2 ms 197.9 ms
login median @5 165.0 ms 217.5 ms
throughput @5 29.1/s 25.4/s
Ubuntu 5.19 on the same host measures 196.9 ms @1 and 216.5 ms @5 — matching
`nfs_fasc=Y` rather than `nfs_fasc=N`, which confirms both the cause and that
the
existing parameter is the fix.
Impact scales with connection rate, not session size: the cost is per connection
(~15 redundant ACCESS RPCs), and it does not vary with how many files a session
touches. A workload of frequent short reconnections never amortises it.
Confirmed affected across every Ubuntu kernel after 5.15. The ACCESS RPC count
per
login is a version-independent signature -- exactly 14.00 wherever the
invalidation is
active, and under 1 wherever it is not:
distro kernel ACCESS/login login median @1 nfs_fasc
22.04 5.15 (fasc=N) 0.65 143.2 ms present, off
22.04 5.15 (fasc=Y) 14.00 197.9 ms present, on
22.04 5.19 14.00 196.9 ms ABSENT
24.04 6.8.0-1061-aws 14.00 211.0 ms ABSENT
26.04 7.0.0-1006-aws 14.00 154.5 ms ABSENT
`/sys/module/nfs/parameters/nfs_fasc` does not exist on 5.19, 6.8 or 7.0 --
verified
with the nfs module loaded on each. Also affected: 6.2 and 6.5 (jammy HWE), and
the
corresponding linux-aws variants.
All hosts above are c6i.xlarge in the same availability zone as the load
generator, with
CPU steal measured at zero.
The invalidation cost does not diminish in newer kernels. Booting the *same*
24.04
userspace on 6.8 and then on 7.0 gives an identical result -- 211 ms and 14.00
ACCESS
RPCs per login either way, with the auth phase at 63.1 ms in both cases.
26.04's lower
total is therefore attributable to its userspace, not to any change in the
kernel
behaviour: on kernel 7.0, 24.04 userspace measures 211 ms and 26.04 userspace
measures
154.5 ms, while both issue the same 14.00 redundant ACCESS RPCs.
Note that 26.04/7.0 shows a much lower total than 24.04/6.8 despite an
identical ACCESS
count of 14.00. That is userspace, not the kernel -- its subsystem-startup
phase measures
34.4 ms against roughly 74 ms on 22.04 and 24.04. It does not indicate the
kernel issue
is absent or reduced on 7.0: the redundant RPC count is unchanged, and the
wasted server
round trips are identical. The RPC count is therefore the metric of record
here, because
latency totals move with the OpenSSH version and distro while 14.00 ACCESS RPCs
per login
holds across every affected kernel from 5.15 through 7.0.
[Test Plan]
On a host with an NFS-mounted home directory and sshd configured with
AuthorizedKeysFile pointing into it:
1. Count NFS ACCESS RPCs per login. Read the per-op "ACCESS" counter for the
mount
from /proc/self/mountstats before and after 20 sequential SFTP logins:
sftp -o BatchMode=yes -i <key> -b /dev/null <user>@<host>
2. On a 5.15 kernel, repeat with the parameter both ways:
echo N | sudo tee /sys/module/nfs/parameters/nfs_fasc # expect ~13 per
20 logins
echo Y | sudo tee /sys/module/nfs/parameters/nfs_fasc # expect ~280 per
20 logins
3. On 5.19 or later, the count matches the `nfs_fasc=Y` case and there is no
parameter
to change.
Expected after fix: `/sys/module/nfs/parameters/nfs_fasc` exists on 5.19+ and
defaults
to N, and the ACCESS count per login returns to the 5.15 default behaviour.
A self-contained reproducer needing no cloud provider, using a loopback knfsd
export,
is included in [Other Info].
[Where problems could occur]
The change restores an existing, already-shipped Ubuntu patch and its default is
`false`, i.e. the pre-LP-#2003053 upstream behaviour. Risk is therefore limited
to:
- Anyone relying on the LP #2003053 behaviour (server-side supplementary group
changes
clearing on a fresh login) would need to set `nfs_fasc=Y` explicitly. This is
the
same trade-off already accepted for 5.15, where the default is off.
- The parameter is writable at runtime (0644), so a wrong setting is
correctable without
a reboot and cannot render a system unbootable.
- The patch touches only fs/nfs/dir.c and adds no new code paths, only
conditionals
around existing ones.
[Other Info]
Upstream rationale, from 0eb43812c027, states the trade-off explicitly:
"While it is reasonable to expect that such group membership changes are
rare, and
that we do not want to optimise the cache to accommodate them, it is also not
unreasonable for the user to expect that if they log out and log back in
again, that
the staleness would clear up."
Note there is no upstream opt-out: `nfs_fasc` is Ubuntu SAUCE and was never
submitted
to linux-nfs. A separate upstream request for a mount option or module
parameter is
warranted, but that does not help currently-supported Ubuntu kernels.
Related upstream commits:
21fd9e8700de NFS: Correct timing for assigning access cache timestamp
(v6.3-rc4)
a10a9233073d NFS: Use parent's objective cred in nfs_access_login_time()
(v6.8-rc1)
Self-contained reproducer, single host, loopback knfsd:
# server
mkdir -p /srv/nfstest/a/b/c/d/.ssh
useradd -u 3028 -M -s /sbin/nologin testu
head -c 6340 /dev/zero | tr '\0' 'k' >
/srv/nfstest/a/b/c/d/.ssh/authorized_keys
chown testu /srv/nfstest/a/b/c/d/.ssh{,/authorized_keys}
chmod 770 /srv/nfstest/a/b/c/d/.ssh
chmod 660 /srv/nfstest/a/b/c/d/.ssh/authorized_keys
echo "/srv/nfstest 127.0.0.1(ro,sync,no_subtree_check,no_root_squash)" >
/etc/exports
systemctl enable --now nfs-server; exportfs -ra
# client -- do NOT use noresvport; knfsd's default `secure` rejects
# unprivileged source ports
mount -t nfs4 -o ro,nfsvers=4.1 127.0.0.1:/srv/nfstest /mnt/t
F=/mnt/t/a/b/c/d/.ssh/authorized_keys
# compare the per-op ACCESS counter for /mnt/t in /proc/self/mountstats
# around each case:
# 20 opens, one per fresh process
for i in $(seq 1 20); do sudo -u testu python3 -c \
'import os,sys; fd=os.open(sys.argv[1],os.O_RDONLY); os.close(fd)' "$F";
done
# the same 20 opens inside a single process
sudo -u testu python3 -c \
'import os,sys
for _ in range(20): fd=os.open(sys.argv[1],os.O_RDONLY); os.close(fd)' "$F"
ACCESS RPCs per 20 opens, one open per fresh process:
5.15 (nfs_fasc=N) 6
5.19 120
6.2 139
6.8 140
Twenty opens inside a single process stay at 6-7 on every kernel, because
`start_time`
is fixed within a process — confirming the invalidation is per-process, not
per-open.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: linux-image-7.0.0-1006-aws 7.0.0-1006.6
ProcVersionSignature: Ubuntu 7.0.0-1006.6-aws 7.0.0
Uname: Linux 7.0.0-1006-aws x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2:
ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.34.0-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: N/A
CasperMD5CheckResult: unknown
CloudArchitecture: x86_64
CloudBuildName: server
CloudID: aws
CloudName: aws
CloudPlatform: ec2
CloudRegion: us-east-1
CloudSerial: 20260604
CloudSubPlatform: metadata (http://169.254.169.254)
Date: Tue Aug 4 21:35:07 2026
Ec2Architecture: x86_64
Ec2Imageid: ami-0b6d9d3d33ba97d99
Ec2Instancetype: c6i.xlarge
Ec2Region: us-east-1
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Amazon EC2 c6i.xlarge
PciMultimedia:
ProcEnviron:
LANG=C.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=xterm-256color
ProcFB: 0 simpledrmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-7.0.0-1006-aws
root=PARTUUID=0258290b-29ad-4f2b-9001-018bec7b8627 ro console=tty1
console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1
RebootRequiredPkgs: Error: path contained symlinks.
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux-aws
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/16/2017
dmi.bios.release: 1.0
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-02adefe268a5c8d71
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias:
dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:br1.0:svnAmazonEC2:pnc6i.xlarge:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:sku:pfa:
dmi.product.name: c6i.xlarge
dmi.sys.vendor: Amazon EC2
** Affects: linux-aws (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug cloud-image nfs resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2162869
Title:
NFS access cache regression returns on 5.19+ — SAUCE "Make NFS file-
access stale cache behaviour opt-in" (LP #2022098) was not forward-
ported
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2162869/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
