Hi Du ChengEn,
Thanks for the clarification on test kernels. I am sad, I did not get to
test your 2nd test kernel, as that would have revealed this issue
immediately.
While the changes to fix LP: #2003053, went in there to fix a "bug" - it
also created this major regression. But the bug fix just tries to get
around the metadata inconsistencies, that has always been an issue with
NFS (by design). Not a hard case IHMO, and also "rare" as pointed out in
the original commit message.
I follow Jan here, that these changes should be reverted, as they create
more problems than they tried to fix.
That does not stop anybody from working on an upstream solution, that
would fit both purposes.
For whatever it is worth, this reproducer points out the pain point:
$ cd /nfsdir
$ touch myfiles.{1..1000}
$ md5sum myfiles.{1..1000} > /dev/null
$ sudo -u <myself> md5sum myfiles.{1..1000} > /dev/null
The latter 'md5sum' command produces 1000 GETATTR and 1000 ACCESS calls.
The GEATTRs are there to ensure close-to-open consistency - fine.
However, ACCESS calls should not be produced in this case, which shows
the issue.
This is even more outspoken, if this is mounted using 'nocto' (as we
do). The GETATTRs are gone (as they should). The ACCESS calls, however
remain. This is where we get that huge increase in numbers.
With regards to potential increase in LOOKUPs - this is still to early.
Caches are still not settled (lightly loaded system)
So, where do we go from here - create a new bug ? Give feedback on the
linux-nfs mail list ?
I can see you already made a post to the mail list - I will be happy to
be of assistance, if you see fit.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/2009325
Title:
NFS deathlock with last Kernel 5.4.0-144.161 and 5.15.0-67.74
Status in linux package in Ubuntu:
In Progress
Status in linux-aws package in Ubuntu:
Invalid
Status in linux source package in Bionic:
Fix Released
Status in linux-aws source package in Bionic:
Fix Released
Status in linux source package in Focal:
Fix Released
Status in linux-aws source package in Focal:
Fix Released
Status in linux source package in Jammy:
Fix Released
Status in linux-aws source package in Jammy:
Fix Released
Status in linux source package in Kinetic:
Fix Released
Status in linux-aws source package in Kinetic:
Fix Released
Status in linux source package in Lunar:
In Progress
Status in linux-aws source package in Lunar:
Invalid
Bug description:
After updating on the kernel
5.4.0-144.161 at Ubuntu 18 and
5.15.0-67.74 at Ubuntu 20,
we have a 100% CPU outlation and 20 to 30 Mbit traffic to the clients for our
NFS servers.
All clients are extremely slow when it comes to access to the NFS
resources.
Restart and use older kernel, fixed the problem.
Ubuntu 18 5.4.0-139-generic
Ubuntu 20 5.15.0-60-Generic
I don't have a NFS problem with this kernel.
Problem came with the last releas on March 3rd, 2023
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Mär 4 15:00 seq
crw-rw---- 1 root audio 116, 33 Mär 4 15:00 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: VMware, Inc. VMware Virtual Platform
Package: linux (not installed)
PciMultimedia:
ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-67-generic
root=/dev/mapper/vg1-root ro net.ifnames=0 biosdevname=0 kvm.nx_huge_pages=auto
elevator=noop
ProcVersionSignature: Ubuntu 5.15.0-67.74~20.04.1-generic 5.15.85
RelatedPackageVersions:
linux-restricted-modules-5.15.0-67-generic N/A
linux-backports-modules-5.15.0-67-generic N/A
linux-firmware 1.187.36
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal
Uname: Linux 5.15.0-67-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 11/12/2020
dmi.bios.release: 4.6
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.ec.firmware.release: 0.0
dmi.modalias:
dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd11/12/2020:br4.6:efr0.0:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:sku:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Mär 4 15:03 seq
crw-rw---- 1 root audio 116, 33 Mär 4 15:03 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.28
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord':
'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: VMware, Inc. VMware Virtual Platform
Package: linux (not installed)
PciMultimedia:
ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.0.0-43-generic
root=/dev/mapper/vg1-root ro net.ifnames=0 biosdevname=0 kvm.nx_huge_pages=auto
elevator=noop
ProcVersionSignature: Ubuntu 5.0.0-43.47~18.04.1-generic 5.0.21
RelatedPackageVersions:
linux-restricted-modules-5.0.0-43-generic N/A
linux-backports-modules-5.0.0-43-generic N/A
linux-firmware 1.173.21
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic
Uname: Linux 5.0.0-43-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 12/12/2018
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias:
dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd12/12/2018:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
