[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
This problem seems very similar https://bugzilla.redhat.com/show_bug.cgi?id=1416532 ** Bug watch added: Red Hat Bugzilla #1416532 https://bugzilla.redhat.com/show_bug.cgi?id=1416532 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Confirmed Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 Package: linux (not installed) PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKern
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
apport information ** Tags added: apport-collected ** Description changed: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Nov 8 03:13 seq + crw-rw 1 root audio 116, 33 Nov 8 03:13 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.20.1-0ubuntu2.1 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 16.04 + HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 + InstallationDate: Installed on 2016-11-07 (3 days ago) + InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) + IwConfig: Error: [Errno 2] No such file or directory + Lsusb: + Bus 002 Device 002: ID 8087:8002 Intel Corp. + Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub + Bus 001 Device 002: ID 8087:800a Intel Corp. + Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + MachineType: Dell Inc. PowerEdge R630 + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + LC_CTYPE=en_US.UTF-8 + TERM=screen-256color + PATH=(custom, no user) + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 EFI VGA + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro + ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 + RelatedPackageVersions: + linux-restricted-modules-4.4.0-45-generic N/A + linux-backports-modules-4.4.0-45-generic N/A + linux-firmware1.157.4 + RfKill: Error: [Errno 2] No such file or d
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
All information should be added now. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Confirmed Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 Package: linux (not installed) PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e6
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.9 kernel[0]. If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'. If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'. Once testing of the upstream kernel is complete, please mark this bug as "Confirmed". Thanks in advance. [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9-rc5 ** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Tags added: kernel-da-key ** Tags added: needs-bisect ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Incomplete Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
We are testing this with different kernel versions right now. I would like to report that NFSv3 workaround does not work with symlinks (the bug still appears), but seems to work with regular files. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Incomplete Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 Package: linux (not installed) PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Confirmed Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 Package: linux (not installed) PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
I was able to reproduce this bug in a bit different manner in vagrant VM
environment.
We were using 16.04 as nfs server and 14.04, 16.04 (with latest kernel
and 4.9-rc6) and 16.10 as clients.
For the testing we were using the script in the end of this post.
Scenario is:
* create 1k symlinks by A on share
* ls that directory
* delete symlinks on A
* create them again on A
* ls the directory again by B
* delete everything
repeat 10 times
Kernels 4.4 (from 16.04) AND 4.8 (from 16.10) have errors like
ls: cannot read symbolic link '/mnt/ubuntu/347': Input/output error
ls: cannot read symbolic link '/mnt/ubuntu/891': Input/output error
ls: cannot read symbolic link '/mnt/ubuntu/248': Input/output error
ls: cannot read symbolic link '/mnt/ubuntu/872': Input/output error
starting from the second ls.
Kernels 3.13 (from 14.04) and upstream 4.9-rc6 do not have these errors.
If you want vagrant files for vms please tell.
Testing script:
#!/bin/zsh
a=9e0147f
b=7329576
function exec_a {
vagrant ssh -c "/bin/bash -c ${(q)*}" $a -- -q
}
function exec_b {
vagrant ssh -c "/bin/bash -c ${(q)*}" $b -- -q
}
SHARE_ROOT=/mnt/ubuntu
function iteration {
exec_a "for i in {0..1000}; do ln -s /dev/null $SHARE_ROOT/\$i ; done"
exec_b "ls -li $SHARE_ROOT | wc -l | paste <(echo \"first ls\") - "
exec_a "find $SHARE_ROOT -type l -delete"
exec_a "for i in {0..1000}; do ln -s /dev/null $SHARE_ROOT/\$i ; done"
exec_b "ls -li $SHARE_ROOT | wc -l | paste <(echo \"second ls\") - "
exec_a "find $SHARE_ROOT -type l -delete"
}
exec_a "find $SHARE_ROOT -type l -delete"
for i in {0..10}; do iteration; done
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1641049
Title:
NFSv4 do not invalidate cached information about deleted files
Status in linux package in Ubuntu:
Confirmed
Bug description:
When using NFSv4 client in Ubuntu 16.04 the the following sequence of
actions fails.
You need a NFS server (S) and two clients (A and B).
Let the NFS share be mounted at /nfsdata
A: echo test > /nfsdata/file
B: cat /nfsdata/file ===> test
A: rm /nfsdata/file
B: cat /nfsdata/file ===> cat: No such file or directory
A: echo test > /nfsdata/file
B: cat /nfsdata/file ===> cat: No such file or directory (!)
Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible
on B.
14.04 works without any problem.
I have attached log from one of our servers, 4.4.0-47 kernel does not fix
this issue.
Syslog contains rpcdebug -m nfs -s all output for the scenario (with
different filenames).
I can do full packet capture of NFS traffic if you want.
Mounting nfs with lookupcache=positive do not fix this issue.
NFSv4 server is CentOS 6.7 in our case.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-45-generic 4.4.0-45.66
ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21
Uname: Linux 4.4.0-45-generic x86_64
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Nov 8 03:13 seq
crw-rw 1 root audio 116, 33 Nov 8 03:13 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
Date: Fri Nov 11 17:06:51 2016
HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1
InstallationDate: Installed on 2016-11-07 (3 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64
(20160719)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 002 Device 002: ID 8087:8002 Intel Corp.
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub
Bus 001 Device 002: ID 8087:800a Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge R630
PciMultimedia:
ProcEnviron:
LC_CTYPE=en_US.UTF-8
TERM=screen-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed
root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro
RelatedPackageVersions:
linux-restricted-modules-4.4.0-45-generic N/A
linux-backports-modules-4.4.0-45-generic N/A
linux-firmware1.157.4
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/12/2016
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.0.1
dmi.board.name: 02C2CP
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias:
dmi:bv
[Kernel-packages] [Bug 1641049] Re: NFSv4 do not invalidate cached information about deleted files
We can perform a "Reverse" bisect to identify the commit that fixes this in v4.9-rc6. We first need to identify the last bad kernel version and the first good one. Does v4.9-rc5 exhibit the bug? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1641049 Title: NFSv4 do not invalidate cached information about deleted files Status in linux package in Ubuntu: Confirmed Bug description: When using NFSv4 client in Ubuntu 16.04 the the following sequence of actions fails. You need a NFS server (S) and two clients (A and B). Let the NFS share be mounted at /nfsdata A: echo test > /nfsdata/file B: cat /nfsdata/file ===> test A: rm /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory A: echo test > /nfsdata/file B: cat /nfsdata/file ===> cat: No such file or directory (!) Doing echo 3 > proc/sys/vm/drop_caches as root makes the file visible on B. 14.04 works without any problem. I have attached log from one of our servers, 4.4.0-47 kernel does not fix this issue. Syslog contains rpcdebug -m nfs -s all output for the scenario (with different filenames). I can do full packet capture of NFS traffic if you want. Mounting nfs with lookupcache=positive do not fix this issue. NFSv4 server is CentOS 6.7 in our case. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-45-generic 4.4.0-45.66 ProcVersionSignature: Ubuntu 4.4.0-45.66-generic 4.4.21 Uname: Linux 4.4.0-45-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Fri Nov 11 17:06:51 2016 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-45-generic.efi.signed root=UUID=11e697e5-38f8-41ae-863c-2755793044a6 ro RelatedPackageVersions: linux-restricted-modules-4.4.0-45-generic N/A linux-backports-modules-4.4.0-45-generic N/A linux-firmware1.157.4 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 02/12/2016 dmi.bios.vendor: Dell Inc. dmi.bios.version: 2.0.1 dmi.board.name: 02C2CP dmi.board.vendor: Dell Inc. dmi.board.version: A01 dmi.chassis.type: 23 dmi.chassis.vendor: Dell Inc. dmi.modalias: dmi:bvnDellInc.:bvr2.0.1:bd02/12/2016:svnDellInc.:pnPowerEdgeR630:pvr:rvnDellInc.:rn02C2CP:rvrA01:cvnDellInc.:ct23:cvr: dmi.product.name: PowerEdge R630 dmi.sys.vendor: Dell Inc. --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Nov 8 03:13 seq crw-rw 1 root audio 116, 33 Nov 8 03:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=7b92d2e2-e481-471a-bc38-e178a9418aa1 InstallationDate: Installed on 2016-11-07 (3 days ago) InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719) IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:8002 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:800a Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R630 Package: linux (not installed) PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 EFI VGA
