= focal verification = ubuntu@client:~$ cat /proc/version_signature Ubuntu 5.4.0-102.115-generic 5.4.174 ubuntu@client:~$ ./elbencho/bin/elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu OPERATION RESULT TYPE FIRST DONE LAST DONE ========= ================ ========== ========= READ Elapsed ms : 470642 471169 Files/s : 4245 4244 Throughput MiB/s : 530 530 Total files : 1998154 2000000 Total MiB : 249770 250000 --- ubuntu@client:~$
Regression: ubuntu@client:~/nfstest$ PYTHONPATH=$(pwd) ./test/nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e /srv/nfstest --trcdelay 4 UID is: 1000 Creating: /mnt/t/nfstest_delegation_20220225_223841_f_001 UID is: 1000 Creating: /mnt/t/nfstest_delegation_20220225_223841_f_002 *** Basic READ delegation test TEST: Running test 'basic01' PASS: Open file for READ should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: READ delegation should be granted TIME: 5.113318s *** Basic WRITE delegation test TEST: Running test 'basic02' PASS: Open file for WRITE should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for WRITE on same process should succeed PASS: Write file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: Open file for WRITE from a different process should succeed PASS: Write file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.940012s *** Basic READ delegation test with file stat TEST: Running test 'basic03' PASS: Stat file to cache file metadata should succeed PASS: Open file for READ should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: LOOKUP operation should be sent PASS: OPEN should be sent PASS: OPEN should be sent with CLAIM_FH PASS: OPEN should be sent with the filehandle of the file to be opened PASS: READ delegation should be granted PASS: OPEN should not be sent for the same file PASS: READs should be sent to the server with the DELEG stateid PASS: READs should return NFS4_OK from the server PASS: READ should not be sent when reading delegated file from a different process PASS: CLOSE should be sent to the server PASS: CLOSE should be sent with correct OPEN stateid PASS: DELEGRETURN should be sent after the close PASS: DELEGRETURN should be sent with the delegation stateid TIME: 4.757017s *** Basic WRITE delegation test with file stat TEST: Running test 'basic04' UID is: 1000 Creating: /mnt/t/nfstest_delegation_20220225_223841_f_004 PASS: Stat file to cache file metadata should succeed PASS: Open file for WRITE should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for WRITE on same process should succeed PASS: Write file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: Open file for WRITE from a different process should succeed PASS: Write file from a different process should succeed PASS: LOOKUP operation should be sent FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_FH FAIL: OPEN should be sent with the filehandle of the file to be opened FAIL: WRITE delegation should be granted TIME: 5.090116s *** Basic READ delegation test with file lock TEST: Running test 'basic05' PASS: Open file for READ should succeed PASS: Lock file with F_RDLCK should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory PASS: READ delegation should be granted PASS: OPEN should not be sent for the same file PASS: LOCK should not be sent to the server PASS: READs should be sent to the server with the DELEG stateid PASS: READs should return NFS4_OK from the server PASS: READ should not be sent when reading delegated file from a different process PASS: CLOSE should be sent to the server PASS: CLOSE should be sent with correct OPEN stateid PASS: DELEGRETURN should be sent after the close PASS: DELEGRETURN should be sent with the delegation stateid TIME: 4.794486s *** Basic WRITE delegation test with file lock TEST: Running test 'basic06' PASS: Open file for WRITE should succeed PASS: Lock file with F_WRLCK should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for WRITE on same process should succeed PASS: Write file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: Open file for WRITE from a different process should succeed PASS: Write file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.962986s *** Basic WRITE delegation test using RDWR open while reading TEST: Running test 'basic07' PASS: Open file for RDWR should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.806131s *** Basic WRITE delegation test using RDWR open while writing TEST: Running test 'basic08' PASS: Open file for RDWR should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.711963s *** Basic WRITE delegation test using RDWR open while reading with file stat TEST: Running test 'basic09' PASS: Stat file to cache file metadata should succeed PASS: Open file for RDWR should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: LOOKUP operation should be sent FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_FH FAIL: OPEN should be sent with the filehandle of the file to be opened FAIL: WRITE delegation should be granted TIME: 4.749760s *** Basic WRITE delegation test using RDWR open while writing with file stat TEST: Running test 'basic10' UID is: 1000 Creating: /mnt/t/nfstest_delegation_20220225_223841_f_007 PASS: Stat file to cache file metadata should succeed PASS: Open file for RDWR should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed PASS: LOOKUP operation should be sent FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_FH FAIL: OPEN should be sent with the filehandle of the file to be opened FAIL: WRITE delegation should be granted TIME: 4.869004s *** Basic WRITE delegation test using RDWR open while reading with file lock TEST: Running test 'basic11' PASS: Open file for RDWR should succeed PASS: Lock file with F_RDLCK should succeed PASS: Read file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.766863s *** Basic WRITE delegation test using RDWR open while writing with file lock TEST: Running test 'basic12' PASS: Open file for RDWR should succeed PASS: Lock file with F_WRLCK should succeed PASS: Write file should succeed PASS: Open file for READ on same process should succeed PASS: Read file on same process should succeed PASS: Open file for READ from a different process should succeed PASS: Read file from a different process should succeed FAIL: OPEN should be sent PASS: OPEN should be sent with CLAIM_NULL PASS: OPEN should be sent with the name of the file to be opened PASS: OPEN should be sent with the filehandle of the directory FAIL: WRITE delegation should be granted TIME: 4.725836s 169 tests (146 passed, 23 failed) Total time: 58.446193s ubuntu@client:~/nfstest$ ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1957986 Title: Recalled NFSv4 files delegations overwhelm server Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Status in linux source package in Hirsute: Fix Released Status in linux source package in Impish: Fix Released Status in linux source package in Jammy: Fix Released Bug description: [Impact] An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 server, causing a significant drop in performance. In my testing, I've seen performance drop from ~700MiB/s down to < 10MiB/s. The same workload using NFSv3 does not have this problem. [Test Case] This can be demonstrated using the elbencho benchmark from https://github.com/breuner/elbencho: $ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu You'll notice the nfsd threads (I stuck w/ the default of 4) start to consume 100% CPU, and the performance of the elbencho benchmark will begin to trickle. [Fix] The following fix solves the problem, but there are a number of patches dependencies required before it will apply to focal: commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad) Author: Trond Myklebust <tron...@gmail.com> Date: Mon Jan 27 09:58:19 2020 -0500 NFSv4: Limit the total number of cached delegations Delegations can be expensive to return, and can cause scalability issues for the server. Let's therefore try to limit the number of inactive delegations we hold. Once the number of delegations is above a certain threshold, start to return them on close. Signed-off-by: Trond Myklebust <trond.mykleb...@hammerspace.com> Signed-off-by: Anna Schumaker <anna.schuma...@netapp.com> [What could go wrong] The fixes are restricted to NFS code, so problems should be limited to NFS users. They could include performance issues, crashes, etc. Because these changes are mostly related to NFS delegations, I use the `nfstest_delegation` test suite from nfstest[*] to try and identify any regressions: ./nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e /srv/nfstest --trcdelay 4 Both before and after applying the fixes, I see the same 146 tests pass and 23 failures. The 23 failures are expected because I was using a Linux-based NFSv4 server which does not support all of the delegation modes that the test checks for. [*] git://git.linux-nfs.org/projects/mora/nfstest.git To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1957986/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp