= focal verification =
ubuntu@client:~$ cat /proc/version_signature 
Ubuntu 5.4.0-102.115-generic 5.4.174
ubuntu@client:~$ ./elbencho/bin/elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k 
/mnt/nfs/ubuntu 
OPERATION RESULT TYPE        FIRST DONE  LAST DONE
========= ================   ==========  =========
READ      Elapsed ms       :     470642     471169
          Files/s          :       4245       4244
          Throughput MiB/s :        530        530
          Total files      :    1998154    2000000
          Total MiB        :     249770     250000
---
ubuntu@client:~$ 


Regression:
ubuntu@client:~/nfstest$ PYTHONPATH=$(pwd) ./test/nfstest_delegation --client 
192.168.42.1 --server 192.168.42.2 -e /srv/nfstest --trcdelay 4
UID is:  1000
Creating:  /mnt/t/nfstest_delegation_20220225_223841_f_001
UID is:  1000
Creating:  /mnt/t/nfstest_delegation_20220225_223841_f_002

*** Basic READ delegation test
    TEST: Running test 'basic01'
    PASS: Open file for READ should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: READ delegation should be granted
    TIME: 5.113318s

*** Basic WRITE delegation test
    TEST: Running test 'basic02'
    PASS: Open file for WRITE should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for WRITE on same process should succeed
    PASS: Write file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: Open file for WRITE from a different process should succeed
    PASS: Write file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.940012s

*** Basic READ delegation test with file stat
    TEST: Running test 'basic03'
    PASS: Stat file to cache file metadata should succeed
    PASS: Open file for READ should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: LOOKUP operation should be sent
    PASS: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_FH
    PASS: OPEN should be sent with the filehandle of the file to be opened
    PASS: READ delegation should be granted
    PASS: OPEN should not be sent for the same file
    PASS: READs should be sent to the server with the DELEG stateid
    PASS: READs should return NFS4_OK from the server
    PASS: READ should not be sent when reading delegated file from a different 
process
    PASS: CLOSE should be sent to the server
    PASS: CLOSE should be sent with correct OPEN stateid
    PASS: DELEGRETURN should be sent after the close
    PASS: DELEGRETURN should be sent with the delegation stateid
    TIME: 4.757017s

*** Basic WRITE delegation test with file stat
    TEST: Running test 'basic04'
UID is:  1000
Creating:  /mnt/t/nfstest_delegation_20220225_223841_f_004
    PASS: Stat file to cache file metadata should succeed
    PASS: Open file for WRITE should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for WRITE on same process should succeed
    PASS: Write file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: Open file for WRITE from a different process should succeed
    PASS: Write file from a different process should succeed
    PASS: LOOKUP operation should be sent
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_FH
    FAIL: OPEN should be sent with the filehandle of the file to be opened
    FAIL: WRITE delegation should be granted
    TIME: 5.090116s

*** Basic READ delegation test with file lock
    TEST: Running test 'basic05'
    PASS: Open file for READ should succeed
    PASS: Lock file with F_RDLCK should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    PASS: READ delegation should be granted
    PASS: OPEN should not be sent for the same file
    PASS: LOCK should not be sent to the server
    PASS: READs should be sent to the server with the DELEG stateid
    PASS: READs should return NFS4_OK from the server
    PASS: READ should not be sent when reading delegated file from a different 
process
    PASS: CLOSE should be sent to the server
    PASS: CLOSE should be sent with correct OPEN stateid
    PASS: DELEGRETURN should be sent after the close
    PASS: DELEGRETURN should be sent with the delegation stateid
    TIME: 4.794486s

*** Basic WRITE delegation test with file lock
    TEST: Running test 'basic06'
    PASS: Open file for WRITE should succeed
    PASS: Lock file with F_WRLCK should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for WRITE on same process should succeed
    PASS: Write file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: Open file for WRITE from a different process should succeed
    PASS: Write file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.962986s

*** Basic WRITE delegation test using RDWR open while reading
    TEST: Running test 'basic07'
    PASS: Open file for RDWR should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.806131s

*** Basic WRITE delegation test using RDWR open while writing
    TEST: Running test 'basic08'
    PASS: Open file for RDWR should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.711963s

*** Basic WRITE delegation test using RDWR open while reading with file stat
    TEST: Running test 'basic09'
    PASS: Stat file to cache file metadata should succeed
    PASS: Open file for RDWR should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: LOOKUP operation should be sent
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_FH
    FAIL: OPEN should be sent with the filehandle of the file to be opened
    FAIL: WRITE delegation should be granted
    TIME: 4.749760s

*** Basic WRITE delegation test using RDWR open while writing with file stat
    TEST: Running test 'basic10'
UID is:  1000
Creating:  /mnt/t/nfstest_delegation_20220225_223841_f_007
    PASS: Stat file to cache file metadata should succeed
    PASS: Open file for RDWR should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    PASS: LOOKUP operation should be sent
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_FH
    FAIL: OPEN should be sent with the filehandle of the file to be opened
    FAIL: WRITE delegation should be granted
    TIME: 4.869004s

*** Basic WRITE delegation test using RDWR open while reading with file lock
    TEST: Running test 'basic11'
    PASS: Open file for RDWR should succeed
    PASS: Lock file with F_RDLCK should succeed
    PASS: Read file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.766863s

*** Basic WRITE delegation test using RDWR open while writing with file lock
    TEST: Running test 'basic12'
    PASS: Open file for RDWR should succeed
    PASS: Lock file with F_WRLCK should succeed
    PASS: Write file should succeed
    PASS: Open file for READ on same process should succeed
    PASS: Read file on same process should succeed
    PASS: Open file for READ from a different process should succeed
    PASS: Read file from a different process should succeed
    FAIL: OPEN should be sent
    PASS: OPEN should be sent with CLAIM_NULL
    PASS: OPEN should be sent with the name of the file to be opened
    PASS: OPEN should be sent with the filehandle of the directory
    FAIL: WRITE delegation should be granted
    TIME: 4.725836s

169 tests (146 passed, 23 failed)

Total time: 58.446193s
ubuntu@client:~/nfstest$ 

** Tags removed: verification-needed-focal
** Tags added: verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1957986

Title:
  Recalled NFSv4 files delegations overwhelm server

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Focal:
  Fix Committed
Status in linux source package in Hirsute:
  Fix Released
Status in linux source package in Impish:
  Fix Released
Status in linux source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 
server, causing a significant drop in performance. In my testing, I've seen 
performance drop from ~700MiB/s down to < 10MiB/s. The same workload using 
NFSv3 does not have this problem.

  [Test Case]
  This can be demonstrated using the elbencho benchmark from 
https://github.com/breuner/elbencho:
   $ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu

  You'll notice the nfsd threads (I stuck w/ the default of 4) start to
  consume 100% CPU, and the performance of the elbencho benchmark will
  begin to trickle.

  [Fix]
  The following fix solves the problem, but there are a number of patches 
dependencies required before it will apply to focal:

  commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad)
  Author: Trond Myklebust <tron...@gmail.com>
  Date:   Mon Jan 27 09:58:19 2020 -0500

      NFSv4: Limit the total number of cached delegations

      Delegations can be expensive to return, and can cause scalability issues
      for the server. Let's therefore try to limit the number of inactive
      delegations we hold.
      Once the number of delegations is above a certain threshold, start
      to return them on close.

      Signed-off-by: Trond Myklebust <trond.mykleb...@hammerspace.com>
      Signed-off-by: Anna Schumaker <anna.schuma...@netapp.com>

  [What could go wrong]
  The fixes are restricted to NFS code, so problems should be limited to NFS 
users. They could include performance issues, crashes, etc. Because these 
changes are mostly related to NFS delegations, I use the `nfstest_delegation` 
test suite from nfstest[*] to try and identify any regressions: 

  ./nfstest_delegation --client 192.168.42.1 --server 192.168.42.2 -e
  /srv/nfstest --trcdelay 4

  Both before and after applying the fixes, I see the same 146 tests
  pass and 23 failures. The 23 failures are expected because I was using
  a Linux-based NFSv4 server which does not support all of the
  delegation modes that the test checks for.

  [*] git://git.linux-nfs.org/projects/mora/nfstest.git

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1957986/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to