Public bug reported:

[Impact]
An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 server, 
causing a significant drop in performance. In my testing, I've seen performance 
drop from ~700MiB/s down to < 10MiB/s. The same workload using NFSv3 does not 
have this problem.

[Test Case]
This can be demonstrated using the elbencho benchmark from 
https://github.com/breuner/elbencho:
 $ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu

You'll notice the nfsd threads (I stuck w/ the default of 4) start to
consume 100% CPU, and the performance of the elbencho benchmark will
begin to trickle.

[Fix]
The following fix solves the problem, but there are a number of patches 
dependencies required before it will apply to focal:

commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad)
Author: Trond Myklebust <tron...@gmail.com>
Date:   Mon Jan 27 09:58:19 2020 -0500

    NFSv4: Limit the total number of cached delegations
    
    Delegations can be expensive to return, and can cause scalability issues
    for the server. Let's therefore try to limit the number of inactive
    delegations we hold.
    Once the number of delegations is above a certain threshold, start
    to return them on close.
    
    Signed-off-by: Trond Myklebust <trond.mykleb...@hammerspace.com>
    Signed-off-by: Anna Schumaker <anna.schuma...@netapp.com>

[What could go wrong]
The fixes are restricted to NFS code, so problems should be limited to NFS 
users. They could include performance issues, crashes, etc.

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Fix Released

** Affects: linux (Ubuntu Focal)
     Importance: Undecided
     Assignee: dann frazier (dannf)
         Status: In Progress

** Affects: linux (Ubuntu Hirsute)
     Importance: Undecided
         Status: Fix Released

** Affects: linux (Ubuntu Impish)
     Importance: Undecided
         Status: Fix Released

** Affects: linux (Ubuntu Jammy)
     Importance: Undecided
         Status: Fix Released

** Also affects: linux (Ubuntu Jammy)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Hirsute)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Impish)
   Importance: Undecided
       Status: New

** Also affects: linux (Ubuntu Focal)
   Importance: Undecided
       Status: New

** Changed in: linux (Ubuntu Jammy)
       Status: New => Fix Released

** Changed in: linux (Ubuntu Impish)
       Status: New => Fix Released

** Changed in: linux (Ubuntu Hirsute)
       Status: New => Fix Released

** Changed in: linux (Ubuntu Focal)
       Status: New => In Progress

** Changed in: linux (Ubuntu Focal)
     Assignee: (unassigned) => dann frazier (dannf)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1957986

Title:
  Recalled NFSv4 files delegations overwhelm server

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Focal:
  In Progress
Status in linux source package in Hirsute:
  Fix Released
Status in linux source package in Impish:
  Fix Released
Status in linux source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  An NFSv4 client that does a lot of opens/closes can overwhelm and NFSv4 
server, causing a significant drop in performance. In my testing, I've seen 
performance drop from ~700MiB/s down to < 10MiB/s. The same workload using 
NFSv3 does not have this problem.

  [Test Case]
  This can be demonstrated using the elbencho benchmark from 
https://github.com/breuner/elbencho:
   $ elbencho -t 40 -r -n 10 -N 5000 -s 128k -b 128k /mnt/nfs/ubuntu

  You'll notice the nfsd threads (I stuck w/ the default of 4) start to
  consume 100% CPU, and the performance of the elbencho benchmark will
  begin to trickle.

  [Fix]
  The following fix solves the problem, but there are a number of patches 
dependencies required before it will apply to focal:

  commit 10717f45639f6c1bc27b56405252c3a027406d92 (refs/bisect/bad)
  Author: Trond Myklebust <tron...@gmail.com>
  Date:   Mon Jan 27 09:58:19 2020 -0500

      NFSv4: Limit the total number of cached delegations
      
      Delegations can be expensive to return, and can cause scalability issues
      for the server. Let's therefore try to limit the number of inactive
      delegations we hold.
      Once the number of delegations is above a certain threshold, start
      to return them on close.
      
      Signed-off-by: Trond Myklebust <trond.mykleb...@hammerspace.com>
      Signed-off-by: Anna Schumaker <anna.schuma...@netapp.com>

  [What could go wrong]
  The fixes are restricted to NFS code, so problems should be limited to NFS 
users. They could include performance issues, crashes, etc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1957986/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to