Leon Koll wrote:
> Hello, gurus
> I need your help. During the benchmark test of NFS-shared ZFS file systems
> at some moment the number of NFS threads jumps to the maximal value,
> 1027 (NFSD_SERVERS was set to 1024).
Hi Leon,
I tried to send this last night, but it didn't make it to the list.
Here's a resend.
Others have spoken to the NFS service threads doing work in
ZFS-land. I thought it would also be good to provide a brief
overview to explain how NFS service threads are managed.
1. Why are there 1027 threads when the max specifies 1024?
The max doesn't encompass all nfsd threads; rather, it
refers to the maximum number of NFS service threads sitting
in the kernel both waiting for work and doing work.
It doesn't include the nfsd threads spawned to
coordinate/manage the NFS thread pool.
2. Why do the NFS service threads "hang around" even after
the workload has been dramatically reduced from its peak?
As a NFS server becomes more heavily loaded, additional service
threads are created as needed (up to the maximum as specified
in /etc/default/nfs). "As needed" means that a new service
thread is created to dispatch a NFS request when all other
service threads are busy. If threads are idle waiting work,
then then the request is assigned to a service thread in a
round-robin fashion.
After processing a NFS request, a service thread will attempt
to process another. If no NFS requests are waiting to be
processed, then the service thread sleeps for a timeout period
(5 sec by default) waiting for work. The thread is destroyed
if it isn't awoken and assigned a NFS request before timing out.
If the stream of incoming NFS requests is large enough such
that the round-robin scheme provides each service thread with at
least 1 request every 5 sec., then all service threads will
live on to process another request. A steady stream of 1024
NFS requests arriving every 5 seconds could be enough traffic
to sustain 1024 server threads.
Given these design points, the number of NFS service threads
ramps up quickly by demand and ramps down slowly by neglect.
NFS service threads are managed to favor peak workloads.
After the peak workload is "over", even a trivial amount of
NFS traffic can sustain a large number of service threads.
Jeff
