Re: Fix/tuning to improve slow NFS writes?
: I should have mentioned, I have 20 nfsiod's running. I started so many :initially to help in the stress testing I was doing, but I left them :running because the servers are handling from 2-4 requests per second and :we have lots of ram in the boxes. Is there a way to figure out how many are :getting used concurrently, or is too many not a problem? : :Thanks, : :Doug You can look at their cpu utilization with ps. You should see high cpu use on a couple of them and then tapering off. The system allocates nfsiods in order depending on the number of outstanding asynchronous requests. If the web server is not bound by write bandwidth, it could be getting bound up in NFS stat calls. I think what you need to do at this point is run ktrace on some of the web server processes to see where the performance bottlenecks are. Let ktrace collect statistics for a while and then make sure to turn it off with 'ktrace -C', and then use 'kdump -R | less' to see the results. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
Alfred Perlstein wrote: > > On Mon, 9 Aug 1999, Doug wrote: > > > Matthew Dillon wrote: > > > > > > : So, the big question is whether there is anything we can tune to > > > speed up > > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > > :most of the web processing. Overall performance on the reads seems to be > > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > > :server directories are soft mounted directly, with no amd currently in > > > use. > > > : > > > :thanks, > > > : > > > :Doug > > > > > > Well, NFS buffers are usually sent over the network the moment they > > > are full. If you are not running any nfsiod's > > > > I should have mentioned, I have 20 nfsiod's running. I started so many > > initially to help in the stress testing I was doing, but I left them > > running because the servers are handling from 2-4 requests per second and > > we have lots of ram in the boxes. Is there a way to figure out how many are > > getting used concurrently, or is too many not a problem? > > ? > > You need to run 'nfsd' on the servers, not nfsiod. Sorry I wasn't clear. That's what I get for writing posts like this when I'm tired. In that paragraph "servers" refers to the freebsd cgi/web servers that are acting as NFS clients to the sun boxes. Doug To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
: I should have mentioned, I have 20 nfsiod's running. I started so many :initially to help in the stress testing I was doing, but I left them :running because the servers are handling from 2-4 requests per second and :we have lots of ram in the boxes. Is there a way to figure out how many are :getting used concurrently, or is too many not a problem? : :Thanks, : :Doug You can look at their cpu utilization with ps. You should see high cpu use on a couple of them and then tapering off. The system allocates nfsiods in order depending on the number of outstanding asynchronous requests. If the web server is not bound by write bandwidth, it could be getting bound up in NFS stat calls. I think what you need to do at this point is run ktrace on some of the web server processes to see where the performance bottlenecks are. Let ktrace collect statistics for a while and then make sure to turn it off with 'ktrace -C', and then use 'kdump -R | less' to see the results. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
Alfred Perlstein wrote: > > On Mon, 9 Aug 1999, Doug wrote: > > > Matthew Dillon wrote: > > > > > > : So, the big question is whether there is anything we can tune to speed up > > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > > :most of the web processing. Overall performance on the reads seems to be > > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > > :server directories are soft mounted directly, with no amd currently in use. > > > : > > > :thanks, > > > : > > > :Doug > > > > > > Well, NFS buffers are usually sent over the network the moment they > > > are full. If you are not running any nfsiod's > > > > I should have mentioned, I have 20 nfsiod's running. I started so many > > initially to help in the stress testing I was doing, but I left them > > running because the servers are handling from 2-4 requests per second and > > we have lots of ram in the boxes. Is there a way to figure out how many are > > getting used concurrently, or is too many not a problem? > > ? > > You need to run 'nfsd' on the servers, not nfsiod. Sorry I wasn't clear. That's what I get for writing posts like this when I'm tired. In that paragraph "servers" refers to the freebsd cgi/web servers that are acting as NFS clients to the sun boxes. Doug To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
On Mon, Aug 09, 1999 at 04:50:51AM -0400, Alfred Perlstein wrote: > On Mon, 9 Aug 1999, Doug wrote: > > > Matthew Dillon wrote: > > > > > > : So, the big question is whether there is anything we can tune to > > > speed up > > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > > :most of the web processing. Overall performance on the reads seems to be > > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > > :server directories are soft mounted directly, with no amd currently in > > > use. Could tuning any of the NFS options in the kernel help? Matt, could you give any tips? > > I should have mentioned, I have 20 nfsiod's running. I started so many > > initially to help in the stress testing I was doing, but I left them > > running because the servers are handling from 2-4 requests per second and > > we have lots of ram in the boxes. Is there a way to figure out how many are > > getting used concurrently, or is too many not a problem? > > You need to run 'nfsd' on the servers, not nfsiod. > > nfsd - run on server > nfsiod - run on client He's talking about the client boxes, just measuring from the server side. Reference the second sentence at top. Greg -- Gregory S. Sutter Failing sardine factory cans employees! mailto:gsut...@pobox.com http://www.pobox.com/~gsutter/ PGP DSS public key 0x40AE3052 To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
On Mon, 9 Aug 1999, Doug wrote: > Matthew Dillon wrote: > > > > : So, the big question is whether there is anything we can tune to > > speed up > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > :most of the web processing. Overall performance on the reads seems to be > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > :server directories are soft mounted directly, with no amd currently in use. > > : > > :thanks, > > : > > :Doug > > > > Well, NFS buffers are usually sent over the network the moment they > > are full. If you are not running any nfsiod's > > I should have mentioned, I have 20 nfsiod's running. I started so many > initially to help in the stress testing I was doing, but I left them > running because the servers are handling from 2-4 requests per second and > we have lots of ram in the boxes. Is there a way to figure out how many are > getting used concurrently, or is too many not a problem? ? You need to run 'nfsd' on the servers, not nfsiod. nfsd - run on server nfsiod - run on client nfsd takes the same -n arg for the number to start. -Alfred To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
On Mon, Aug 09, 1999 at 04:50:51AM -0400, Alfred Perlstein wrote: > On Mon, 9 Aug 1999, Doug wrote: > > > Matthew Dillon wrote: > > > > > > : So, the big question is whether there is anything we can tune to speed up > > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > > :most of the web processing. Overall performance on the reads seems to be > > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > > :server directories are soft mounted directly, with no amd currently in use. Could tuning any of the NFS options in the kernel help? Matt, could you give any tips? > > I should have mentioned, I have 20 nfsiod's running. I started so many > > initially to help in the stress testing I was doing, but I left them > > running because the servers are handling from 2-4 requests per second and > > we have lots of ram in the boxes. Is there a way to figure out how many are > > getting used concurrently, or is too many not a problem? > > You need to run 'nfsd' on the servers, not nfsiod. > > nfsd - run on server > nfsiod - run on client He's talking about the client boxes, just measuring from the server side. Reference the second sentence at top. Greg -- Gregory S. Sutter Failing sardine factory cans employees! mailto:[EMAIL PROTECTED] http://www.pobox.com/~gsutter/ PGP DSS public key 0x40AE3052 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
On Mon, 9 Aug 1999, Doug wrote: > Matthew Dillon wrote: > > > > : So, the big question is whether there is anything we can tune to speed up > > :the writes. The freebsd machines are NFS clients to the sun servers doing > > :most of the web processing. Overall performance on the reads seems to be > > :best with nfs v3 over udp, which is what I'm using now. All of the web > > :server directories are soft mounted directly, with no amd currently in use. > > : > > :thanks, > > : > > :Doug > > > > Well, NFS buffers are usually sent over the network the moment they > > are full. If you are not running any nfsiod's > > I should have mentioned, I have 20 nfsiod's running. I started so many > initially to help in the stress testing I was doing, but I left them > running because the servers are handling from 2-4 requests per second and > we have lots of ram in the boxes. Is there a way to figure out how many are > getting used concurrently, or is too many not a problem? ? You need to run 'nfsd' on the servers, not nfsiod. nfsd - run on server nfsiod - run on client nfsd takes the same -n arg for the number to start. -Alfred To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
Matthew Dillon wrote: > > : So, the big question is whether there is anything we can tune to > speed up > :the writes. The freebsd machines are NFS clients to the sun servers doing > :most of the web processing. Overall performance on the reads seems to be > :best with nfs v3 over udp, which is what I'm using now. All of the web > :server directories are soft mounted directly, with no amd currently in use. > : > :thanks, > : > :Doug > > Well, NFS buffers are usually sent over the network the moment they > are full. If you are not running any nfsiod's I should have mentioned, I have 20 nfsiod's running. I started so many initially to help in the stress testing I was doing, but I left them running because the servers are handling from 2-4 requests per second and we have lots of ram in the boxes. Is there a way to figure out how many are getting used concurrently, or is too many not a problem? Thanks, Doug To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
: So, the big question is whether there is anything we can tune to speed up :the writes. The freebsd machines are NFS clients to the sun servers doing :most of the web processing. Overall performance on the reads seems to be :best with nfs v3 over udp, which is what I'm using now. All of the web :server directories are soft mounted directly, with no amd currently in use. : :thanks, : :Doug Well, NFS buffers are usually sent over the network the moment they are full. If you are not running any nfsiod's, the RPC will be done synchronously and this will severely impact your performance. Try running: nfsiod -n 4 This will start up a couple of special background processes to handle NFS I/O asynchronously. Without any nfsiod's running my file tests weigh in at 2 MBytes/sec. With a couple running I get 8 MBytes/sec over my network with NFSv3. -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
Matthew Dillon wrote: > > : So, the big question is whether there is anything we can tune to speed up > :the writes. The freebsd machines are NFS clients to the sun servers doing > :most of the web processing. Overall performance on the reads seems to be > :best with nfs v3 over udp, which is what I'm using now. All of the web > :server directories are soft mounted directly, with no amd currently in use. > : > :thanks, > : > :Doug > > Well, NFS buffers are usually sent over the network the moment they > are full. If you are not running any nfsiod's I should have mentioned, I have 20 nfsiod's running. I started so many initially to help in the stress testing I was doing, but I left them running because the servers are handling from 2-4 requests per second and we have lots of ram in the boxes. Is there a way to figure out how many are getting used concurrently, or is too many not a problem? Thanks, Doug To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Fix/tuning to improve slow NFS writes?
: So, the big question is whether there is anything we can tune to speed up :the writes. The freebsd machines are NFS clients to the sun servers doing :most of the web processing. Overall performance on the reads seems to be :best with nfs v3 over udp, which is what I'm using now. All of the web :server directories are soft mounted directly, with no amd currently in use. : :thanks, : :Doug Well, NFS buffers are usually sent over the network the moment they are full. If you are not running any nfsiod's, the RPC will be done synchronously and this will severely impact your performance. Try running: nfsiod -n 4 This will start up a couple of special background processes to handle NFS I/O asynchronously. Without any nfsiod's running my file tests weigh in at 2 MBytes/sec. With a couple running I get 8 MBytes/sec over my network with NFSv3. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Fix/tuning to improve slow NFS writes?
Well my NFS reliability at work has gone way up, to the point where we are now having time to look at the finer things in life, like performance tuning. Performance on reads has been quite good, but when the cgi scripts that the users are running on these machines have large writes to do (like re-indexing a database) the writes go really slow. So slow in fact that they sometimes time out, the CGI engine dies, and the job is never completed, causing it to be restarted by the user. So, the big question is whether there is anything we can tune to speed up the writes. The freebsd machines are NFS clients to the sun servers doing most of the web processing. Overall performance on the reads seems to be best with nfs v3 over udp, which is what I'm using now. All of the web server directories are soft mounted directly, with no amd currently in use. thanks, Doug To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Fix/tuning to improve slow NFS writes?
Well my NFS reliability at work has gone way up, to the point where we are now having time to look at the finer things in life, like performance tuning. Performance on reads has been quite good, but when the cgi scripts that the users are running on these machines have large writes to do (like re-indexing a database) the writes go really slow. So slow in fact that they sometimes time out, the CGI engine dies, and the job is never completed, causing it to be restarted by the user. So, the big question is whether there is anything we can tune to speed up the writes. The freebsd machines are NFS clients to the sun servers doing most of the web processing. Overall performance on the reads seems to be best with nfs v3 over udp, which is what I'm using now. All of the web server directories are soft mounted directly, with no amd currently in use. thanks, Doug To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
