Re: Max NFSD processes
On Fri, May 21, 2004 at 01:06:41PM -0500, Eric Anderson wrote: > > > > Disk design issues matter cause it is the ultimate > >bottleneck. RAID is you friend. > > > > Just curious - what RAID level/configuration do you use? I've typically > used RAID5, but RAID0+1 might be acceptable. > Depends. RAID5 has worst right performance, so it might be bad for something like mail. Generally I use RAID0+1 for mail and RAID5 for web data. Also, depending on your budget and/or concerns about failure/downtime, you might want to vinum mirror the entire base system disks. Be sure to have the root partition vinumed also. There is good documentation in the handbook and on gregs page. Actually all my nfs servers are "diskless" except for the data and boot off of boot servers using PXE. The boot servers have vinumed system disks. -steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
Steve Shorter wrote: Thanks - any help/hints is appreciated. Disk design issues matter cause it is the ultimate bottleneck. RAID is you friend. Just curious - what RAID level/configuration do you use? I've typically used RAID5, but RAID0+1 might be acceptable. Eric -- -- Eric Anderson Sr. Systems AdministratorCentaur Technology Today is the tomorrow you worried about yesterday. -- ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
On Thu, May 20, 2004 at 10:44:14AM -0500, Eric Anderson wrote: > > > > That's good to hear. Did you do any other tweaks? sysctl settings? > mbufs? net.inet.udp.recvspace=524288 kern.ipc.maxsockbuf=1048576 net.inet.icmp.drop_redirect=1 net.inet.icmp.log_redirect=1 Telus-nfs1:# w 12:41PM up 212 days, 15 hrs, 3 users, load averages: 0.49, 0.62, 0.71 USER TTY FROM LOGIN@ IDLE WHAT nfs1:# netstat -m 441/1488/65536 mbufs in use (current/peak/max): 406 mbufs allocated to data 35 mbufs allocated to packet headers 242/1040/16384 mbuf clusters in use (current/peak/max) 2452 Kbytes allocated to network (4% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines > > > >>Thanks - any help/hints is appreciated. > >> > >> Disk design issues matter cause it is the ultimate bottleneck. RAID is you friend. Lots of RAM helps. I use 4G. and compile a custom kernel with maxusers at 256 and KVA_PAGES at 512. You can check/verify kvm usage with sysclt's vm.kvm_size and vm.kvm_free > >> > > > > You probably also want good nics (fxp0) and to > >increase UDP buffer space. I have found that nfs over udp > >offers supperior performance than tcp on a good LAN > > > > > I'm currently using 3com's (xl0,xl1) and Intel Gigabit cards (em0,em1). > Most of my clients are using udp. > > What did you set your buffer space to? Which sysctl did you change? > udp recvspace. see above. -steve ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
Steve Shorter wrote: On Wed, May 19, 2004 at 04:55:20PM -0500, Eric Anderson wrote: I have several heavily used NFS servers, currently running FreeBSD 4.9-RELEASE. I'm getting jammed up with all my nfsd processes being busy, so clients see slow connections to the server. I have the nfsd starting with a count of 20, which is the max set in the nfsd.c file. Are there any risks I should be aware of before bumping up the max to say 40, or even 50? Depending on where the bottlnecks in the system are, you can crank this up to whatever works for you. I have tested and am running nfs servers with 80 and 100 nfsd's with no problems at all That's good to hear. Did you do any other tweaks? sysctl settings? mbufs? What would it take to make this a sysctl adjustable value? This isn't neccessary IM0, because the number of nfd's can be set at runtime. But the default max in nfsd.c should be increased. I meant a sysctl for the MAXNFSDCNT setting in nfsd.c. Should the max be bumped higher by default nowdays? Yep. Thanks - any help/hints is appreciated. You probably also want good nics (fxp0) and to increase UDP buffer space. I have found that nfs over udp offers supperior performance than tcp on a good LAN I'm currently using 3com's (xl0,xl1) and Intel Gigabit cards (em0,em1). Most of my clients are using udp. What did you set your buffer space to? Which sysctl did you change? Thanks! Eric -- -- Eric Anderson Sr. Systems AdministratorCentaur Technology Today is the tomorrow you worried about yesterday. -- ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
Christian Hiris wrote: About a year ago i observed strong nfs performance decrease when using RLT8139A nics. Nfs transfers leaded into high system load, because of an excessive high packet retransmission rate. Switching over to 3Com nics solved my problem. The specific model and it's close relatives is only suitable for light use, like basic web surfing, small remote-monitoring applications, etc. The more recent realtek chips support more sane ways to access the hardware and wastly increased performance. You'll want RTL8139C+, RTL8169, etc. which use the "re" driver instead of the "rl" driver. Pete ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
On Wednesday 19 May 2004 23:55, Eric Anderson wrote: > I have several heavily used NFS servers, currently running FreeBSD > 4.9-RELEASE. I'm getting jammed up with all my nfsd processes being > busy, so clients see slow connections to the server. I have the nfsd > starting with a count of 20, which is the max set in the nfsd.c file. > > Are there any risks I should be aware of before bumping up the max to > say 40, or even 50? > > What would it take to make this a sysctl adjustable value? > > Should the max be bumped higher by default nowdays? > > Thanks - any help/hints is appreciated. > > Eric About a year ago i observed strong nfs performance decrease when using RLT8139A nics. Nfs transfers leaded into high system load, because of an excessive high packet retransmission rate. Switching over to 3Com nics solved my problem. regards ch -- Christian Hiris <[EMAIL PROTECTED]> | OpenPGP KeyID 0x941B6B0B OpenPGP-Key at hkp://wwwkeys.eu.pgp.net and http://pgp.mit.edu pgpenSizGulou.pgp Description: signature
Re: Max NFSD processes
Dan Nelson wrote: In the last episode (May 19), Eric Anderson said: I have several heavily used NFS servers, currently running FreeBSD 4.9-RELEASE. I'm getting jammed up with all my nfsd processes being busy, so clients see slow connections to the server. I have the nfsd starting with a count of 20, which is the max set in the nfsd.c file. Are there any risks I should be aware of before bumping up the max to say 40, or even 50? What would it take to make this a sysctl adjustable value? Should the max be bumped higher by default nowdays? What's the output of "ps ax | grep nfsd"? How much CPU does the last nfsd process have? If your backend storage is a RAID with lots of disks, and your last nfsd is actually getting some use, then bumping up the nfsds will probably help. Although if you're hitting a kernel bottleneck (locking for example), more nfsds won't do any good. Here's the output: 97 ?? Is 0:00.01 nfsd: master (nfsd) 99 ?? S 4:52.61 nfsd: server (nfsd) 100 ?? S 1:15.74 nfsd: server (nfsd) 101 ?? S 0:44.05 nfsd: server (nfsd) 102 ?? S 0:31.79 nfsd: server (nfsd) 103 ?? S 0:26.15 nfsd: server (nfsd) 104 ?? S 0:20.36 nfsd: server (nfsd) 105 ?? S 0:18.47 nfsd: server (nfsd) 106 ?? S 0:16.86 nfsd: server (nfsd) 107 ?? S 0:19.11 nfsd: server (nfsd) 108 ?? S 0:16.68 nfsd: server (nfsd) 109 ?? S 0:13.59 nfsd: server (nfsd) 110 ?? S 0:13.60 nfsd: server (nfsd) 111 ?? S 0:12.30 nfsd: server (nfsd) 112 ?? S 0:12.44 nfsd: server (nfsd) 113 ?? S 0:13.84 nfsd: server (nfsd) 114 ?? S 0:12.65 nfsd: server (nfsd) 115 ?? S 0:13.57 nfsd: server (nfsd) 116 ?? S 0:11.31 nfsd: server (nfsd) 117 ?? S 0:11.21 nfsd: server (nfsd) 118 ?? I 0:11.99 nfsd: server (nfsd) The machine has been up now less than 5 hours, and this is a 'quiet' time. During the 'slow' time, top showed the nfsd processes in "biorw" and "inode" states. All were consumed in those states. The machine has two raid 5 arrays, with a hardware raid controller. iostat showed xfer speeds to the first array about 2MB/s, and nothing really abnormal about it. Clients had difficulty with simple things like 'ls' on the partition. mountd was responding quickly with mount requests (I believe), but once the mount was made, accessing the nfs disk was horribly slow. Any more ideas? I'm not subscribed on -questions or -net, so please keep me on the cc's. Eric -- -- Eric Anderson Sr. Systems AdministratorCentaur Technology Today is the tomorrow you worried about yesterday. -- ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Max NFSD processes
In the last episode (May 19), Eric Anderson said: > I have several heavily used NFS servers, currently running FreeBSD > 4.9-RELEASE. I'm getting jammed up with all my nfsd processes being > busy, so clients see slow connections to the server. I have the nfsd > starting with a count of 20, which is the max set in the nfsd.c file. > > Are there any risks I should be aware of before bumping up the max to > say 40, or even 50? > > What would it take to make this a sysctl adjustable value? > > Should the max be bumped higher by default nowdays? What's the output of "ps ax | grep nfsd"? How much CPU does the last nfsd process have? If your backend storage is a RAID with lots of disks, and your last nfsd is actually getting some use, then bumping up the nfsds will probably help. Although if you're hitting a kernel bottleneck (locking for example), more nfsds won't do any good. -- Dan Nelson [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Max NFSD processes
I have several heavily used NFS servers, currently running FreeBSD 4.9-RELEASE. I'm getting jammed up with all my nfsd processes being busy, so clients see slow connections to the server. I have the nfsd starting with a count of 20, which is the max set in the nfsd.c file. Are there any risks I should be aware of before bumping up the max to say 40, or even 50? What would it take to make this a sysctl adjustable value? Should the max be bumped higher by default nowdays? Thanks - any help/hints is appreciated. Eric -- -- Eric Anderson Sr. Systems AdministratorCentaur Technology Today is the tomorrow you worried about yesterday. -- ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
