Re: High Availability, High Volume and NFS
Hi, I'm from netapp, but what I can say for you, a netapp is a dedicated box for this kind of usage (high load) if you have any questions don't hesitate Thx Fab. -Original Message- From: John White [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 24, 2001 12:49 AM To: [EMAIL PROTECTED] Subject: Re: High Availability, High Volume and NFS On Wed, May 23, 2001 at 01:40:13PM -0500, Duane Schaub wrote: > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. Others may point out that an observed weakness of the stock linux kernel from RH 6.1 has been shown to have weak NFS performance when compared to some of the BSD O/S family. If you feel comfortable recompiling your kernel, check out http://nfs.sourceforge.net/ > The result was that the qmail machines were BARELY able to keep up. If > there were any pauses on the NFS server, the POP sessions would build to > 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail > exceeded about 70 sessions, it was beyond the point of return and would not > recover. Have you thought about the stopgap measure of throttling down on the number of concurrent pop3 sessions each machine is allowed? Say you want to cap it at 50 total. Just use 50/n, where n is the number of client machines, as the max concurrency for tcpserver (-c). You can increase the client backlog so all the clients see is a pause (-b). http://cr.yp.to/ucspi-tcp/tcpserver.html > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the > performance was abysmal! Performing an ls in a user/new directory took 21 > seconds for a response. It will be tempting to throw more hardware at the problem. Depends on your budget. Right now, I like the the new DDR RAM chipsets for Athlon processors. I like the idea of 3ware hardware IDE RAID which looks like a SCSI controller to the system. Balance the bugetary requirements of upgrading your hardware (without knowing what the effect will be) vs. changing your O/S (with some benchmarking already in hand). Oh, check this out: http://innominate.org/%7Etgr/projects/tuning/ Check out slide 37 for relevant conclusions, but the entire presentation is interesting. > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What > NFS experiences are out there? I've read repeated positive reviews with a netapp, but I still would explore FreeBSD performance first. John White
Re: High Availability, High Volume and NFS
On Wed, May 23, 2001 at 01:40:13PM -0500, Duane Schaub wrote: > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. Others may point out that an observed weakness of the stock linux kernel from RH 6.1 has been shown to have weak NFS performance when compared to some of the BSD O/S family. If you feel comfortable recompiling your kernel, check out http://nfs.sourceforge.net/ > The result was that the qmail machines were BARELY able to keep up. If > there were any pauses on the NFS server, the POP sessions would build to > 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail > exceeded about 70 sessions, it was beyond the point of return and would not > recover. Have you thought about the stopgap measure of throttling down on the number of concurrent pop3 sessions each machine is allowed? Say you want to cap it at 50 total. Just use 50/n, where n is the number of client machines, as the max concurrency for tcpserver (-c). You can increase the client backlog so all the clients see is a pause (-b). http://cr.yp.to/ucspi-tcp/tcpserver.html > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the > performance was abysmal! Performing an ls in a user/new directory took 21 > seconds for a response. It will be tempting to throw more hardware at the problem. Depends on your budget. Right now, I like the the new DDR RAM chipsets for Athlon processors. I like the idea of 3ware hardware IDE RAID which looks like a SCSI controller to the system. Balance the bugetary requirements of upgrading your hardware (without knowing what the effect will be) vs. changing your O/S (with some benchmarking already in hand). Oh, check this out: http://innominate.org/%7Etgr/projects/tuning/ Check out slide 37 for relevant conclusions, but the entire presentation is interesting. > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What > NFS experiences are out there? I've read repeated positive reviews with a netapp, but I still would explore FreeBSD performance first. John White
Re: High Availability, High Volume and NFS
On 23 May 2001, Mark Delany wrote:
> I don't want to start an OS war, but if you want to use NFS on an
> Intel box, I strongly suggest one of the BSDs. I was in a situation
> where I had to use Linux NFS servers - that was until they failed
> miserabled. They were replaced with FreeBSD and the problems went
> away.
Also, check how your OS supports turning UDP checksumming off, and make
sure it's off. It's of no great help on a local switched segment and
affects NFS performance. Are you using traditional NFS or TCP-based NFS?
If performance is the real desired goal, UDP-based NFS is going to be a
lot faster, if not as secure. But, hey, once you're in NFS-land, you're
going to want to keep it all on a tight, local segment if security's even
vaguely your issue.
Finally, you may want to put the server on a gig connection into the
switch, and the client servers on the same switch on 100Mbit FDX.
-M
> Regards.
>
>
> On Wed, May 23, 2001 at 01:40:13PM -0500, Duane Schaub allegedly wrote:
> >
> > I want to set up multiple qmail machines to access an NFS backend. We have
> > about 10,000 users (running maildir) and an average of 5 emails/user/dat and
> > av. 10K in size. On average, there are 6 simultaneous pop sessions with
> > approx. 200 new sessions/min.
> >
> > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients.
> > The result was that the qmail machines were BARELY able to keep up. If
> > there were any pauses on the NFS server, the POP sessions would build to
> > 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail
> > exceeded about 70 sessions, it was beyond the point of return and would not
> > recover.
> >
> > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a
> > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the
> > performance was abysmal! Performing an ls in a user/new directory took 21
> > seconds for a response.
> >
> > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What
> > NFS experiences are out there?
> >
> > If you wish - respond privately [EMAIL PROTECTED]
> >
> > Duane.
> >
Michael Brian Scher [EMAIL PROTECTED] [EMAIL PROTECTED]
Sr. Research Consultant
Attorney, Anthropologist, Part-Time Guru
Mailaise: n, ('mail-aze). See Outlook.
RE: High Availability, High Volume and NFS
Unleash the daemon! ;) -Original Message- From: Mark Delany [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 23, 2001 2:12 PM To: Qmail List Subject: Re: High Availability, High Volume and NFS I don't want to start an OS war, but if you want to use NFS on an Intel box, I strongly suggest one of the BSDs. I was in a situation where I had to use Linux NFS servers - that was until they failed miserabled. They were replaced with FreeBSD and the problems went away. Regards. On Wed, May 23, 2001 at 01:40:13PM -0500, Duane Schaub allegedly wrote: > > I want to set up multiple qmail machines to access an NFS backend. We have > about 10,000 users (running maildir) and an average of 5 emails/user/dat and > av. 10K in size. On average, there are 6 simultaneous pop sessions with > approx. 200 new sessions/min. > > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. > The result was that the qmail machines were BARELY able to keep up. If > there were any pauses on the NFS server, the POP sessions would build to > 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail > exceeded about 70 sessions, it was beyond the point of return and would not > recover. > > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the > performance was abysmal! Performing an ls in a user/new directory took 21 > seconds for a response. > > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What > NFS experiences are out there? > > If you wish - respond privately [EMAIL PROTECTED] > > Duane. > > > > President, | Terra World, Inc. > Terra World, Inc.| 200 ARCO Place, Suite 252 > (888)332-1616| Independence, KS 67301 > (620)332-1616| When your work counts, Use > www.terraworld.net |T E R R A W O R L D > > >
Re: High Availability, High Volume and NFS
Duane Schaub <[EMAIL PROTECTED]> wrote: > > I want to set up multiple qmail machines to access an NFS backend. We have > about 10,000 users (running maildir) and an average of 5 emails/user/dat and > av. 10K in size. 50,000 messages a day, 10k each? Not a huge load. We handle about a tenth of that on one machine with a P733 and SCSI disks. System load is typically << 0.1 (i.e., negligible). > On average, there are 6 simultaneous pop sessions with approx. 200 new > sessions/min. This could be the big problem. > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. You're not using the default kernel shipped with RH6.1, are you? That would be 2.2.5-something. Early 2.2 kernels are known to have absolutely terrible NFS performance as NFS servers, and client-side isn't a whole lot better. NFS has gotten somewhat better in 2.2.19, but server-side is still not up to snuff. You may want to run OpenBSD or FreeBSD as the NFS server. > The NFS server was nothing special (P350/IDE 256Mb RAM). This is your biggest problem. This is not a performance workstation class machine, let alone a high-performance server. You'll notice a huge difference just in switching to SCSI with Linux, even cheap 7200rpm SCSI disks on an average controller. For your load, I would recommend good 15kRPM SCSI disks in a hardware RAID setup (not software RAID). This would probably cure your performance issues on the NFS server. Adding additional RAM would likely benefit as well, as more of the filesystem could be cached. Memory is so cheap these days that you might want to jump right to 1GB. > The result was that the qmail machines were BARELY able to keep up. If > there were any pauses on the NFS server, the POP sessions would build to > 50-60 very quickly with qmail crashing at about 300 sessions. If qmail is "crashing" (prove it!) due to POP3 load, you've seriously misconfigured your server. You should be using tcpserver's concurrency limits to enforce a maximum limit on concurrent POP3 connections to a level that your system can actually handle. qmail itself doesn't crash for any reason related to POP3 access. > If you wish - respond privately [EMAIL PROTECTED] No, I think I'll keep it on the list. That way others can benefit from your experiences and learn from them, rather than asking the same question next week, next month, next year... Charles -- --- Charles Cazabon<[EMAIL PROTECTED]> GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: High Availability, High Volume and NFS
I don't want to start an OS war, but if you want to use NFS on an Intel box, I strongly suggest one of the BSDs. I was in a situation where I had to use Linux NFS servers - that was until they failed miserabled. They were replaced with FreeBSD and the problems went away. Regards. On Wed, May 23, 2001 at 01:40:13PM -0500, Duane Schaub allegedly wrote: > > I want to set up multiple qmail machines to access an NFS backend. We have > about 10,000 users (running maildir) and an average of 5 emails/user/dat and > av. 10K in size. On average, there are 6 simultaneous pop sessions with > approx. 200 new sessions/min. > > We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. > The result was that the qmail machines were BARELY able to keep up. If > there were any pauses on the NFS server, the POP sessions would build to > 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail > exceeded about 70 sessions, it was beyond the point of return and would not > recover. > > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the > performance was abysmal! Performing an ls in a user/new directory took 21 > seconds for a response. > > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What > NFS experiences are out there? > > If you wish - respond privately [EMAIL PROTECTED] > > Duane. > > > > President, | Terra World, Inc. > Terra World, Inc.| 200 ARCO Place, Suite 252 > (888)332-1616| Independence, KS 67301 > (620)332-1616| When your work counts, Use > www.terraworld.net |T E R R A W O R L D > > >
Re: High Availability, High Volume and NFS
> I want to set up multiple qmail machines to access an NFS backend. We have > about 10,000 users (running maildir) and an average of 5 emails/user/dat and > av. 10K in size. On average, there are 6 simultaneous pop sessions with > approx. 200 new sessions/min. I'm working on something similar. > The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a > Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the > performance was abysmal! Performing an ls in a user/new directory took 21 > seconds for a response. > > I think NFS would work, but I don't really want a Netapp F5 ($50,000). What > NFS experiences are out there? Well, our solution was to use an existing EMC Cellera, but I think that's beyond your means :) On the other hand, if you go with a speedy CPU (Athalon 800Mhz+, I think), and at least 1GB of RAM, and a really, really good NIC (I like the NetGear FA310TX, and I think they've come out with a newer one), you should be ok. > > If you wish - respond privately [EMAIL PROTECTED] > > Duane. > Dave
High Availability, High Volume and NFS
I want to set up multiple qmail machines to access an NFS backend. We have about 10,000 users (running maildir) and an average of 5 emails/user/dat and av. 10K in size. On average, there are 6 simultaneous pop sessions with approx. 200 new sessions/min. We have tried a Redhat6.1 backend on the NFS with Redhat 6.1 NFS clients. The result was that the qmail machines were BARELY able to keep up. If there were any pauses on the NFS server, the POP sessions would build to 50-60 very quickly with qmail crashing at about 300 sessions. Once qmail exceeded about 70 sessions, it was beyond the point of return and would not recover. The NFS server was nothing special (P350/IDE 256Mb RAM). We also tried a Dell 2300 (Dual 400/RAID5) NT server running Intergraph NFS But the performance was abysmal! Performing an ls in a user/new directory took 21 seconds for a response. I think NFS would work, but I don't really want a Netapp F5 ($50,000). What NFS experiences are out there? If you wish - respond privately [EMAIL PROTECTED] Duane. President, | Terra World, Inc. Terra World, Inc.| 200 ARCO Place, Suite 252 (888)332-1616| Independence, KS 67301 (620)332-1616| When your work counts, Use www.terraworld.net |T E R R A W O R L D
