Re: QMail Performance Question Miscellaneous Issues
On Thu, May 11, 2000 at 10:54:37PM -0700, John White wrote: If you're looking for queue speed, you want RAID 1+0 with a NVRAM cache to accellerate the small block writes. zeroseek would be even cooler. Neil -- There are two rules for success in life: Rule 1: Don't tell people everything you know.
Re: QMail Performance Question Miscellaneous Issues
On Fri, May 12, 2000 at 12:40:28AM -0600, Neil Schemenauer wrote: On Thu, May 11, 2000 at 10:54:37PM -0700, John White wrote: If you're looking for queue speed, you want RAID 1+0 with a NVRAM cache to accellerate the small block writes. zeroseek would be even cooler. Don't use zeroseek! Any writes made to vaporware are automatically lost!!! :) John
RE: QMail Performance Question Miscellaneous Issues
To throw my $.02 at this issue, if this is looking like a "low level" speed issue, why not tinker with the hardware? Taking the two disks that hold /var and putting them on a RAID0 set should give you a serious speed boost without touching your (otherwise working) qmail config. -Original Message- On a dual celeron 466 with 512Mb ram. and 3 10k scsi drives (one for /var/qmail/queue, one for /var/log, one for /usr/home) While it's hard to tell without looking, by guess is that your inbound submission rate is killing the spindle that your disk lives on. If nothing is coming in, the remote processes usually are 20-80, and only on a very rare occasion would get close to my 200 concurrencyremote. That suggests to me that something is awry. I have never seen any properly setup system achieve the configured concurrencyremote. Not getting your concurrencyremote implies that qmail-send (via qmail-rspawn) cannot fork prepare a message and fork a qmail-remote process quickly enough to keep up with the exit rate. Thus *usually* means that qmail-send hasn't sufficient resources (such as spindle or file system performance). So .. eh... would it likely be my disk I/O that slows it down (how do I test that?), or should I be switching to FreeBSD, or am I doing something stupid?
Re: QMail Performance Question Miscellaneous Issues
On Thu, May 11, 2000 at 09:01:27AM -0400, Steve Craft wrote: To throw my $.02 at this issue, if this is looking like a "low level" speed issue, why not tinker with the hardware? Taking the two disks that hold /var and putting them on a RAID0 set should give you a serious speed boost without touching your (otherwise working) qmail config. Definitely don't do that. :) With disk striping, any disk failure = total loss of the queue. If you're looking for queue speed, you want RAID 1+0 with a NVRAM cache to accellerate the small block writes. John
Re: QMail Performance Question Miscellaneous Issues
At 02:40 AM 5/10/2000, Neil Schemenauer wrote: You should find the bottleneck before you jump to any conclusions. What version of the Linux kernel are you using? 2.2.12 compiled with higher process limit (4090), higher file and inode limits (16000/48000), smp support, and drivers for SCSI and network card compiled in. Do you have any strange looking error messages in your log files? Nothing that looks unusual. A ton of "in.identd started" messages. No unusual error messages. What does "vmstat 1" show? I didn't know that one. I'll check when the servers are busy the next time. Generally top shows relatively low memory use and no swap space used. Perhaps you should install sar(sp?) and profile your disk IO. I don't think sar or sarcheck runs on Linux at this point, as far as I can quickly figure out. But something like that would be very useful. At 07:09 AM 5/10/2000, Ricardo D. Albano wrote: Are you using syslogd ? No. Or, rather, it is there, but I'm not using it for qmail, so it is not doing much. Back when I was using sendmail and syslogd, that was indeed a big bottleneck, ending up consuming a majority of server resources. At 10:09 AM 5/10/2000, [EMAIL PROTECTED] wrote: It's not clear to me that these are valid comparisons. Is the 12mill per day mean 12M individual messages individually queued? Are is it a much smaller number of messages with a larger number of recipients? I'd like to know that too. What I'm doing is individual messages individually queued, so if somebody can get 12M that way, I'm certainly paying attention. I'd be perfectly happy doing 2M per machine without crashing anything. While it's hard to tell without looking, my guess is that your inbound submission rate is killing the spindle that your disk lives on. Sort of looks like it. I suppose there is no meaningful way of separating the stuff I put in from what needs to go out, as it is obviously the same queue. And, still, I don't get it. I can't seem to feed much more than 60,000 messages per hour into the queue. That's between two machines standing next to each other, on a 100Mbps switched network. SMTP or QMTP seems to make no difference. That's no faster than the machine can go and deliver the messages remotely, when it is in a good mood, and nothing is coming in at the time. I would be able to create 60,000 mail message files in a couple of minutes. Should I be thinking along the lines of putting the files directly into the queue myself? I'd really be much more comfortable leaving that kind of stuff to qmail-qmtp and qmail-queue. In other words, expect to reach your concurrencyremote. Not getting their when everything appears right, is a sign of some other underlying problem. Now, after saying all of this, I did get a hint yesterday that my outgoing bottleneck might possibly relate to bandwidth problems. I was mailing from 3 machines at the same time, each with around 100,000 messages in the queue and concurrencyremote set at 200. And for the first time I saw one of them actually sneak up to 200, with ~3Mbps outgoing traffic, without even working very hard. However, while the other two were idling around 20-30. And a little later another of the machines went up towards 200, while the first one dropped down to 20-30. Seemed kind of bizarre, but might indicate some kind of "smart" network switch that's trying to apportion out bandwidth according to some algorhitm. I'm looking into that. - Flemming
Re: QMail Performance Question Miscellaneous Issues
On Wed, May 10, 2000 at 01:30:53AM -0700, Flemming Funch wrote: So .. eh... would it likely be my disk I/O that slows it down (how do I test that?), or should I be switching to FreeBSD, or am I doing something stupid? You should find the bottleneck before you jump to any conclusions. What version of the Linux kernel are you using? Do you have any strange looking error messages in your log files? What does "vmstat 1" show? Perhaps you should install sar(sp?) and profile your disk IO. Regarding FreeBSD... rant The VM system in Linux 2.2 and 2.3 is fucked. I am (used to be?) a big fan of Linux but I can't believe what is happening on linux-kernel lately. Rather than sitting down and figuring out the problems people just keep screwing around with things hoping somehow it will get fixed. Things have gotten progressively worse. The latest kernel (pre7) is a joke. Many people report their system is unusable. The 2.4 release is approaching. Now is not the time to be totally overhauling the VM code. Being unafraid to shake things up and change code is one thing but this is ridiculous. The same thing happened before the 2.2 release. Major changes were made right up to the release. I think significant changes happened even in the 2.2 releases (people say 2.2.10 is better than 2.2.15). David Miller is supposed to be working on a new design similar to the FreeBSD VM. I still have some faith that he will do a good job. In the mean time I have installed FreeBSD-stable on an extra partition and am planning my cutover. /rant Okay, I'm feeling much better now. Sorry for that. If it doesn't cost you too much time, trying FreeBSD would be a very interesting experiment. If you do switch would you mind posting a summary of the results? Neil -- Segmentation fault (core dumped)
RE: QMail Performance Question Miscellaneous Issues
At 09:39 AM 5/9/2000, Matthew B. Henniges wrote: On a dual celeron 466 with 512Mb ram. and 3 10k scsi drives (one for /var/qmail/queue, one for /var/log, one for /usr/home) concurrency remote at 500 concurrency local at 50 FreeBSD 3.4-S localhost dnscache It will push 12 Million on a good day. (4% local delivery). This is qmail 1.03 + big-todo + big-concurrency + qmailqueue I'm green with envy. Now, I administer around 6 qmail servers. Typically a dual-600PIII with 1G of RAM, with /var on a 10K SCSI, and everything else on other disks. I also use qmail 1.03 + big-todo + big-concurrency. Remote concurrency set for 200. Queue set for a split of 293. Linux RH6.1 or 2. Outgoing mail is handled on different servers than the incoming. The machines are co-located on several different networks with plenty of bandwidth. The machines are mostly sending out daily newsletters which are being fed in from another machine by smtp or qmtp (seems to make no difference in performance which I use), and I've experimented with various numbers of incoming smtp processes. If I'm sending more in more than a couple of smtp connections at the same time (e.g. 10 or 20), concurrent remote processes drop to a crawl of 2-10, the machine's load gets really high, 6-20, and the queue gets filled up quickly. If nothing is coming in, the remote processes usually are 20-80, and only on a very rare occasion would get close to my 200 concurrencyremote. So .. eh... would it likely be my disk I/O that slows it down (how do I test that?), or should I be switching to FreeBSD, or am I doing something stupid? What is localhost dnscache about? A local name server, to limit outgoing DNS lookups? - Flemming
RE: QMail Performance Question Miscellaneous Issues
I have a question about qmail regarding its mail handling capacity. How many remote emails can qmail send simulataneously, assuming it is run on a Dual-CPU PIII 500Mhz with 512Mb RAM and a SCSI hard disk? The internet bandwidth is 10 Mbps. On a dual celeron 466 with 512Mb ram. and 3 10k scsi drives (one for /var/qmail/queue, one for /var/log, one for /usr/home) concurrency remote at 500 concurrency local at 50 FreeBSD 3.4-S localhost dnscache It will push 12 Million on a good day. (4% local delivery). This is qmail 1.03 + big-todo + big-concurrency + qmailqueue What is the general number of emails that a machine with the above specifications can send per second/hour/day? How do I fine-tune it to send off millions? I only know of changing the "concurrencyremote" figure in /var/qmail/control/concurrencyremote. I set it to 100 for testing. What should be a good figure assuming that I will do free email hosting on the server for hundreds of thousands of users? whoa there...hundreds of thousands of users? You are going to need much better disk performance than one scsi disk will give you. More info below I have also noticed that some free email services like Yahoo also uses QMail (if I'm not mistaken). They have millions of users, so I assume they host the email service on multiple machines. I think that's a safe bet. How is it possible to do load-balancing for emails on multiple machines? 'cause everyone will have an email address like [EMAIL PROTECTED], but how does Yahoo redirect portions of users to different machines for mail receival/sending? Well, I don't work at yahoo, but off the top of my head something like this comes to mind: nfs1.dom.com has a large, fast raid array attached to it. smtp1.dom.com smtp2.dom.com smtp3.dom.com ... smtpn.dom.com are all servers running qmail/qmail-smtpd. There are set up to do local delivery to maildirs in /usr/home/popuser (For more information on running multiple pop boxes under one UID, follow some links on the qmail home page (like vpopmail and pop toaster). /usr/home/popuser is mounted via nfs from the machine nfs1.dom.com via a separate 100mbit ethernet segment. pop1.dom.com pop2.dom.com pop3.dom.com ... popn.dom.com are all servers running qmail-pop3d and friends... There serve the pop boxes from /usr/home/popuser, which they mount from nfs1.dom.com smtp.dom.com points to smtp1, smtp2, smtp3 ... pop.dom.com points to pop1, pop2, pop3 ... That setup should be able to scale pretty well, as long as the NFS box is up to the challenge... (quad zeon connected to 3 firewire raid 5 arrays and running software raid 0 over them? :D) This sound reasonably to the rest of you? Matthew B. Henniges CoPresident Axl.net Communications http://www.axl.net (203) 552-1714
Re: QMail Performance Question Miscellaneous Issues
Another allows increasing the maximum number of concurrent remotes beyond 250. The patch allows up to 500 but that limit seems to be linux related. I would imagine that to be because Linux by default only allows 1024 file handles to be open at once. If each of the qmail-remotes has a message in the queue open, then having too many can cause "bad things", as we found when we needed to run a few hundred extra virtual domains on our web server, and the same thing happened. Despite the docs at RedHat.com, saying how easy it is to increase the file-handle limit on the new kernels, I found that it simply didn't work. Editing the source and recompiling the kernel (as you had to in older kernels) did the trick. It's not even that hard, that was the first time I had ever fiddled with the Kernel source. We went to 4096, which should allow for quite a few qmail-remotes. : ) steve
Re: QMail Performance Question Miscellaneous Issues
On Sun, May 07, 2000 at 01:17:13PM -0600, Steve Wolfe wrote: Despite the docs at RedHat.com, saying how easy it is to increase the file-handle limit on the new kernels, I found that it simply didn't work. Editing the source and recompiling the kernel (as you had to in older kernels) did the trick. The documentation of RedHat.com is technically accurate, just not complete. There are two limits. One is the total number of files handles for all processes. This is adjustable through /proc/sys/fs/file-max. The other limit is the number of file handles opened by a single process. This is dynamic in the newer kernels. You are probably running into a resource limit. Try: ulimit -n I just opened 2000 files after using "ulimit -n 2048" (the default was 1024). No recompile was required. Neil
Re: QMail Performance Question Miscellaneous Issues
It's not even that hard, that was the first time I had ever fiddled with the Kernel source. We went to 4096, which should allow for quite a few qmail-remotes. : ) Do you have any feel for how to evaluate what is an optimum number of remotes? At 400 remotes I still have 80% CPU idle time. The load average is around 4. This suggests to me that the system is limited by disk I/O, network I/O or the responsiveness of remote servers. If it is Disk I/O more remotes won't help. I track network traffic through our router fairly closely and don't think that is the bottle neck. If it is the responsiveness of remote servers then more remotes will help.
Re: QMail Performance Question Miscellaneous Issues
The documentation of RedHat.com is technically accurate, just not complete. There are two limits. One is the total number of files handles for all processes. This is adjustable through /proc/sys/fs/file-max. The other limit is the number of file handles opened by a single process. This is dynamic in the newer kernels. You are probably running into a resource limit. Try: ulimit -n I just opened 2000 files after using "ulimit -n 2048" (the default was 1024). No recompile was required. Ah. In this case, the number of files that a user can have open (1024) was the same as the total number of files that the entire system could have - I imagine that raising user files above total system files doesn't do much good. : ) steve
Re: QMail Performance Question Miscellaneous Issues
On Mon, May 08, 2000 at 10:56:43AM -0400, Bryan White wrote: It's not even that hard, that was the first time I had ever fiddled with the Kernel source. We went to 4096, which should allow for quite a few qmail-remotes. : ) Do you have any feel for how to evaluate what is an optimum number of remotes? At 400 remotes I still have 80% CPU idle time. The load average is around 4. This suggests to me that the system is limited by disk I/O, network I/O or the responsiveness of remote servers. If it is Disk I/O more remotes won't help. I track network traffic through our router fairly closely and don't think that is the bottle neck. If it is the responsiveness of remote servers then more remotes will help. As always, there is a bottleneck. I like to see everything on an email system as a set of queues. While some are tangible, such as the physical mail queue, some are less so, such as the queue of I/O requests to the disks, the queue of packets coming in and out of the interface, the queue of processes to be run, the receiving servers at the other end, etc. It is very rarely the case that all of these queues are in perfect balance such that they all fill up and degrade at the same rate. Consequently there has to be a slowest queue. That's your current bottleneck. the slowest of those queues is going to be your current bottleneck. You fix that, then the next slowest queue becomes your bottleneck. Keep fixing until the first bottleneck comes around again, and you start the whole process over. As you say, in your case the bottleneck may well be disk or network I/O or even a queue at the other end, namely the remote servers. If you have good connectivity and a diverse range of recipients and a large concurrencyremote, then it's probably not the remote servers. But there's only one way to find out. Analyze each queue and find out where it's currently at in terms of utilization. Start tuning the queue with the highest utilization and you'll probably see improvements. Regards.
Re: QMail Performance Question Miscellaneous Issues
"Bryan White" [EMAIL PROTECTED] wrote: Do you have any feel for how to evaluate what is an optimum number of remotes? Measure the delivery rate at various settings of concurrencyremote. Choose the setting that yields the highest delivery rate. At 400 remotes I still have 80% CPU idle time. 400 qmail-remote processes running, or concurrencyremote=400? Some systems never hit their concurrencyremote due to I/O restriction. The load average is around 4. This suggests to me that the system is limited by disk I/O, network I/O or the responsiveness of remote servers. You should be able to monitor disk I/O rates and see if they go up as concurrencyremote is raised. Same with network I/O. ... If it is the responsiveness of remote servers then more remotes will help. So add more remotes. If it helps, you aren't I/O (network or disk) limited yet. -Dave
Re: QMail Performance Question Miscellaneous Issues
I have a question about qmail regarding its mail handling capacity. How many remote emails can qmail send simulataneously, assuming it is run on a Dual-CPU PIII 500Mhz with 512Mb RAM and a SCSI hard disk? The internet bandwidth is 10 Mbps. If I run 2 parallel processes that sends out emails, does it mean twice the amount of emails get sent off simultaneously? if so, does that mean the more processes I run in parallel, the more emails get sent off at the same time? or will most of the emails get stored in the queue? What is the general number of emails that a machine with the above specifications can send per second/hour/day? How do I fine-tune it to send off millions? I only know of changing the "concurrencyremote" figure in /var/qmail/control/concurrencyremote. I set it to 100 for testing. What should be a good figure assuming that I will do free email hosting on the server for hundreds of thousands of users? There are patches available that you will need to apply to achive maximum throughput. You can find them on the qmail web page. One allows you to run much larger queues efficiently by increasing the number of directoies the queue is spread accross. Another allows increasing the maximum number of concurrent remotes beyond 250. The patch allows up to 500 but that limit seems to be linux related. Set /var/qmail/control/timeoutremote to a number lower than the default. I think the default is about 600 (10 minutes). I run it at 120 and have been tempted to lower it to 60. A few weeks ago yahoo.com had a problem where they I could connect to there smtp servers and then it would hand on there side. This means that remote would sit around for the timeout period. 80 or 90 percent of my remotes were tied up in connections to yahoo. I have played with removing flush statements from qmail-queue.c. This dramatically increases the rate at which qmail-inject puts stuff into the queue. This led to very large queues (my sending process backs off when the queue gets to 100K messages). Overall throughput was not improved much. Also if a box does crash then messages will be lost. My guess is your SCSI drive makes this unneeded. I currently run 6 qmail boxes set at 400 remotes. A PIII 650 with 512MB, one IDE disk for the OS and another for /var/qmail gives me about 45-60 K emails per hour. I had similar results with a Dual Celeron 500's but the two boxes I had would crash about once a day. I don't know if there is a problem with dual celerans or not. When I pulled the 2nd CPU out of both boxes they became rock solid. They now get 30 to 40 K emails per hour. Note my email mix is not random. We are sending newsletters / ezines. They are typically 8KB in size and the individual boxes are sending and handling bounces only. All other incomming email comes to another box setup for that purpose.
Re: QMail Performance Question Miscellaneous Issues
On Sun, May 07, 2000 at 08:41:11AM -0400, Bryan White wrote: [snip] I have played with removing flush statements from qmail-queue.c. This dramatically increases the rate at which qmail-inject puts stuff into the queue. This led to very large queues (my sending process backs off when the queue gets to 100K messages). Overall throughput was not improved much. Also if a box does crash then messages will be lost. My guess is your SCSI drive makes this unneeded. The moment qmail-queue says flush, the OS buffers come into play, including a context switch. That takes time, even with SCSI. There is no reliability problem. The first flush() is there to make sure the 'Received' line gets written anyway, to identify potential troublemakers (and neutralize them - blatant rip from Killing in the name). Hmmm... now that I look again, the second flush actually writes the mail out.. I think that is relevant to actual delivery :) Greetz, Peter. -- Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder | | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++
Re: QMail Performance Question Miscellaneous Issues
From: "Steve Wolfe" [EMAIL PROTECTED] Date: Sat, 6 May 2000 21:22:30 -0600 . . . Another question is about the Mail header. What is the header that I should add into a generated email so that undelivered/bounced emails go to this specific email address instead? For example, I send an email to [EMAIL PROTECTED], it will by default bounce back to my sending account. How do I make it bounce to another account instead? Should I use Errors-To:, or Undeliverables-To: or any other header? I've heard of "Bounce-to:" being used, but I'm not hip on the RFC's like others on the list. : ) steve I probably don't qualify as "RFC-hip", but I'd like to be, so this query prompted me to do some digging. The only thing I could find was a mention of "Errors-To:" in RFC2076. Here's what RFC2076 says (bottom of page 10): Address to which notifications Errors-To:,Non-standard, are to be sent and a request to Return-discouraged. get delivery notifications. Receipt-To: Internet standards recommend, however, the use of RCPT TO and Return-Path, not Errors-To, for where delivery notifications are to be sent. So the "official" word would seem to be "use the envelope FROM address." I know that many mailing lists use "Errors-To:", but I'm not sure what percentage of mailers actually use this header. (Of course, qmail does not, nor does the one sendmail installation to which I have access.) [At the top of p11 are a handful of X.400 delivery-related headers, which are probably even less widely implemented.] -- Bob Rogers
QMail Performance Question Miscellaneous Issues
Dear QMail users, I am a QMail newbie and I would like some advice on some qmail performance issues. I have a question about qmail regarding its mail handling capacity. How many remote emails can qmail send simulataneously, assuming it is run on a Dual-CPU PIII 500Mhz with 512Mb RAM and a SCSI hard disk? The internet bandwidth is 10 Mbps. If I run 2 parallel processes that sends out emails, does it mean twice the amount of emails get sent off simultaneously? if so, does that mean the more processes I run in parallel, the more emails get sent off at the same time? or will most of the emails get stored in the queue? What is the general number of emails that a machine with the above specifications can send per second/hour/day? How do I fine-tune it to send off millions? I only know of changing the "concurrencyremote" figure in /var/qmail/control/concurrencyremote. I set it to 100 for testing. What should be a good figure assuming that I will do free email hosting on the server for hundreds of thousands of users? I have also noticed that some free email services like Yahoo also uses QMail (if I'm not mistaken). They have millions of users, so I assume they host the email service on multiple machines. How is it possible to do load-balancing for emails on multiple machines? 'cause everyone will have an email address like [EMAIL PROTECTED], but how does Yahoo redirect portions of users to different machines for mail receival/sending? Another question is about the Mail header. What is the header that I should add into a generated email so that undelivered/bounced emails go to this specific email address instead? For example, I send an email to [EMAIL PROTECTED], it will by default bounce back to my sending account. How do I make it bounce to another account instead? Should I use Errors-To:, or Undeliverables-To: or any other header? Sorry for my huge number of questions. Your prompt help is greatly appreciated. Yours sincerely, Andy.
Re: QMail Performance Question Miscellaneous Issues
I have a question about qmail regarding its mail handling capacity. How many remote emails can qmail send simulataneously, assuming it is run on a Dual-CPU PIII 500Mhz with 512Mb RAM and a SCSI hard disk? The internet bandwidth is 10 Mbps. A lot. : ) There is a hard-coded limit to the numebr of concurrent remote sessions, somewhere around 200, if I recall, but you can certainly increase that in the source, if necessary. Below 200, it is controlled by a "soft limit" contained in the file "concurrencyremote". I'm sure that your machine could handle 200 with ease. If I run 2 parallel processes that sends out emails, does it mean twice the amount of emails get sent off simultaneously? if so, does that mean the more processes I run in parallel, the more emails get sent off at the same time? or will most of the emails get stored in the queue? As long as you aren't maxing out CPU or bandwidth, then yes, sending them in parallel is better. That way, things like waiting for DNS lookups don't kill your performance. That's why qmail spawns a new instance of qmail-remote for each message to be delivered. What is the general number of emails that a machine with the above specifications can send per second/hour/day? How do I fine-tune it to send off millions? I only know of changing the "concurrencyremote" figure in /var/qmail/control/concurrencyremote. I set it to 100 for testing. What should be a good figure assuming that I will do free email hosting on the server for hundreds of thousands of users? I'd watch your bandwidth, but I'd bet that 200 would be perfectly reasonable. I've raised it to 100 on a Pentium 233 w/ 512k bandwidth, and it definitely speeded things up quite a bit. 500 emails took ~45 seconds to clear, perhaps a bit less. That's about a million per day. With your system, you should be able to handle more, although sooner or later, you'll hit the major limiting factor: I/O. It sounds like you're just starting up, so you should be fine for some time. Once things get severely large, you'll need to at least go to a hardware RAID setup, with multiple disks on multiple channels/controllers to get the throughput. Another question is about the Mail header. What is the header that I should add into a generated email so that undelivered/bounced emails go to this specific email address instead? For example, I send an email to [EMAIL PROTECTED], it will by default bounce back to my sending account. How do I make it bounce to another account instead? Should I use Errors-To:, or Undeliverables-To: or any other header? I've heard of "Bounce-to:" being used, but I'm not hip on the RFC's like others on the list. : ) steve
