Re: Can Qmail send out 2 million mails in 12 hour window?
On Tue, Feb 27, 2001 at 08:11:10PM +0100, Peter van Dijk wrote: > As long as you are injecting messages, qmail won't perform at full > speed. Play with that rate, maybe no limiting *is* the best option. Maybe an idea would also be to "disable" the trigger mechanism in qmail-queue/qmail-send (changing permission on trigger would be sufficient) and change qmail-send's sleep timeout to some 60 seconds. This can be easily done by changing qmail-send: #define SLEEP_TODO 1500 /* check todo/ every 25 minutes in any case */ This would cause qmail to "bulk", i.e. scan todo and organize, send out the mails, and then start again. With that one could measure (system and bulk job dependant) how many emails qmail can send out in a certain interval. Then one could synch the SLEEP_TODO and the number of injects per SLEEP_TODO. \Maex P.S. as some ppl wondered ... no, i didn't get lost ;-) only had no time to read the list which bestowed me 1200 unread messages :/ but I'm nearly though ;-) -- SpaceNet AG| Joseph-Dollinger-Bogen 14 | Fon: +49 (89) 32356-0 Research & Development | D-80807 Muenchen| Fax: +49 (89) 32356-299 Stress is when you wake up screaming and you realize you haven't fallen asleep yet.
Re: Can Qmail send out 2 million mails in 12 hour window?
On Tue, Feb 27, 2001 at 08:39:28PM +0100, Jurjen Oskam wrote: > Could you please point to some documentation about how this is done? > > I checked cr.yp.to and www.ezmlm.org but couldn't find it. (I have not > searched extremely well though, so ofcourse I could have missed it.) Look for VERP. Greetz, Peter.
Re: Can Qmail send out 2 million mails in 12 hour window?
On Tue, Feb 27, 2001 at 08:11:10PM +0100, Peter van Dijk wrote: > ezmlm is an instantly-working mailinglist tool, that saves you time > coding. It also injects a message into the queue *once*, which means > there is no I/O problem. qmail then delivers this *one* message to > *all* recipients, in such a way that ezmlm can do reliable > bouncehandling. Could you please point to some documentation about how this is done? I checked cr.yp.to and www.ezmlm.org but couldn't find it. (I have not searched extremely well though, so ofcourse I could have missed it.) Thanks. -- Jurjen Oskam * http://www.stupendous.org/ for PGP key * Q265230 pro-life bombing bush hacker attack USA president 2600 decss assassinate nuclear strike terrorism gun control eta military disrupt economy encryption 7:40pm up 2 days, 22:45, 2 users, load average: 0.08, 0.02, 0.01
RE: Can Qmail send out 2 million mails in 12 hour window?
On Tue, 27 Feb 2001, Brandon Yu wrote: > The messages themselves have been pre-generated and exist as a file and is > qmail-injected to place them in the queue. The message is the same across > the board with the exception of some personalization, such as the name. > Since there are so many messages, we use a perl script to place them in the > queue at a certain rate, i.e. 50 msgs/second, or whatever rate we choose. By > injecting them at this rate, we can see whether qmail can keep up with our > intended rate. With this in mind, does this lessen the burden of disk I/O? Don't use qmail-inject unless you have to. Use qmail-remote directly. If qmail-remote reports a transient failure, then inject them into the queue for later delivery. If qmail-remote reports a permanent failure then the message cannot be delivered so it can be ignored. See the qmail-remote man page for driving instructions. Using qmail-inject will increase your disk I/O and impose qmails concurrency limits. If you use qmail-remote directly you can bybass these limits and impose your own :) -- Regards Peter -- Peter Samuel[EMAIL PROTECTED] http://www.e-smith.org (development)http://www.e-smith.com (corporate) Phone: +1 613 368 4398 Fax: +1 613 564 7739 e-smith, inc. 1500-150 Metcalfe St, Ottawa, ON K2P 1P1 Canada "If you kill all your unhappy customers, you'll only have happy ones left"
Re: Can Qmail send out 2 million mails in 12 hour window?
On Tue, Feb 27, 2001 at 10:13:28AM -0800, Brandon Yu wrote: > The messages themselves have been pre-generated and exist as a file and is > qmail-injected to place them in the queue. The message is the same across > the board with the exception of some personalization, such as the name. > Since there are so many messages, we use a perl script to place them in the > queue at a certain rate, i.e. 50 msgs/second, or whatever rate we choose. By > injecting them at this rate, we can see whether qmail can keep up with our > intended rate. With this in mind, does this lessen the burden of disk I/O? As long as you are injecting messages, qmail won't perform at full speed. Play with that rate, maybe no limiting *is* the best option. > I have heard of DNSCACHE, currently BIND is running directly on those > machines. Would it be worthwhile changing out? Can I expect to relive a > bottleneck in this process? If you're sending out that many mails, I wouldn't be surprised if BIND kept on caching all data and eventually filled all available memory. Since BIND doesn't limit it's memory use, it will grow as long as records don't expire. Extreme growth breaks your server, less extreme growth still limits the amount of memory your OS can dedicate to disk caching and buffering, which hurts performance. Use dnscache :) > Why would you avoid sorting by domain? I would think it would be more > efficient handshaking. If you insert 10.000 messages to the same domain together, qmail will be spending lots of time on all these messages before it handles messages for another domain. If this domain is down, it will bottleneck the rest. If you don't care about domains at all, it's much harder for a single domain to use all your remote concurrency. > No, I am not using EZMLM. How could I benefit from using it? ezmlm is an instantly-working mailinglist tool, that saves you time coding. It also injects a message into the queue *once*, which means there is no I/O problem. qmail then delivers this *one* message to *all* recipients, in such a way that ezmlm can do reliable bouncehandling. Greetz, Peter.
RE: Can Qmail send out 2 million mails in 12 hour window?
Brandon Yu <[EMAIL PROTECTED]> wrote: >The messages themselves have been pre-generated and exist as a file and is >qmail-injected to place them in the queue. The message is the same across >the board with the exception of some personalization, such as the name. Ouch. That's gonna cost you dearly. >Since there are so many messages, we use a perl script to place them in the >queue at a certain rate, i.e. 50 msgs/second, or whatever rate we choose. By >injecting them at this rate, we can see whether qmail can keep up with our >intended rate. With this in mind, does this lessen the burden of disk I/O? You're going to run into the qmail-send bottleneck: qmail-send won't dispatch messages to qmail-local or qmail-remote while there are messages in queue/todo. So not much will go out until all 2M messages are injected. Therefore, throttling the injections is counterproductive. >I have heard of DNSCACHE, currently BIND is running directly on those >machines. Would it be worthwhile changing out? Can I expect to relive a >bottleneck in this process? The only bottleneck you need to relieve is the current bottleneck. If/when you get to the point that DNS lookups are the bottleneck, then dnscache would be the thing to try. Your first bottleneck is going to be qmail-send handling new injections. After that, it'll be disk I/O or concurrencyremote, most likely. >Why would you avoid sorting by domain? I would think it would be more >efficient handshaking. qmail-send is going to pass each delivery to a separate qmail-remote. Sorting will only cause qmail to pound on each domain sequentially. If your first 2000 subscribers are on AOL and your concurrencyremote is 400, you're going to initiate 400 simultaneous connections to AOL right off the bat. >No, I am not using EZMLM. How could I benefit from using it? If you need customized messages, you can't. If you could use it, you'd benefit by automatic bounce handling and much more efficient delivery--one message with 2M recipients vs. 2M messages with 1 recipient. -Dave
RE: Can Qmail send out 2 million mails in 12 hour window?
The messages themselves have been pre-generated and exist as a file and is qmail-injected to place them in the queue. The message is the same across the board with the exception of some personalization, such as the name. Since there are so many messages, we use a perl script to place them in the queue at a certain rate, i.e. 50 msgs/second, or whatever rate we choose. By injecting them at this rate, we can see whether qmail can keep up with our intended rate. With this in mind, does this lessen the burden of disk I/O? I have heard of DNSCACHE, currently BIND is running directly on those machines. Would it be worthwhile changing out? Can I expect to relive a bottleneck in this process? Why would you avoid sorting by domain? I would think it would be more efficient handshaking. No, I am not using EZMLM. How could I benefit from using it? Brandon > -Original Message- > From: John White [mailto:[EMAIL PROTECTED]] > Sent: Monday, February 26, 2001 10:17 PM > To: Brandon Yu > Subject: Re: Can Qmail send out 2 million mails in 12 hour window? > > > On Mon, Feb 26, 2001 at 04:28:43PM -0800, Brandon Yu wrote: > > I have been given the task to send out 2 million emails in > a 12 hour time > > window. All the emails will be sent remotely, to a list of > users of which is > > 90% accurate (I figure 10% of the emails will bounceback > because of bad > > email addresses) I have all the bandwidth I need (servers > are located in > > co-location) and will be sorting the email list by domain name. > > > > My initial idea is to have 2 dedicated qmail servers, ( > Redhat Linux 6.2, > > Pentium 600, 500Megs RAM, IDE drives) configured with a > concurrency limit of > > 400. Other than that, the qmail install will be out of the box. > > The thing you're going to run up against is disk i/o. > > Are the message bodies customized to the recipient? If so, > you'll have > problems getting all the messages into the queue. Is this the case? > > If not, the best way is to send one message with all the recipients. > One way to do this is to call qmail-queue directly. Another way is to > call qmail-inject with the flag that tells it to look for > recipients in > the header, and attach each recipient as a BCC header. > > I would avoid sorting by domain. > > I would have dnscache installed on the qmail machines. > > Are you using list management software like ezmlm? > > John >
Re: Can Qmail send out 2 million mails in 12 hour window?
On Mon, Feb 26, 2001 at 04:28:43PM -0800, Brandon Yu wrote: >Can I reasonably meet this rate ? Do you have any suggestions? It largely depends on the actual number of bounces and where the remote machines are (if you're sending to a lot of international addresses, for example). 2 million recipients in 12 hours means that (based on my experience) you'll be running at peak capacity of two high-end machines. If you *HAVE* to hit the 12-hour window, I'd recommend you look at a third machine or possibly thinking about ways to take advantage of some tricks to get a bit better performance. Sean -- Program *INTO* a language, not *IN* it. -- David Gries Sean Reifschneider, Inimitably Superfluous <[EMAIL PROTECTED]> tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
Re: Can Qmail send out 2 million mails in 12 hour window?
On Mon, Feb 26, 2001 at 04:28:43PM -0800, Brandon Yu wrote: > I have been given the task to send out 2 million emails in a 12 hour time > window. All the emails will be sent remotely, to a list of users of which is > 90% accurate (I figure 10% of the emails will bounceback because of bad > email addresses) I have all the bandwidth I need (servers are located in > co-location) and will be sorting the email list by domain name. > > My initial idea is to have 2 dedicated qmail servers, ( Redhat Linux 6.2, > Pentium 600, 500Megs RAM, IDE drives) configured with a concurrency limit of > 400. Other than that, the qmail install will be out of the box. > > Can I reasonably meet this rate ? Do you have any suggestions? A higher concurrency *may* be beneficial. It's not gonna hurt, anyway. I'm assuming you are using ezmlm or something similar for this. Anything else would be suicide. Greetz, Peter.
Can Qmail send out 2 million mails in 12 hour window?
I have been given the task to send out 2 million emails in a 12 hour time window. All the emails will be sent remotely, to a list of users of which is 90% accurate (I figure 10% of the emails will bounceback because of bad email addresses) I have all the bandwidth I need (servers are located in co-location) and will be sorting the email list by domain name. My initial idea is to have 2 dedicated qmail servers, ( Redhat Linux 6.2, Pentium 600, 500Megs RAM, IDE drives) configured with a concurrency limit of 400. Other than that, the qmail install will be out of the box. Can I reasonably meet this rate ? Do you have any suggestions? Thanks in advance
