Re: Lotsa messages with qmail-remote?
Mark Delany wrote: On Wed, May 16, 2001 at 02:38:38PM -0400, John R Levine wrote: I have a spam-like application that will be sending out thousands of customized single-recipient messages. (It's spam-like because it says you wrote to us about on , but unlike spam, they really did write and I have the saved messages to prove it.) Rather than dumping them all into qmail-inject or qmail-queue which would cause constipation unless I install the big-todo patch which is a pain, I was thinking of calling qmail-remote directly, then qmail-queue if qmail-remote didn't work, with a bunch of remotes going at once. The addresses come out of a database and the customization is trivial, so I was planning to write it in perl. (The main bottleneck is the network delays for qmail-remote.) But before I do, has someone already written this? I recently did one of these - it was more designed for mass customized mailings and used a pool of sender servers and a distributed queue - we're talking millions and millions of email per day here... It's a complex system and I haven't the code, but I have some experiences that I can share. I used IO::select to handle running multiple qmail-remotes at the same time. qmail-remote has a really small footprint so you can run 1000s of them concurrently on a modest sized server. It takes a fair amount of code to manage multiplexed pipes in conjunction with handling stdout and stderr (execution errors) responses and exit conditions. (I see that there is an IO::Poll in which case I'd probably use that in preference to IO::Select because of some of the select limit issues on some OSes). Can you shed any more light on this. I am very interested as I may write something similar soon, and any ideas / help would be much appreciated. The next thing you have to worry about is managing your own queue and retries for delivery failures. This can be much simpler and faster than a full qmail-send type queue of course, such as a single flat file for the whole delivery run with an occassional sync. Here I was thinking of a just using a local qmail install (keep things simple) Regards.
Re: Lotsa messages with qmail-remote?
On Thu, May 17, 2001 at 08:29:37AM +, Greg Cope wrote: I used IO::select to handle running multiple qmail-remotes at the same time. qmail-remote has a really small footprint so you can run 1000s of them concurrently on a modest sized server. It takes a fair amount of code to manage multiplexed pipes in conjunction with handling stdout and stderr (execution errors) responses and exit conditions. (I see that there is an IO::Poll in which case I'd probably use that in preference to IO::Select because of some of the select limit issues on some OSes). Can you shed any more light on this. I am very interested as I may write something similar soon, and any ideas / help would be much appreciated. Well, that's more a perl/Unix issue than a qmail one so this isn't the right place to discussed it. If you're asking about the benefits of poll vs select, there is plenty of material on the net about this. (Now if kqueue gets into enough Unixen and someone write a perl interface for it, well, that'd be something to talk about : ) Regards.
Re: Lotsa messages with qmail-remote?
Mark Delany wrote: On Thu, May 17, 2001 at 08:29:37AM +, Greg Cope wrote: I used IO::select to handle running multiple qmail-remotes at the same time. qmail-remote has a really small footprint so you can run 1000s of them concurrently on a modest sized server. It takes a fair amount of code to manage multiplexed pipes in conjunction with handling stdout and stderr (execution errors) responses and exit conditions. (I see that there is an IO::Poll in which case I'd probably use that in preference to IO::Select because of some of the select limit issues on some OSes). Can you shed any more light on this. I am very interested as I may write something similar soon, and any ideas / help would be much appreciated. Well, that's more a perl/Unix issue than a qmail one so this isn't the right place to discussed it. If you're asking about the benefits of poll vs select, there is plenty of material on the net about this. (Now if kqueue gets into enough Unixen and someone write a perl interface for it, well, that'd be something to talk about : ) What I was interested in was using perl to drive qmail-remote, not a discussion of poll vs select, although that would be handy. Greg Regards.
Re: Lotsa messages with qmail-remote?
Can you shed any more light on this. I am very interested as I may write something similar soon, and any ideas / help would be much appreciated. Well, that's more a perl/Unix issue than a qmail one so this isn't the right place to discussed it. If you're asking about the benefits of poll vs select, there is plenty of material on the net about this. (Now if kqueue gets into enough Unixen and someone write a perl interface for it, well, that'd be something to talk about : ) What I was interested in was using perl to drive qmail-remote, not a discussion of poll vs select, although that would be handy. Well, it's no different from running any other program within perl. The interface to qmail-remote is completely documented in the qmail-remote manpage. The only trap is that you cannot use open(... |qmail-remote) as you need to set up a bi-directional pipes. I did it the hard way with fork/exec and manipulated the fds, but you could possibly use IPC::Open2 available from your friendly CPAN server. But this is mostly perl/Unix talk, not qmail. Regards.
Lotsa messages with qmail-remote?
I have a spam-like application that will be sending out thousands of customized single-recipient messages. (It's spam-like because it says you wrote to us about on , but unlike spam, they really did write and I have the saved messages to prove it.) Rather than dumping them all into qmail-inject or qmail-queue which would cause constipation unless I install the big-todo patch which is a pain, I was thinking of calling qmail-remote directly, then qmail-queue if qmail-remote didn't work, with a bunch of remotes going at once. The addresses come out of a database and the customization is trivial, so I was planning to write it in perl. (The main bottleneck is the network delays for qmail-remote.) But before I do, has someone already written this? Regards, John Levine, [EMAIL PROTECTED], Primary Perpetrator of The Internet for Dummies, Information Superhighwayman wanna-be, http://iecc.com/johnl, Sewer Commissioner Finger for PGP key, f'print = 3A 5B D0 3F D9 A0 6A A4 2D AC 1E 9E A6 36 A3 47
Re: Lotsa messages with qmail-remote?
John R Levine [EMAIL PROTECTED] wrote: I have a spam-like application that will be sending out thousands of customized single-recipient messages. (It's spam-like because it says you wrote to us about on , but unlike spam, they really did write and I have the saved messages to prove it.) Out of curiosity, does the above boil down to this is not unsolicited email, or to the users wrote to me, and according to the TOS they clicked through, this gives me the right to send them commercial mail? Rather than dumping them all into qmail-inject or qmail-queue which would cause constipation unless I install the big-todo patch which is a pain, I was thinking of calling qmail-remote directly, then qmail-queue if qmail-remote didn't work, with a bunch of remotes going at once. This scheme is used elsewhere, and has been discussed in the qmail list in the past. It is frequently recommended when people ask about multi-million per-recipient customized mailings. Check the list archives for other details. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Lotsa messages with qmail-remote?
On Wed, May 16, 2001 at 02:38:38PM -0400, John R Levine wrote: I have a spam-like application that will be sending out thousands of customized single-recipient messages. (It's spam-like because it says you wrote to us about on , but unlike spam, they really did write and I have the saved messages to prove it.) Rather than dumping them all into qmail-inject or qmail-queue which would cause constipation unless I install the big-todo patch which is a pain, I was thinking of calling qmail-remote directly, then qmail-queue if qmail-remote didn't work, with a bunch of remotes going at once. The addresses come out of a database and the customization is trivial, so I was planning to write it in perl. (The main bottleneck is the network delays for qmail-remote.) But before I do, has someone already written this? I recently did one of these - it was more designed for mass customized mailings and used a pool of sender servers and a distributed queue - we're talking millions and millions of email per day here... It's a complex system and I haven't the code, but I have some experiences that I can share. I used IO::select to handle running multiple qmail-remotes at the same time. qmail-remote has a really small footprint so you can run 1000s of them concurrently on a modest sized server. It takes a fair amount of code to manage multiplexed pipes in conjunction with handling stdout and stderr (execution errors) responses and exit conditions. (I see that there is an IO::Poll in which case I'd probably use that in preference to IO::Select because of some of the select limit issues on some OSes). The next thing you have to worry about is managing your own queue and retries for delivery failures. This can be much simpler and faster than a full qmail-send type queue of course, such as a single flat file for the whole delivery run with an occassional sync. Bounces of course you'll handle with some sort of VERP address. Having said all that, are you talking less than, say, 10,000 mails? If so, one simple strategy is to inject each mail at the rate of say 1 per second. At that rate 1000 mails are injected in about 16 minutes, ten thousand in a little less than 3 hours. That sort of injection rate should not require bigtodo patches so if you don't mind your delivery script running for 3 hours, then that might be the easiest strategy. Regards.
