fifo smtproutes Re: Qmail and Large Scale Dynamic Mailings
what if /var/qmail/control/smtproutes was replaced with a fifo that
gave a different relay every time it was read?
#!/usr/local/bin/perl
while(++$count){
unlink '/var/qmail/control/smtproutes';
system 'mkfifo /var/qmail/control/smtproutes';
$c=$count % 5; # or however many there are
open R,"/var/qmail/control/smtproutes"; #block until it is read
print R ":bsdrelay$c.macrosys.com\n";
};
__END__
Will the above cause unexpected freezes?
A less intensive solution might be to overwrite the tenth
character in the static file every few seconds, to load up that
relay.
"Collin B. McClendon" wrote:
Hello,
Sounds good.
Thanks,
Collin
-Original Message-
From: David L. Nicol [mailto:[EMAIL PROTECTED]]
Sent: Thursday, January 11, 2001 9:58 AM
To: Collin B. McClendon
Subject: Re: Qmail and Large Scale Dynamic Mailings
several slave BSD boxes with high concurrencies and a hacked qmail-remote
that round-robins through them.
--
David Nicol 816.235.1187 [EMAIL PROTECTED]
"people with fish eyes and brown socks"
Re: fifo smtproutes Re: Qmail and Large Scale Dynamic Mailings
On Thu, Jan 11, 2001 at 03:46:55PM +, David L. Nicol wrote:
what if /var/qmail/control/smtproutes was replaced with a fifo that
gave a different relay every time it was read?
#!/usr/local/bin/perl
while(++$count){
unlink '/var/qmail/control/smtproutes';
system 'mkfifo /var/qmail/control/smtproutes';
$c=$count % 5;# or however many there are
open R,"/var/qmail/control/smtproutes"; #block until it is read
print R ":bsdrelay$c.macrosys.com\n";
};
__END__
Will the above cause unexpected freezes?
Perhaps not, but unexpected results:
1. Nothing stops a pipe being opened by multiple readers at once. What
will the second and subsequent reader get? Almost certainly eof with
no data.
2. You have a timing window between the mkfifo and the unlink where a
qmail-remote instance will find no smtproutes file.
Doesn't plan9 have something that does this properly, where you open a
file system object and each open gets a matching process on the other
side?
Regards.
RE: Qmail and Large Scale Dynamic Mailings
Hello All, I'm having wierd issues with my large mailing lists. I have one with 9k and one with 22k of subscribers. I've applied big-concurrency, but not the sysctl.conf changes mentioned here. I had this machine serving just fine with OpenBSD but after adding listserv I had to change OS's. I have noticed that although we haven't added any subsribers to the first list and just a few to the second it has slowly over the last week gotten slower and slower at delivering. I am running RedHat Linux 7.0 with Q-mail and Listserv. I am using a concurrency both local and remote of 255. I increased __FD_SETSIZE to 4096 and conf-spawn to 255. Unlike in BSD I couldn't get the concurrency raised past 255. Any suggestions? Is my que perhaps not flushing over time or perhaps file system fragmentation? (stupid thing to say I suppose with ext2...) I'd appreciate any suggestions, Thanks, Collin -Original Message- From: Sean Reifschneider [mailto:[EMAIL PROTECTED]] Sent: Thursday, December 07, 2000 8:21 PM To: Thomas Duterme Cc: [EMAIL PROTECTED] Subject: Re: Qmail and Large Scale Dynamic Mailings On Wed, Dec 06, 2000 at 11:45:49AM -0500, Thomas Duterme wrote: 2) Qmail is set to open 20 SMTP connections at a time Ding ding ding ding! We have a winner! Try at least doubling that. Unfortunately, you can't say "20 per destination" or "20 per domain", but setting it to "20 total" is really going to kill performance. Going from 240 to 20 may have been a bit of an over-reaction. Can you try 120? That'll help a LOT. If you have to leave it at 20, you may want to turn your SMTP timeout down to like 5 seconds at the beginning of a mailing, so all the slow to respond mail servers are bypassed, then increase it to like 30 after an hour or two, then to 200 after another hour or two and do a "killall -ALRM qmail-send". Wacky, but it might get the job done... Sean -- Why are Bush supporters acting like they won, when Gore has 350,000 more popular votes? Sean Reifschneider, Inimitably Superfluous [EMAIL PROTECTED] tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
Re: Qmail and Large Scale Dynamic Mailings
if the newsletters are all the same, you could pre-process the list, organizing by recipent domain and starting qmail-remote processes with a few dozen recipients, and only give qmail the ones that don't go through on the first attempt, and even then after some back-off time. That way you won't clog up all incoming channels somewhere, denying service, and getting barred. Some perl code that can work with qmail-remote is available at http://www.davidnicol.com/qmail.html You could open one channel per recipent domain by inserting fork and next; right after the for(keys %Recipients) line. Adding a random backoff timer, something like sleep(rand 3600) before invoking qmail-inject might ease things too, as long as your machine has enough swap space to have all these forked perls sitting around sleeping. YMMV. Henning Brauer wrote: Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. Hmm, even with 20 concurrent connections our servers was blocked by some braindead freemailer's servers when one of our customers sent out a newsletter... I don't think there is a common "highest polite limit", you have to figure it out for your country, even for your typical recipients. Greetings Henning -- Henning Brauer | BS Web Services Hostmaster BSWS| Roedingsmarkt 14 [EMAIL PROTECTED] | 20459 Hamburg www.bsws.de| Germany -- David Nicol 816.235.1187 [EMAIL PROTECTED] idamfino/i
Re: Qmail and Large Scale Dynamic Mailings
Sean Reifschneider [EMAIL PROTECTED] writes: Ding ding ding ding! We have a winner! Try at least doubling that. Unfortunately, you can't say "20 per destination" or "20 per domain", but setting it to "20 total" is really going to kill performance. Going from 240 to 20 may have been a bit of an over-reaction. Can you try 120? That'll help a LOT. If there's one domain in particular which tends to croak or deny your connections if you make too many connections, you could also set up a separate qmail installation with a low concurrencyremote and use smtproutes to route all mails to that domain via that installation. This can be done either as a separate server or as a separate installation with its own queue directory on your current server (although in that case it must of course listen on another port, and you'll need to include the port number in your smtproutes entry). It's an ugly kludge, but it works. It also has the advantage that the mails to other domains won't be held up while qmail is trying to process the queue for the domain which is having trouble. -- "I live in the heart of the machine. We are one."
Re: Qmail and Large Scale Dynamic Mailings
I feel our discussion starts to become OT. I tried to be calm and I won't discuss it any more after this. From: "Ken Jones" [EMAIL PROTECTED] Ah.. So you are not a spammer, except you assume all your "customers" want your email. Besides that moral issue, do you have measured information about the delivery statistics of qmail version other options? I said "subscriber", that means we send e-mail to users who willingly "subscribe" our newsletters on our web page. We don't send unsolicited emails to people who did not subscribe our newsletters. We surveyed other email options. We chose qmail because the information we gathered told us that qmail is one of the fastest MTA sending outgoing email. And we like the VERPs capability of qmail. With it we could effectively eliminate expired addresses. Are you saying that your country is a pirate economy? Shall I embarrase you and trace your email? or do you wish to reveal which country you are from? Does your country harbor pirates? If you so insist, I am from Taiwan. Here is my company: http://www.etwebs.com/ http://www.etmc.com.tw/ These pages are writen in Chines language. Welcome to give us a little more pageview count. Sir. If your users are on your machines, you can just copy your emails to thier directories. But.. it seems like your users are spread over other peoples machines. Hence.. by definition, you are a spammer, sir. What's your definition of "spammer"? If you mean anyone who send large quantity of email on the net, regardless unsolicited or solicited. I feel that this classification may not be fair. I don't know why you are so harsh at this issue. Maybe you have seen a lot of hateful spammers in your email admin career? But I say it again, we don't want to cause any people annoyance. If you feel discussing mass-mailing techniques on the list will benefit or encourage potential spammers. I agree. I did not thought about this danger at first. But please don't berascal people (and their country!) who just ask questions. Thank you. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
Wayne Chu wrote: I feel our discussion starts to become OT. I tried to be calm and I won't discuss it any more after this. Obviously this is a hot topic for you. So, without furthre evidence, you are obviously a person who will exploit the edge of spam related email. I have a question for you Sir. Are you going to be sending these "emails" to your local list of accounts. Or will this be going out to other lists? How have you received these lists of emails? Ken Jones From: "Ken Jones" [EMAIL PROTECTED] Ah.. So you are not a spammer, except you assume all your "customers" want your email. Besides that moral issue, do you have measured information about the delivery statistics of qmail version other options? I said "subscriber", that means we send e-mail to users who willingly "subscribe" our newsletters on our web page. We don't send unsolicited emails to people who did not subscribe our newsletters. We surveyed other email options. We chose qmail because the information we gathered told us that qmail is one of the fastest MTA sending outgoing email. And we like the VERPs capability of qmail. With it we could effectively eliminate expired addresses. Are you saying that your country is a pirate economy? Shall I embarrase you and trace your email? or do you wish to reveal which country you are from? Does your country harbor pirates? If you so insist, I am from Taiwan. Here is my company: http://www.etwebs.com/ http://www.etmc.com.tw/ These pages are writen in Chines language. Welcome to give us a little more pageview count. Sir. If your users are on your machines, you can just copy your emails to thier directories. But.. it seems like your users are spread over other peoples machines. Hence.. by definition, you are a spammer, sir. What's your definition of "spammer"? If you mean anyone who send large quantity of email on the net, regardless unsolicited or solicited. I feel that this classification may not be fair. I don't know why you are so harsh at this issue. Maybe you have seen a lot of hateful spammers in your email admin career? But I say it again, we don't want to cause any people annoyance. If you feel discussing mass-mailing techniques on the list will benefit or encourage potential spammers. I agree. I did not thought about this danger at first. But please don't berascal people (and their country!) who just ask questions. Thank you. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
Ken Jones [EMAIL PROTECTED] writes: I appologize to easily catorgorize this person as a spammer. Howver, the bulk of email produced on the internet is from spam sites. Jenny, how much experience do you have with spam sites? have you designed, operated and maintained a spam site? I am postmaster at one of the largest ISPs in Sweden, and I have been working with antispam issues since 1995. I certainly have never been in any way involved in designing, operating and maintaining spam sites. But I am involved in running a mailinglist server, for confirmed opt-in mailinglists, and find the issues of how to handle mailinglist traffic in a non-abusive way quite important. Consider, for instance, that one of our mailinglist (say, the one about cat care and feeding) has a whole lot of subscribers from one of our competitors - not an unlikely scenario for a Swedish mailinglist, as there aren't really that many big providers here. When a listmail goes out, depending on our concurrencyremote settings , it's entirely possible that that provider's mailserver will be tied up handling the list mail for quite some time. This might upset them - I know I'd get upset if someone was doing it to my servers. In fact, I've been forced to temporarily block maillist servers which behaved that way - not because the mails themselves were spam, but because the behaviour of the listserver caused problems for my servers. The original poster was trying to discuss how to handle this very issue, as an MTA which makes one connection per address may quite easily cause such problems. Thus, his question was quite on-topic for this list. Your reply was not. However, he has not shown any evidence that he is, or is not a spammer. Exactly. Shall we all now agreee that anyone who sends out bulk emails is not a spammer? No. Anyone who sends out *unsolicited* bulk email is a spammer. The original poster has repeatedly stated that he is working with solicited email. Unless you have any reason to doubt this, please stop making false statements. I see a lack of evidence that this person is not a spammer. Why else would a person be sending out large amounts of emails to non local domains How about because they run confirmed opt-in mailinglists which are interesting to a lot of people outside their own domains? However, this has nothing to do with qmail. If you have any further issues with bulk emailing, I suggest you take them to SPAM-L or some other forum where it is on-topic. -- "I live in the heart of the machine. We are one."
Re: Qmail and Large Scale Dynamic Mailings
Jenny Holmberg wrote: Ken Jones [EMAIL PROTECTED] writes: I appologize to easily catorgorize this person as a spammer. Howver, the bulk of email produced on the internet is from spam sites. Jenny, how much experience do you have with spam sites? have you designed, operated and maintained a spam site? I am postmaster at one of the largest ISPs in Sweden, and I have been working with antispam issues since 1995. I certainly have never been in any way involved in designing, operating and maintaining spam sites. But I am involved in running a mailinglist server, for confirmed opt-in mailinglists, and find the issues of how to handle mailinglist traffic in a non-abusive way quite important. A very valid point of view. However you do not offer any proof of the prior persons orignal posting. Of course there are valid reasons why we should develop the most effiecent method to deliver large quantities of email. Of course. This is part of most large ISP's daily activies. Consider, for instance, that one of our mailinglist (say, the one about cat care and feeding) has a whole lot of subscribers from one of our competitors - not an unlikely scenario for a Swedish mailinglist, as there aren't really that many big providers here. When a listmail goes out, depending on our concurrencyremote settings , it's entirely possible that that provider's mailserver will be tied up handling the list mail for quite some time. This might upset them - I know I'd get upset if someone was doing it to my servers. In fact, I've been forced to temporarily block maillist servers which behaved that way - not because the mails themselves were spam, but because the behaviour of the listserver caused problems for my servers. Excuse me, My point was not a valid email list. My point was the original poster did not present a valid argument why this activity should occur. We are not stupid. We know this type of activity is valid for a norrmal ISP. My point is: shall we provide technology to pirates? Please excuse me for deleting the rest of yoru email.. I agree with you on your points. My only disagreement is that we are handing away exploitive technology to abusive people. Not only are we giving away exploitive technology, we are helping them become rich. And hence offering them a step up into the uppper reaches of society. Do you want to support pornographers becoming the benefactors in your local town. I do not want to. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
On Thu, Dec 07, 2000 at 12:39:26AM -0600, Ken Jones wrote:
Wayne Chu wrote:
No, I am NOT spamming.
Our company runs serveral daily e-newsletter, with totally about a
half million of subscribers. We are planning to make an "open"
newsletter plateform, let our web site members create their own
personal newsletter ( authenticated and supervised by our staff to
prevent spam mail ). we estimated the total number of subscribers and
the number of newsletter will grow even more. Surely our member
would want their newsletters to be sent ASAP. So we have to
increase concurrency.
Ah.. So you are not a spammer, except you assume all your
"customers" want your email. Besides that moral issue,
do you have measured information about the delivery
statistics of qmail version other options?
If these are **personalized** emails, then how they get queued will
depend on how they are built. One person wants a,g,f,t,z and
another wants a,t,q,p,z. If they want them ASAP, then it's not
obvious concurrency is an issue at all; perhaps newsletters with
p go out whenever p changes and triggers them. Maybe changes in
some of the topics do not trigger new email, but wait on a clock,
etc In either case, they may still be different per subscriber
because of the other selections (else they are not personalized).
Even with half a million subscribers, depending on the number of
options and the subscriber base you might still be sending only
a very few messages to each site. And if they are truly personalized,
it's not at all clear to me how one would build them so they could
go in the queue more than one by one.
Roughly:
SELECT email_address, topics FROM subscribers WHERE email ='$1' ORDER
BY topics,email_address;
for (@email_address) {
build message(@topics);
qmail-inject message # we do these ourselves one-by-one; personalized
}
Qmail will not be the hard part :-)
If, OTOH, they are a personal selection of independant, non-personalized
messages, batching will make sense - but that's not "personalized".
As usual, a clear statement of the situation will help.
cfm
--
Christopher F. Miller, Publisher [EMAIL PROTECTED]
MaineStreet Communications, Inc 208 Portland Road, Gray, ME 04039
1.207.657.5078 http://www.maine.com/
Content management, electronic commerce, internet integration, Debian linux
RE: Qmail and Large Scale Dynamic Mailings
Could someone direct me to the instructions to un-subscribe from this list?
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 07, 2000 9:02 AM
To: Ken Jones
Cc: Wayne Chu; [EMAIL PROTECTED]
Subject: Re: Qmail and Large Scale Dynamic Mailings
On Thu, Dec 07, 2000 at 12:39:26AM -0600, Ken Jones wrote:
Wayne Chu wrote:
No, I am NOT spamming.
Our company runs serveral daily e-newsletter, with totally about a
half million of subscribers. We are planning to make an "open"
newsletter plateform, let our web site members create their own
personal newsletter ( authenticated and supervised by our staff to
prevent spam mail ). we estimated the total number of subscribers and
the number of newsletter will grow even more. Surely our member
would want their newsletters to be sent ASAP. So we have to
increase concurrency.
Ah.. So you are not a spammer, except you assume all your
"customers" want your email. Besides that moral issue,
do you have measured information about the delivery
statistics of qmail version other options?
If these are **personalized** emails, then how they get queued will
depend on how they are built. One person wants a,g,f,t,z and
another wants a,t,q,p,z. If they want them ASAP, then it's not
obvious concurrency is an issue at all; perhaps newsletters with
p go out whenever p changes and triggers them. Maybe changes in
some of the topics do not trigger new email, but wait on a clock,
etc In either case, they may still be different per subscriber
because of the other selections (else they are not personalized).
Even with half a million subscribers, depending on the number of
options and the subscriber base you might still be sending only
a very few messages to each site. And if they are truly personalized,
it's not at all clear to me how one would build them so they could
go in the queue more than one by one.
Roughly:
SELECT email_address, topics FROM subscribers WHERE email ='$1' ORDER
BY topics,email_address;
for (@email_address) {
build message(@topics);
qmail-inject message # we do these ourselves one-by-one; personalized
}
Qmail will not be the hard part :-)
If, OTOH, they are a personal selection of independant, non-personalized
messages, batching will make sense - but that's not "personalized".
As usual, a clear statement of the situation will help.
cfm
--
Christopher F. Miller, Publisher [EMAIL PROTECTED]
MaineStreet Communications, Inc 208 Portland Road, Gray, ME 04039
1.207.657.5078 http://www.maine.com/
Content management, electronic commerce, internet integration, Debian linux
Re: Qmail and Large Scale Dynamic Mailings
On Thu, Dec 07, 2000 at 05:36:13AM -0600, Ken Jones wrote: Wayne Chu wrote: I feel our discussion starts to become OT. I tried to be calm and I won't discuss it any more after this. Obviously this is a hot topic for you. So, without furthre evidence, you are obviously a person who will exploit the edge of spam related email. I have a question for you Sir. Are you going to be sending these "emails" to your local list of accounts. Or will this be going out to other lists? How have you received these lists of emails? I have a suggestion for you: stop busting this guy's balls unless you can prove he is sending unsolicited bulk mail. PGP signature
Re: Qmail and Large Scale Dynamic Mailings
On Wed, Dec 06, 2000 at 11:45:49AM -0500, Thomas Duterme wrote: 2) Qmail is set to open 20 SMTP connections at a time Ding ding ding ding! We have a winner! Try at least doubling that. Unfortunately, you can't say "20 per destination" or "20 per domain", but setting it to "20 total" is really going to kill performance. Going from 240 to 20 may have been a bit of an over-reaction. Can you try 120? That'll help a LOT. If you have to leave it at 20, you may want to turn your SMTP timeout down to like 5 seconds at the beginning of a mailing, so all the slow to respond mail servers are bypassed, then increase it to like 30 after an hour or two, then to 200 after another hour or two and do a "killall -ALRM qmail-send". Wacky, but it might get the job done... Sean -- Why are Bush supporters acting like they won, when Gore has 350,000 more popular votes? Sean Reifschneider, Inimitably Superfluous [EMAIL PROTECTED] tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
Re: Qmail and Large Scale Dynamic Mailings
From: Thomas Duterme [EMAIL PROTECTED] Well, I'm based out of China actually, and a large majority of our users are using email servers which are based in China... Yeah.. I suppose that makes it worst, right? I think I can assume that connectivity to China should be much lower than to the US... hmmm we had a problem once when we set concurrency to 240...one time all of our mail bounced back and we found that many of the Chinese ISPs blocked us since we had ended up flooding their servers...that's why we scaled back to the modest 20 That might have to do with ... huh... the great firewall of China . :) Seriously, tough, sew below. oohh, I think I understand. Call qmail-inject directly to bypass the local SMTP step we've been doing, right? Just on a sidenote, opening so many local ports like I've been doingwhat adverse effects would this have on tha mailout. There is no problem with that many ports, assuming the OS is correctly setup. Also, would you advise changing the qmail-inject source directly to sort which queue the mail is injected in? Yes, this qmail server only acts as a sender and doesn't recieve any mail or bounces. Not exactly. To operate multiple qmail queues, you in fact have multiple qmail installations. For example, /var/qmail1/... /var/qmail2/.. /var/qmail3/.. To direct a mail to a specific queue, you call the specific qmail-inject: /var/qmail2/bin/qmail-inject Now, there is a trick in all of this: your Java proggie must me changed so as to call the various qmail-injects, *and* to avoid sending to the same destination domain from the same queue. Note, I say -avoid-, not prevent. The advantage of this approach comparing it to something like the big concurrency patch is that slower domains do not clog the queue. In your case, you can even fine tune the system, for example directing the mail for specific servers to a queue with a limited concurrency: /var/qmail-20concurrency/... /var/qmail-100concurrency/.. Armando
Re: Qmail and Large Scale Dynamic Mailings
Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. My newsletter program calls qmail-qmqpc directly. Does qmail send mails to recpt in the order I write the address to qmail-qmqpc? For example, if I wrote addresses A, B, C to qmail-qmqpc. Will qmail first invokes qmail-remote to send mail to A And then (concurrently, before the first qmail-remote finishes) invokes another qmail-remote to B, and then to C? If so, maybe I can sort my subscriber list first, that subscribers in the same mail server will be distrubuted among the whole list evenly. So I can minimize the chance of overflooding a certain server?
Re: Qmail and Large Scale Dynamic Mailings
On Thu, Dec 07, 2000 at 11:15:45AM +0800, Wayne Chu wrote: Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. My newsletter program calls qmail-qmqpc directly. Does qmail send mails to recpt in the order I write the address to qmail-qmqpc? It surely does. But that's ultimately just a queuing order and it doesn't necessarily mean a delivery order. For example, if I wrote addresses A, B, C to qmail-qmqpc. Will qmail first invokes qmail-remote to send mail to A And then (concurrently, before the first qmail-remote finishes) invokes another qmail-remote to B, and then to C? As it happens, yes. I don't believe that it is gauranteed in any documentation, therefore relying on this current behaviour may be risky. If so, maybe I can sort my subscriber list first, that subscribers in the same mail server will be distrubuted among the whole list evenly. So I can minimize the chance of overflooding a certain server? Sounds like it might help, but consider the case of a particular domain that is not accepting mail. At the time of the first retry, perhaps the only recipients left are to that domain in which case it may well get hit with the full concurrency of your server. My point is that this *may* help in some circumstances, but it's by no means bullet-proof. Regards.
Re: Qmail and Large Scale Dynamic Mailings
Mark Delany wrote: On Thu, Dec 07, 2000 at 11:15:45AM +0800, Wayne Chu wrote: Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. My newsletter program calls qmail-qmqpc directly. Does qmail send mails to recpt in the order I write the address to qmail-qmqpc? It surely does. But that's ultimately just a queuing order and it doesn't necessarily mean a delivery order. Delivery order depends on DNS servicing smtp servicing and the exact code you are using to inject emails into qmail-pmpqc. For example, if I wrote addresses A, B, C to qmail-qmqpc. Will qmail first invokes qmail-remote to send mail to A And then (concurrently, before the first qmail-remote finishes) invokes another qmail-remote to B, and then to C? As it happens, yes. I don't believe that it is gauranteed in any documentation, therefore relying on this current behaviour may be risky. Your delivery order depends on your remote concurrency limitations, the availablity of dns results and your disk IO. Why are you so concerned about delivery order? What is your ultimate goal? Why is delivery order such a concern? If so, maybe I can sort my subscriber list first, that subscribers in the same mail server will be distrubuted among the whole list evenly. So I can minimize the chance of overflooding a certain server? Sounds like it might help, but consider the case of a particular domain that is not accepting mail. At the time of the first retry, perhaps the only recipients left are to that domain in which case it may well get hit with the full concurrency of your server. My point is that this *may* help in some circumstances, but it's by no means bullet-proof. Regards. You are wondering how you can spread your out going smtp deliveries across multiple recipient smtp servers. Why? What kind of email load are you imposing on the internet? Are you perhaps a person who has long lists of email accounts? Perhaps they are all sorted based on host name? Perhaps alot of the host names are yahoo.com? So.. You sit back.. and launch your spam list on the internet, and you wonder why.. when it gets to the yahoo.com list... it stalls with 255 remote deliveries, and they all take a long time to complete. And you are upset because you can't get your spam list delivered? Just what kind of email are you delivering? Your questions smack of spam problems. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
On Wed, Dec 06, 2000 at 10:41:28PM -0600, Ken Jones wrote: Mark Delany wrote: On Thu, Dec 07, 2000 at 11:15:45AM +0800, Wayne Chu wrote: Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. My newsletter program calls qmail-qmqpc directly. Does qmail send mails to recpt in the order I write the address to qmail-qmqpc? It surely does. But that's ultimately just a queuing order and it doesn't necessarily mean a delivery order. Delivery order depends on DNS servicing smtp servicing and the exact code you are using to inject emails into qmail-pmpqc. Right. But all things being equal, their will be a very strong correlation between the injection order and the delivery order. DNS lookups may certainly perturb this, but don't fundamentally change it in anyway. As it happens, yes. I don't believe that it is gauranteed in any documentation, therefore relying on this current behaviour may be risky. Your delivery order depends on your remote concurrency limitations, the availablity of dns results and your disk IO. No, yes, no. Concurrency does not change order. DNS results may perturb order, disk IO is unlikely to change order. In other words, there remains a strong correlation. Your point is? Why are you so concerned about delivery order? What is your ultimate goal? I think the original poster made that clear. He wants to minimize the concurrency to any one domain by sorting the recipients in such a way that the recipient domains are distributed across the whole list. Why is delivery order such a concern? How is this a different question from "Why are you so concerned" etc. So.. You sit back.. and launch your spam list on the internet, and you wonder why.. when it gets to the yahoo.com list... it stalls with 255 remote deliveries, and they all take a long time to complete. And you are upset because you can't get your spam list delivered? Just what kind of email are you delivering? Ken. You're assuming a spammer here - if you're wrong what do you think you're achieving apart from besmerching his name? In any event, if you've worked with large lists and large delivery capabilities, you'll know that many sites *do* block based on concurrency. Why only a month or so ago I was working with a company that does 20+million deliveries on a busy day and they were blocked by hotmail based on exceeding concurrent connection limits. It took quite an amount of work to have hotmail remove their automated block (phone calls, ceo to ceo, blah blah blah) - but they did so in the end, solely because they were ultimately convinced of the opt-in nature of the email. These problems do happen regularly in real life with legitimate lists. Perhaps the original poster has suffered the same problem? Your questions smack of spam problems. Your answers smack of speculation. Regards.
Re: Qmail and Large Scale Dynamic Mailings
No, I am NOT spamming. Our company runs serveral daily e-newsletter, with totally about a half million of subscribers. We are planning to make an "open" newsletter plateform, let our web site members create their own personal newsletter ( authenticated and supervised by our staff to prevent spam mail ). we estimated the total number of subscribers and the number of newsletter will grow even more. Surely our member would want their newsletters to be sent ASAP. So we have to increase concurrency. And our company is not in USA. In our country, there are only few large major ISPs dominate the market. I don't know what's the case in US, but nearly half of our current subscribers come from only 4 ISPs. We encountered only slight SMTP blocking problems now. But we expect the problem will grow with our subscribers and concurrency settings. That's why we are looking for ways to deliver large number of newsletters with maximum speed possible, without overloading remote mail servers. If my questions cause controversy, I apologize. But never did I intend to abuse Internet e-mail. You are wondering how you can spread your out going smtp deliveries across multiple recipient smtp servers. Why? What kind of email load are you imposing on the internet? Are you perhaps a person who has long lists of email accounts? Perhaps they are all sorted based on host name? Perhaps alot of the host names are yahoo.com? So.. You sit back.. and launch your spam list on the internet, and you wonder why.. when it gets to the yahoo.com list... it stalls with 255 remote deliveries, and they all take a long time to complete. And you are upset because you can't get your spam list delivered? Just what kind of email are you delivering? Your questions smack of spam problems. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
Mark Delany wrote: On Wed, Dec 06, 2000 at 10:41:28PM -0600, Ken Jones wrote: Mark Delany wrote: On Thu, Dec 07, 2000 at 11:15:45AM +0800, Wayne Chu wrote: Am Mittwoch, 6. Dezember 2000 22:04 schrieb Thomas Duterme: How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. My newsletter program calls qmail-qmqpc directly. Does qmail send mails to recpt in the order I write the address to qmail-qmqpc? It surely does. But that's ultimately just a queuing order and it doesn't necessarily mean a delivery order. Delivery order depends on DNS servicing smtp servicing and the exact code you are using to inject emails into qmail-pmpqc. Right. But all things being equal, their will be a very strong correlation between the injection order and the delivery order. DNS lookups may certainly perturb this, but don't fundamentally change it in anyway. Okay. All things being equal. I can understand that. Would you please explain what you mean "all things being equal". How do we relate an unprecise english phrase to email delivery? Unless you post exact information about your exact case with exact log information.. Perhaps we should all bow down and worhship the mystical order of the "blah blah". My meaning is.. Unless you post your exact information, you are just posing a hypothetical situation. If you want answers to your "hypothetical" situation, that's fine. But I expect you have a defined goal in mind. And I expect any hypothetical answer will not solve your particular problem. If you wish for me to attempt to solve your problem. I'm willing to devote my limited resources. However, you have not posted your dns information, nor your email delivery information. Shall we attempt to devine your intensions from you unclear descriptions? As it happens, yes. I don't believe that it is gauranteed in any documentation, therefore relying on this current behaviour may be risky. Your delivery order depends on your remote concurrency limitations, the availablity of dns results and your disk IO. No, yes, no. Concurrency does not change order. DNS results may perturb order, disk IO is unlikely to change order. In other words, there remains a strong correlation. Your point is? DNS order of service information completely depends on the DNS service you are relying on. Do you attempt to decieve us and say that all DNS services act the same? Please provide exact information about the dns results you are receiveing so that others may attempt to duplicate your results. Why are you so concerned about delivery order? What is your ultimate goal? I think the original poster made that clear. He wants to minimize the concurrency to any one domain by sorting the recipients in such a way that the recipient domains are distributed across the whole list. So, you wish to minimize the concurrency on your machine, or do you wish to minimize the delivery speed. I suspect the original poster is attempting to minimize the delivery to the recipient. Can you please post delivery speed information for multiple reciepients to a singular domain versus alternative mail delivery agents? Or are you just another poser? Why is delivery order such a concern? How is this a different question from "Why are you so concerned" etc. Because, unless you have been living in a cave, all of us email admins have to deal with people who are concerned about "email throughput". And most of those people are illegal spammers who are hard to take to court. Why did you evade my question? If you are a real person with a non-illegal concern, please post your company information, email address dns host name, company phone number, corporate officers names to this list. I'm sure you won't. So.. You sit back.. and launch your spam list on the internet, and you wonder why.. when it gets to the yahoo.com list... it stalls with 255 remote deliveries, and they all take a long time to complete. And you are upset because you can't get your spam list delivered? Just what kind of email are you delivering? Ken. You're assuming a spammer here - if you're wrong what do you think you're achieving apart from besmerching his name? Prove me wrong sir. In any event, if you've worked with large lists and large delivery capabilities, you'll know that many sites *do* block based on concurrency. Why only a month or so ago I was working with a company that does 20+million deliveries on a busy day and they were blocked by hotmail based on exceeding concurrent connection limits. It took quite an amount of work to have hotmail remove their automated block (phone calls, ceo to ceo, blah blah blah)
Re: Qmail and Large Scale Dynamic Mailings
Wayne Chu wrote: No, I am NOT spamming. Our company runs serveral daily e-newsletter, with totally about a half million of subscribers. We are planning to make an "open" newsletter plateform, let our web site members create their own personal newsletter ( authenticated and supervised by our staff to prevent spam mail ). we estimated the total number of subscribers and the number of newsletter will grow even more. Surely our member would want their newsletters to be sent ASAP. So we have to increase concurrency. Ah.. So you are not a spammer, except you assume all your "customers" want your email. Besides that moral issue, do you have measured information about the delivery statistics of qmail version other options? And our company is not in USA. In our country, there are only few large major ISPs dominate the market. I don't know what's the case in US, but nearly half of our current subscribers come from only 4 ISPs. We encountered only slight SMTP blocking problems now. But we expect the problem will grow with our subscribers and concurrency settings. Are you saying that your country is a pirate economy? Shall I embarrase you and trace your email? or do you wish to reveal which country you are from? Does your country harbor pirates? That's why we are looking for ways to deliver large number of newsletters with maximum speed possible, without overloading remote mail servers. If my questions cause controversy, I apologize. But never did I intend to abuse Internet e-mail. Sir. If your users are on your machines, you can just copy your emails to thier directories. But.. it seems like your users are spread over other peoples machines. Hence.. by definition, you are a spammer, sir. Ken Jones
Re: Qmail and Large Scale Dynamic Mailings
On Thu, Dec 07, 2000 at 12:29:35AM -0600, Ken Jones wrote: Okay. All things being equal. I can understand that. Would you please explain what you mean "all things being equal". Beautiful. I couldn't have said it better. I'm way out of my league here - that's for sure. Regards.
Re: Qmail and Large Scale Dynamic Mailings
Ken Jones [EMAIL PROTECTED] writes: Sir. If your users are on your machines, you can just copy your emails to thier directories. But.. it seems like your users are spread over other peoples machines. Hence.. by definition, you are a spammer, sir. Please. You have no evidence to show he's a spammer. If the newsletters are run properly (i.e. with confirmed opt-in), he is doing the Right Thing. Furthermore, he's trying to be a responsible net-neighbour and not cause problem to other people's mailservers. That is something a lot of maillistserver admins would do well to emulate. -- "I live in the heart of the machine. We are one."
Re: Qmail and Large Scale Dynamic Mailings
Hi Armando, Thanks very much for your helpful reply. *But* you're using SMTP, and many sites (hotmail.com for one) take enormous time to handle SMTP. This in fact reduces the usable bandwidth (again for hotmail.com, around 600kb/s on a good day from Europe, in my experience). You can increase parallelism (concurrencyremote) to try to send more mails at the same time and optimize the bandwidth, but this only goes so far. Well, I'm based out of China actually, and a large majority of our users are using email servers which are based in China... 1) Setup additional qmail-queues in the same machine (using different disks for each queue, so as to minimize seek time). All of those set to 240 concurrency local (requires trivial change on compiling qmail) hmmm we had a problem once when we set concurrency to 240...one time all of our mail bounced back and we found that many of the Chinese ISPs blocked us since we had ended up flooding their servers...that's why we scaled back to the modest 20 concurrent connections. Have you had any of the same thing happen? I've done this at 120 concurrent connections, but I don't want to push it too much. 2) Change the dynamic Java thing to send to each queue in turn, and use qmail-inject instead of SMTP (this avoids having to create as many SMTP ports) oohh, I think I understand. Call qmail-inject directly to bypass the local SMTP step we've been doing, right? Just on a sidenote, opening so many local ports like I've been doingwhat adverse effects would this have on tha mailout. Also, would you advise changing the qmail-inject source directly to sort which queue the mail is injected in? Yes, this qmail server only acts as a sender and doesn't recieve any mail or bounces. 3) All bounces would be handled by a single qmail, of course yup. I'm with you on this one. Note that your gain would still be limited by the bandwidth: 1 M messages of that size a day requires more bandwidth or smaller messages.
Re: Qmail and Large Scale Dynamic Mailings
Hi Marlon, Thanks very much for your reply. How about increasing your concurrencyremote to something like 100? you most likely are hitting your limits. Good point. Will try that tonight. I've gotten some problems before from ISP's blocking us when I went up to 240...I'm not quite sure what the highest polite limit on this should be. If you are running plain vanilla qmail, why not running 4 instances at a time? each running under a different IP address and under a different disk? then you would have a concurrencyremote of a total of 400 and I/O is balanced accross 4 disks. This sounds really good. I think we'll try it out. you just have to modify your smtp injection script to load balance accross the 4 instances. Thanks, Marlon
