Re: Multiple recipients to remote domain
Joshua Nichols [EMAIL PROTECTED] wrote: 2. If so, how does one message with many recipients save memory or run faster? I wasn't 100% sure about which phase of delivery you're talking about; if you're talking about queue injection, one message to X recipients only has to be injected into the queue once (with the appropriate number of seeks syncs that that requires), where X messages to 1 recipient each requires X queue injections -- this can get to be an expensive operation, and the sudden surge in todo/ can temporarily cause qmail-send to not start any new deliveries, slowing overall delivery down. If you're talking about SMTP deliveries to remote systems, some MTAs do SMTP connection caching for multiple messages to the same host or MX -- however, as SMTP on the 'net at large is mostly latency bound, it's usually faster to just fire up multiple parallel qmail-remotes each delivering a single message, and this is in fact what qmail does. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 09:51:46AM -0600, Roger Walker wrote: I just ran a test where I created a single message file, with all headers, and the BCC list was 10,000+ copies of my own address on a remote domain. I used qmail-inject to send the message. What I expected to happen was for the local QMAIL to make a single connection to the remote domain, deliver the single message, and have the remote system make 10,000+ deliveries to the remote account. Nope. The local QMAIL logs seem to indicate that the local system actually sent 10,000+ separate messages to the remote domain, possibly in small batches. The log appeared to grow in spurts, but I am not sure if It did 10.000 SMTP connections to send the message 10.000 times total. If I can, I'll try to decifer the logs on the remote system to see what happened, but I am wondering (asking) what QMAIL's behavior is supposed to be when there are multiple recipients on a single message being sent to a remote domain. Is there only a single copy of the message sent to the remote host in a single transaction? Does the local QMAIL break down the local message into discrete messages, one per recipient, and then send them to the (same) remote host? Yes. qmail always handles only one recipient per SMTP connection. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Multiple recipients to remote domain
Roger Walker [EMAIL PROTECTED] wrote: I just ran a test where I created a single message file, with all headers, and the BCC list was 10,000+ copies of my own address on a remote domain. I used qmail-inject to send the message. What I expected to happen was for the local QMAIL to make a single connection to the remote domain, deliver the single message, and have the remote system make 10,000+ deliveries to the remote account. The local QMAIL logs seem to indicate that the local system actually sent 10,000+ separate messages to the remote domain, possibly in small batches. qmail never batches recipients to remote domains. One of the (several) reasons for this is it would make VERP impossible. If I can, I'll try to decifer the logs on the remote system to see what happened, but I am wondering (asking) what QMAIL's behavior is supposed to be when there are multiple recipients on a single message being sent to a remote domain. qmail delivers each separately, as you have observed. Is there only a single copy of the message sent to the remote host in a single transaction? No. Does the local QMAIL break down the local message into discrete messages, one per recipient, and then send them to the (same) remote host? Yes. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 09:51:46AM -0600, Roger Walker wrote: If I can, I'll try to decifer the logs on the remote system to see what happened, but I am wondering (asking) what QMAIL's behavior is supposed to be when there are multiple recipients on a single message being sent to a remote domain. Is there only a single copy of the message sent to the remote host in a single transaction? No. Does the local QMAIL break down the local message into discrete messages, one per recipient, and then send them to the (same) remote host? Yes. As documented somewhere on cr.yp.to/qmail.html. -- * Henning Brauer, [EMAIL PROTECTED], http://www.bsws.de * * Roedingsmarkt 14, 20459 Hamburg, Germany * Unix is very simple, but it takes a genius to understand the simplicity. (Dennis Ritchie)
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 10:20:57AM -0600, Charles Cazabon wrote: [snip] Is there only a single copy of the message sent to the remote host in a single transaction? No. Hmm, this confuses me. Every transaction contains one message. There are 10.000 transactions. This may or may not be a 'yes' or 'no' to the question. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Multiple recipients to remote domain
Peter van Dijk [EMAIL PROTECTED] wrote: On Mon, Jun 11, 2001 at 10:20:57AM -0600, Charles Cazabon wrote: [snip] Is there only a single copy of the message sent to the remote host in a single transaction? No. Hmm, this confuses me. Every transaction contains one message. There are 10.000 transactions. This may or may not be a 'yes' or 'no' to the question. To clarify: no, qmail will not send one copy of the message to 1 recipients at a single MX in a single transaction. Instead, qmail will establish 1 SMTP sessions (transactions) to that MX, and deliver a single copy of the message to one recipient in each of those sessions. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Multiple recipients to remote domain
Thanks, Peter and Charles. Looks like I'll have to script a solution that telnets to port 25 on the remote host and issues 10,000+ (650,000+ actually) rcpt to: lines. A manual test seems to have worked, so now I'll have to try automating it. On Mon, Jun 11, 2001 at 09:51:46AM -0600, Roger Walker wrote: I just ran a test where I created a single message file, with all headers, and the BCC list was 10,000+ copies of my own address on a remote domain. I used qmail-inject to send the message. What I expected to happen was for the local QMAIL to make a single connection to the remote domain, deliver the single message, and have the remote system make 10,000+ deliveries to the remote account. Nope. It did 10.000 SMTP connections to send the message 10.000 times total. Yes. qmail always handles only one recipient per SMTP connection. -- Roger Walker Tier III Messaging/News Team Internet Applications, National Consumer IP TELUS Corporation 780-493-2471
Re: Multiple recipients to remote domain
Charles Cazabon writes: qmail never batches recipients to remote domains. One of the (several) reasons for this is it would make VERP impossible. Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. Actually there's no protocol reason why qmail-remote couldn't look for VERP in the EHLO tags, and send over all recipients for the current piece of email along with a VERP envelope sender. Of course, the current structure of qmail-send won't allow for that (even though qmail-remote will). -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | 521 Pleasant Valley Rd. | +1 315 268 1925 voice | John Hartford, RIP Potsdam, NY 13676-3213 | +1 315 268 9201 FAX |
Re: Multiple recipients to remote domain
Roger Walker [EMAIL PROTECTED] wrote: Thanks, Peter and Charles. Looks like I'll have to script a solution that telnets to port 25 on the remote host and issues 10,000+ (650,000+ actually) rcpt to: lines. Unnecessary. There's lots of tools to do this; other MTAs do it, and djb wrote serialmail for these sorts of situations. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Multiple recipients to remote domain
Russell Nelson [EMAIL PROTECTED] wrote: Charles Cazabon writes: qmail never batches recipients to remote domains. One of the (several) reasons for this is it would make VERP impossible. Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. Actually there's no protocol reason why qmail-remote couldn't look for VERP in the EHLO tags, and send over all recipients for the current piece of email along with a VERP envelope sender. Of course, the current structure of qmail-send won't allow for that (even though qmail-remote will). Good clarification. Would you settle for Making VERP work with per-MX batched recipients of a message would require extensive core changes to qmail? I think that basically it boils down to this: if per-MX batching is critical for a particular installation (bandwidth limitations or other issues), and serialmail isn't sufficient, perhaps qmail isn't the right MTA for the job. Of course, the number of installations where this would be true should be very small. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 02:33:53PM -0400, Russell Nelson wrote: Charles Cazabon writes: qmail never batches recipients to remote domains. One of the (several) reasons for this is it would make VERP impossible. Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. Actually there's no protocol reason why qmail-remote couldn't look for VERP in the EHLO tags, and send over all recipients for the current piece of email along with a VERP envelope sender. Of course, the current structure of qmail-send won't allow for that (even though qmail-remote will). qmail-remote would have to start saying EHLO ofcourse, in that case. Is VERP a registered EHLO tag? Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 01:02:40PM -0600, Charles Cazabon wrote: Roger Walker [EMAIL PROTECTED] wrote: Thanks, Peter and Charles. Looks like I'll have to script a solution that telnets to port 25 on the remote host and issues 10,000+ (650,000+ actually) rcpt to: lines. Unnecessary. There's lots of tools to do this; other MTAs do it, and djb wrote serialmail for these sorts of situations. serialmail still does one recipient per mail, but does feed all mails through one SMTP link. What Roger wants is one mail with 650.000 recipients. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
RE: Multiple recipients to remote domain
Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. All this talk of delivery optimization and VERP actually raises a few question for me: 1. Is there a seperate instance of qmail-remote for each bcc: header? 2. If so, how does one message with many recipients save memory or run faster? Wouldn't there be an identical number of messages in memory as sending many messages with one recipient? I'm assuming the answer is no, otherwise it wouldn't be recommended, right? 4. Does the existing qmail-verh patch work on the body of the message? The archives suggest that this would be VERB, not VERP or VERH. 5. If qmail-verh won't do replacements on the body, did anyone ever write a qmail-verb patch? 6. Does implementing VERB or VERH negate the benefits of 1 message, many recipients? Lyris and L-soft both claim that their mtas are better (faster) because they will do domain batching. If they are not misleading the masses, has anyone thought of ways (or developed patches) to implement this behavior in qmail? Russ? Perhaps this is all misguided conversation, but it seems to me that most of the threads on the list fall into 1 of 2 categories: 1. Qmail doesn't work (read as I broke it * ). 2. How can I get ___ to work better? (Expect What problem are you trying to solve?) What are people's thoughts? Feel free to respond off-list if you feel this is off-topic. I am thinking of assembling a document containing (founded upon) the best advice from the gurus, because these sorts of issues so often make it to the list (and past the archives). --joshua. *often: I broke it, but am so used to paying too much for crappy software, that I naturally assume that something is wrong with the program.
Re: Multiple recipients to remote domain
Charles Cazabon writes: Good clarification. Would you settle for Making VERP work with per-MX batched recipients of a message would require extensive core changes to qmail? No. VERP support has little to do with it. The problem is that qmail-send / qmail-rspawn are strongly oriented around one recipient per invocation of qmail-remote. VERP just makes it slightly more complicated. qmail-send and qmail-rspawn could offer qmail-remote a list of recipients for the same message at the same host[1], and qmuail-remote could decide if VERP support from the SMTP client was needed. If it wasn't present then qmail-remote could *still* re-use the TCP connection. Given the state of the art in TCP/IP stacks, this would be a good thing because TCP retry timers are not shared between TCP sessions. [1] I'm not talking about per-MX, but instead per-host. Collating by MX entry means a gratuitious MX lookup (admittedly, probably cached). Instead, I'm just talking about a textual comparison between hostnames. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | 521 Pleasant Valley Rd. | +1 315 268 1925 voice | John Hartford, RIP Potsdam, NY 13676-3213 | +1 315 268 9201 FAX |
Re: Multiple recipients to remote domain
Peter van Dijk writes: qmail-remote would have to start saying EHLO ofcourse, in that case. Is VERP a registered EHLO tag? I'm sure it isn't. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | 521 Pleasant Valley Rd. | +1 315 268 1925 voice | John Hartford, RIP Potsdam, NY 13676-3213 | +1 315 268 9201 FAX |
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 05:00:42PM -0400, Joshua Nichols wrote: Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. All this talk of delivery optimization and VERP actually raises a few question for me: 1. Is there a seperate instance of qmail-remote for each bcc: header? There is a separate instance of qmail-remote for every recipient. If a recipient is named twice, that counts as two recipients. 2. If so, how does one message with many recipients save memory or run faster? Wouldn't there be an identical number of messages in memory as sending many messages with one recipient? I'm assuming the answer is no, otherwise it wouldn't be recommended, right? The message only enters the queue once, from which point on qmail only has to flag a recipient when a message is sent. If you send lots of separate identical messages each with one recipient, these all have to go into the queue, chewing up diskspace and disk bandwidth. qmail is very good at spewing out one message lots of times. That's why it rocks for mailinglists :) 6. Does implementing VERB or VERH negate the benefits of 1 message, many recipients? No. The patches exist explicitly to allow some extra flexibility while keeping that benefit. Lyris and L-soft both claim that their mtas are better (faster) because they will do domain batching. If they are not misleading the masses, has anyone thought of ways (or developed patches) to implement this behavior in qmail? Russ? They are misleading the masses. Perhaps this is all misguided conversation, but it seems to me that most of the threads on the list fall into 1 of 2 categories: 1. Qmail doesn't work (read as I broke it * ). 2. How can I get ___ to work better? (Expect What problem are you trying to solve?) Might be quite a correct observation :) What are people's thoughts? Feel free to respond off-list if you feel this is off-topic. I am thinking of assembling a document containing (founded upon) the best advice from the gurus, because these sorts of issues so often make it to the list (and past the archives). This is ofcourse good, but check the archives. Several FAQ-like pages already exists. www.qmail.org links to most of m. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
RE: Multiple recipients to remote domain
Joshua Nichols writes: Not true. It simply means that the remote system would have to implement VERP when qmail-remote tells the smtpd that the envelope sender is list-@[]@host.example.com. Unfortunately, qmail-remote and VERP-compatible smtp servers do not cooperate in that manner. All this talk of delivery optimization and VERP actually raises a few question for me: 1. Is there a seperate instance of qmail-remote for each bcc: header? There is a separate instance of qmail-remote for every recipient regardless of how the recipient got to qmail-queue (and yes, bcc: is one of the ways). 2. If so, how does one message with many recipients save memory or run faster? It saves a HUGE amount of disk space and disk bandwidth. Otherwise, each recipient gets its own copy of the message body. 4. Does the existing qmail-verh patch work on the body of the message? The archives suggest that this would be VERB, not VERP or VERH. No, it doesn't modify the body. What if a message contained '#'? 5. If qmail-verh won't do replacements on the body, did anyone ever write a qmail-verb patch? I did, but it's not freely copyable, and I charge mucho dinero to install it. 6. Does implementing VERB or VERH negate the benefits of 1 message, many recipients? No, because the per-recipient substitution is done inside qmail-remote. Lyris and L-soft both claim that their mtas are better (faster) because they will do domain batching. If they are not misleading the masses, has anyone thought of ways (or developed patches) to implement this behavior in qmail? Russ? Dan has said that qmail has been measured to be faster and use less bandwidth than sendmail's implementation of the same idea. I don't know if Lyris and L-soft have better implementations than sendmail. Changing qmail to do this kind of batching would be a significant change. I hope that qmail 2 will address this problem. Whether it's a real problem or not, it's certainly a marketing problem. And it doesn't matter if you've got the world's most secure software if people have reasons to not use it. Perhaps this is all misguided conversation, but it seems to me that most of the threads on the list fall into 1 of 2 categories: 1. Qmail doesn't work (read as I broke it * ). 2. How can I get ___ to work better? (Expect What problem are you trying to solve?) :-) -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | 521 Pleasant Valley Rd. | +1 315 268 1925 voice | John Hartford, RIP Potsdam, NY 13676-3213 | +1 315 268 9201 FAX |
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 12:09:40PM -0600, Roger Walker wrote: Thanks, Peter and Charles. Looks like I'll have to script a solution that telnets to port 25 on the remote host and issues 10,000+ (650,000+ actually) rcpt to: lines. You can also use qmail-remote manually to do this. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8 PGP signature
Re: Multiple recipients to remote domain
On Mon, Jun 11, 2001 at 05:12:44PM -0600, Bruce Guenter allegedly wrote: On Mon, Jun 11, 2001 at 12:09:40PM -0600, Roger Walker wrote: Thanks, Peter and Charles. Looks like I'll have to script a solution that telnets to port 25 on the remote host and issues 10,000+ (650,000+ actually) rcpt to: lines. You can also use qmail-remote manually to do this. And Net::SMTP from your local CPAN makes life pretty easy if you want to have a more programmatic interface. Regards.
