Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 05:34:59AM -0400, Philip Mak wrote: [snip] My question about fastforward is: Will my existing .qmail-* files stop working? If so, how can I make the ezmlm aliases still work? e.g. one of Yes, they will work. fastforward will just sit in .qmail-default and handle anything that's not being handled by their own .qmail file. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 05:34:59AM -0400, Philip Mak wrote: Hmm, looks like it could work. The Speed tests section of http://cr.yp.to/fastforward.html says that it takes only 6 seconds to regenerate an alias db with 5 entries. I could run a cron job every two hours to regenerate the cdb. I'd be more worried about speed of delivery than speed of DB regeneration. Note that it's still a program delivery, albeit done through a more efficient program than your existing perlDB script. As an aside, has anyone done any performance comparisons between fastforward and .qmail-forwarding for large numbers of aliases (10,000)? My question about fastforward is: Will my existing .qmail-* files stop working? fastforward is traditionally invoked in ~alias/.qmail-default, so unless you have some other delivery instructions already in that file, everything else should still work. -- Adrian HoTinker, Drifter, Fixer, Bum [EMAIL PROTECTED] ListArchive: http://marc.theaimsgroup.com/?l=qmail Useful URLs: http://cr.yp.to/qmail.html http://www.qmail.org http://www.lifewithqmail.org/ http://qmail.faqts.com/
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, 28 Jul 2001, Adrian Ho wrote: Hmm, looks like it could work. The Speed tests section of http://cr.yp.to/fastforward.html says that it takes only 6 seconds to regenerate an alias db with 5 entries. I could run a cron job every two hours to regenerate the cdb. I'd be more worried about speed of delivery than speed of DB regeneration. Note that it's still a program delivery, albeit done through a more efficient program than your existing perlDB script. Oh, I see now; fastforward is a program that I specify to be called in .qmail-default. I thought it was a patch to be applied to qmail. So it would still have the overhead of having to read a message from qmail, and then write that message back to qmail. That overhead would be unavoidable if I'm doing program delivery, I guess. I wonder if in the future, they'll make an alias delivery option in qmail; that is, it calls an external program, but instead of sending the entire message to the program, it just sends the RCPT TO: address to the program and the program returns to it which mailbox(es) should be delivered to. Then again, this could turn out to be an ugly piece of 'feature creep'.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote: I wonder if in the future, they'll make an alias delivery option in qmail; that is, it calls an external program, but instead of sending the entire message to the program, it just sends the RCPT TO: address to the program and the program returns to it which mailbox(es) should be delivered to. That's trivially done today -- no extra options required: |forward `my-redirector $RECIPIENT` Sounds like you haven't read The Big Qmail Picture by Andre Oppermann http://www.nrg4u.com/. Three years old and only 4 pages long, but still very useful for illuminating the little-known corners of qmail. -- Adrian HoTinker, Drifter, Fixer, Bum [EMAIL PROTECTED] ListArchive: http://marc.theaimsgroup.com/?l=qmail Useful URLs: http://cr.yp.to/qmail.html http://www.qmail.org http://www.lifewithqmail.org/ http://qmail.faqts.com/
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho allegedly wrote: On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote: I wonder if in the future, they'll make an alias delivery option in qmail; that is, it calls an external program, but instead of sending the entire message to the program, it just sends the RCPT TO: address to the program and the program returns to it which mailbox(es) should be delivered to. That's trivially done today -- no extra options required: |forward `my-redirector $RECIPIENT` Nope. You cut too much out of the original posting. He said: So it would still have the overhead of having to read a message from qmail, and then write that message back to qmail. That overhead would be unavoidable if I'm doing program delivery, I guess. In other words he doesn't want each mail to go thru the queue twice as your solution implies. To answer Philip's question: Yes, that overhead is unavailable as there is no standard qmail solution for redirecting mail without it going thru the queue at least once. Having said that your concern about overhead may be misplaced. What sort of volume are you expecting on what sort of system? Regards.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho wrote: On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote: I wonder if in the future, they'll make an alias delivery option in qmail; that is, it calls an external program, but instead of sending the entire message to the program, it just sends the RCPT TO: address to the program and the program returns to it which mailbox(es) should be delivered to. That's trivially done today -- no extra options required: |forward `my-redirector $RECIPIENT` That's not what he means. This still reads the message and reinjects it. His proposal (which I have been pondering about for months already :) means that a program can tell qmail 'send this mail you are trying to give to me, to this address' without reinjection. This could save a lot of disk bandwidth, IMHO. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On 28 Jul 2001, MarkD wrote: To answer Philip's question: Yes, that overhead is un[avoidable] as there is no standard qmail solution for redirecting mail without it going thru the queue at least once. Having said that your concern about overhead may be misplaced. What sort of volume are you expecting on what sort of system? 24000 total users on a Pentium III 850MHz with 768 MB of RAM (not sure how many are active though...at least a couple thousand). I currently use .qmail-default to run a perl script which connects to a MySQL database, performs a lookup on the alias, then opens a pipe to sendmail to deliver the message. Upon activating this system, the load average of the machine has increased from 1-2 to 10! I suspect most of the time is being spent compiling the perl script and connecting to the MySQL database, though. If I switch to fastforward (or if I rewrite the script in C, and use a persistent database connection handle somehow, maybe by storing it in an flock'd file) maybe the load average will drop back down to normal.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 10:29:31AM -0400, Philip Mak wrote: [snip] Upon activating this system, the load average of the machine has increased from 1-2 to 10! I suspect most of the time is being spent compiling the perl script and connecting to the MySQL database, though. If I switch to fastforward (or if I rewrite the script in C, and use a persistent database connection handle somehow, maybe by storing it in an flock'd file) maybe the load average will drop back down to normal. You can't store persistent database connection handles in flock'd files. There are two ways to persistent database connections: - do your stuff from within qmail (probably not the right spot) - have your perl/C program connect to a daemon that has a couple of persistent connections But I don't think this will help anything - most of your time is probably spent compiling perl. My recommendation is to use fastforward. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
That's not what he means. This still reads the message and reinjects it. His proposal (which I have been pondering about for months already :) means that a program can tell qmail 'send this mail you are trying to give to me, to this address' without reinjection. This could save a lot of disk bandwidth, IMHO. The qmail architecture does not lend itself well to this though does it? qmail-remote is the only code that knows how to remotely deliver a message and qmail-smtpd would have to be (extensively) modified to call that instead of qmail-queue. It would have been a cute touch if DjB had made the interface to qmail-remote the same as qmail-queue. In fact, one wonders whether all the inter-program delivery of mail in qmail should use some sort of common protocol such as that used by qmail-remote. Better yet would be to universally use QMTP/QMQP between programs. Anyway, even overcoming the interface obstacles, you have the nasty problem of inbound multiple recipients to deal with. qmail-remote only handles multiple recipients if they all happen to be going to the same domain. You could simply punt to qmail-queue of course if there is more than one recipient, but now it's starting to get messy as your delivery paths will be substantially different for the same recipient simply depending on whether they are part of an inbound multiple recipient mail or not. Regards.
Re: Fastforward question (was Re: Mail Forwarding Service)
24000 total users on a Pentium III 850MHz with 768 MB of RAM (not sure how many are active though...at least a couple thousand). By volume I meant how many emails per hour. Number of users is largely irrelevant. Upon activating this system, the load average of the machine has increased from 1-2 to 10! I suspect most of the time is being spent compiling the perl script and connecting to the MySQL database, though. If I switch to If you're doing this per delivery, I'm not surprised. But it should be easy to measure for sure with vmstat/top/acct, etc. fastforward (or if I rewrite the script in C, and use a persistent database connection handle somehow, maybe by storing it in an flock'd file) maybe the load average will drop back down to normal. Maintaining a persistent connection across multiple local deliveries is possible with some skull-hackery and a cooperating peer process, but it's not easy, it's not possible using flock and it does raise the issue of multiple deliveries using the same connection at the same instant. Tell us more about the deliveries? How many per hour, what is your concurrencylocal? Are the deliveries keeping up? An unadulterated snapshot of your qmail log would tell us a lot. At this stage, periodic rebuilding of a fastforward file sure sounds easiest - perhaps triggered by database changes. Regards.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 02:44:08PM +, MarkD wrote: That's not what he means. This still reads the message and reinjects it. His proposal (which I have been pondering about for months already :) means that a program can tell qmail 'send this mail you are trying to give to me, to this address' without reinjection. This could save a lot of disk bandwidth, IMHO. The qmail architecture does not lend itself well to this though does it? qmail-remote is the only code that knows how to remotely deliver a message and qmail-smtpd would have to be (extensively) modified to call that instead of qmail-queue. You are missing the point. We are just saying that a program invoked by qmail-local should have a way to communicate back to qmail 'change the address to blah', instead of having to reinject it. This would then still happen for every recipient like it does now. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 02:50:15PM +, MarkD wrote: [snip] At this stage, periodic rebuilding of a fastforward file sure sounds easiest - perhaps triggered by database changes. A 'select * from ...' followed by a fastforward cdb rebuild should pose no interesting load when executed, say, every 5 minutes, depending on volume. For 24.000 aliases (just assuming for now that you have about as much aliases as users), every minute can be feasible. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
The qmail architecture does not lend itself well to this though does it? qmail-remote is the only code that knows how to remotely deliver a message and qmail-smtpd would have to be (extensively) modified to call that instead of qmail-queue. You are missing the point. We are just saying that a program invoked by qmail-local should have a way to communicate back to qmail 'change the address to blah', instead of having to reinject it. This would then still happen for every recipient like it does now. Ug. That's even harder and it saves less than half of your queuing costs! That approach means that the message changes from a local to a remote delivery - the queue structure does not lend itself to making this change easily without incurring most of the cost of a queue injection. It also likely that the length of the recipient address will change - again the queue structure is poorly suited to this for multiple recipient emails as recipients are stored as a series of \0 terminated strings. It'll be interesting to see how you propose to atomically make such queue changes while incurring a worthwhile queueing cost saving. Regards.
Re: Fastforward question (was Re: Mail Forwarding Service)
My rough guess (see grep | wc above) is 7000 deliveries per hour. About 2 a second, that's not huge, but it's starting to get busy when you have to invoke multiple programs and establish a socket each time to your database. I see status: local 10/10 a lot. It goes back down in a few seconds, then can come back up in a few seconds too. (Should I increase concurrencylocal?) No. quite the opposite if anything. Think of concurrencylocal as a peak load that you want qmail to impose on your database - or your local file system for that matter. Do you really want to have more than 10 concurrent connections to your database? Better to keep that number safely within the capabilities of your system. I suspect that 10 is a good starting point for your delivery rates and it's much better to have an occassional local delivery delayed than to grind your database into dust. |forward `database-lookup $RECIPIENT` where database-lookup is a simple C program that connects to MySQL, looks up the recipient in the database, and prints it to standard output. This would be easy to write. It might not be as efficient as fastforward due to having to open a new connection to MySQL every time, but if it's efficient enough I think it's better because then (1) the database is always in perfect synch and (2) I don't have to worry about cron jobs to synchronize the fastforward db with the MySQL db. I'll have to try it and see what happens. That sounds like a good plan. Regards.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 11:20:37AM -0400, Philip Mak wrote: [snip] I'm also considering putting in my .qmail-default file: |forward `database-lookup $RECIPIENT` where database-lookup is a simple C program that connects to MySQL, looks up the recipient in the database, and prints it to standard output. shameless plug You may want to look at dteq (www.dataloss.nl/software/dteq/) in that case, which is a commandline mysql query tool /shameless plug However, using `` allows you no way to detect errors in the query. If the mysqld is down, where does the mail go? Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 04:21:29PM +0200, Peter van Dijk wrote: On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho wrote: |forward `my-redirector $RECIPIENT` That's not what he means. This still reads the message and reinjects it. Oops! That means it's time to hit the sack. 8-) But since I'm still (barely) conscious... His proposal (which I have been pondering about for months already :) means that a program can tell qmail 'send this mail you are trying to give to me, to this address' without reinjection. This could save a lot of disk bandwidth, IMHO. Unless the destination address happens to be in a virtual domain on the same machine, in which case the standard reinjection actually trumps the above by one unneeded SMTP transaction from the machine to itself. In any case, it sounds to me like we're entering the realm of pinhole optimization (or some equivalent concept). Is the performance boost worth the kinks it'll likely introduce in the existing qmail architecture? I'm not sure... -- Adrian HoTinker, Drifter, Fixer, Bum [EMAIL PROTECTED] ListArchive: http://marc.theaimsgroup.com/?l=qmail Useful URLs: http://cr.yp.to/qmail.html http://www.qmail.org http://www.lifewithqmail.org/ http://qmail.faqts.com/
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, 28 Jul 2001, Adrian Ho wrote: Unless the destination address happens to be in a virtual domain on the same machine, in which case the standard reinjection actually trumps the above by one unneeded SMTP transaction from the machine to itself. In any case, it sounds to me like we're entering the realm of pinhole optimization (or some equivalent concept). Is the performance boost worth the kinks it'll likely introduce in the existing qmail architecture? I'm not sure... Is it really that complicated to get the forwarding alias from a program? I'm thinking---at the moment when qmail is reading the .qmail-default file, it can encounter: [EMAIL PROTECTED] At this point, it has the capability of specifying a local or remote e-mail address to forward the message to. Doesn't it? So it should also be able to easily run an external program at this point to determine the e-mail address to forward to. I am not familiar with the internals of qmail, but from what I have seen, this would make sense.
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 03:36:02PM +, MarkD wrote: [snip] It'll be interesting to see how you propose to atomically make such queue changes while incurring a worthwhile queueing cost saving. I have no such proposal. I just feel that with some changes to the queueing structure, this might be feasible. On a 100mbyte mail, this saves reading+writing 100mbyte when a mail is forwarded. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 12:42:14PM -0400, Philip Mak wrote: [snip] I am not familiar with the internals of qmail, but from what I have seen, this would make sense. Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself, or talk to /var/qmail/bin/forward. Greetz, Peter -- Against Free Sex! http://www.dataloss.nl/Megahard_en.html
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, 28 Jul 2001, Peter van Dijk wrote: I am not familiar with the internals of qmail, but from what I have seen, this would make sense. Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself, or talk to /var/qmail/bin/forward. Oh, so you're saying if e.g. on mydomain.com I have the file .qmail-pmak that says: [EMAIL PROTECTED] and someone sends e-mail to [EMAIL PROTECTED], the message actually gets injected twice into qmail---first time to send to [EMAIL PROTECTED], then qmail re-injects it again for delivery to [EMAIL PROTECTED]?
Re: Fastforward question (was Re: Mail Forwarding Service)
On Sat, Jul 28, 2001 at 02:22:25PM -0400, Philip Mak allegedly wrote: On Sat, 28 Jul 2001, Peter van Dijk wrote: I am not familiar with the internals of qmail, but from what I have seen, this would make sense. Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself, or talk to /var/qmail/bin/forward. Oh, so you're saying if e.g. on mydomain.com I have the file .qmail-pmak that says: [EMAIL PROTECTED] and someone sends e-mail to [EMAIL PROTECTED], the message actually gets injected twice into qmail---first time to send to [EMAIL PROTECTED], then qmail re-injects it again for delivery to [EMAIL PROTECTED]? Yep. The way to do this optimally is to have a program that does the lookup and then execs /var/qmail/bin/forward (or possibly qmail-queue for a minor performance gain). Regards.
