Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

On Sat, Jul 28, 2001 at 02:22:25PM -0400, Philip Mak allegedly wrote:
> On Sat, 28 Jul 2001, Peter van Dijk wrote:
> 
> > > I am not familiar with the internals of qmail, but from what I have seen,
> > > this would make sense.
> >
> > Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself,
> > or talk to /var/qmail/bin/forward.
> 
> Oh, so you're saying if e.g. on mydomain.com I have the file .qmail-pmak
> that says:
> 
> &[EMAIL PROTECTED]
> 
> and someone sends e-mail to [EMAIL PROTECTED], the message actually gets
> injected twice into qmail---first time to send to [EMAIL PROTECTED], then
> qmail re-injects it again for delivery to [EMAIL PROTECTED]?

Yep.

The way to do this optimally is to have a program that does the lookup
and then execs /var/qmail/bin/forward (or possibly qmail-queue for a
minor performance gain).


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Philip Mak]

On Sat, 28 Jul 2001, Peter van Dijk wrote:

> > I am not familiar with the internals of qmail, but from what I have seen,
> > this would make sense.
>
> Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself,
> or talk to /var/qmail/bin/forward.

Oh, so you're saying if e.g. on mydomain.com I have the file .qmail-pmak
that says:

&[EMAIL PROTECTED]

and someone sends e-mail to [EMAIL PROTECTED], the message actually gets
injected twice into qmail---first time to send to [EMAIL PROTECTED], then
qmail re-injects it again for delivery to [EMAIL PROTECTED]?

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 12:42:14PM -0400, Philip Mak wrote:
[snip]
> I am not familiar with the internals of qmail, but from what I have seen,
> this would make sense.

Yes. This program could then just talk to /var/qmail/bin/qmail-queue itself,
or talk to /var/qmail/bin/forward.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 03:36:02PM +, MarkD wrote:
[snip]
> It'll be interesting to see how you propose to atomically make such
> queue changes while incurring a worthwhile queueing cost saving.

I have no such proposal. I just feel that with some changes to the
queueing structure, this might be feasible. On a 100mbyte mail, this
saves reading+writing 100mbyte when a mail is forwarded.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[Philip Mak]

On Sat, 28 Jul 2001, Adrian Ho wrote:

> Unless the destination address happens to be in a virtual domain on the
> same machine, in which case the standard reinjection actually trumps the
> above by one unneeded SMTP transaction from the machine to itself.
>
> In any case, it sounds to me like we're entering the realm of pinhole
> optimization (or some equivalent concept).  Is the performance boost
> worth the kinks it'll likely introduce in the existing qmail architecture?
> I'm not sure...

Is it really that complicated to get the forwarding alias from a program?
I'm thinking---at the moment when qmail is reading the .qmail-default
file, it can encounter:

&[EMAIL PROTECTED]

At this point, it has the capability of specifying a local or remote
e-mail address to forward the message to. Doesn't it?

So it should also be able to easily run an external program at this point
to determine the e-mail address to forward to.

I am not familiar with the internals of qmail, but from what I have seen,
this would make sense.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Adrian Ho]

On Sat, Jul 28, 2001 at 04:21:29PM +0200, Peter van Dijk wrote:
> On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho wrote:
> >   |forward `my-redirector $RECIPIENT`
> 
> That's not what he means. This still reads the message and reinjects
> it.

Oops!  That means it's time to hit the sack.  8-)

But since I'm still (barely) conscious...

> His proposal (which I have been pondering about for months already
> :) means that a program can tell qmail 'send this mail you are trying
> to give to me, to this address' without reinjection. This could save a
> lot of disk bandwidth, IMHO.

Unless the destination address happens to be in a virtual domain on the
same machine, in which case the standard reinjection actually trumps the
above by one unneeded SMTP transaction from the machine to itself.

In any case, it sounds to me like we're entering the realm of pinhole
optimization (or some equivalent concept).  Is the performance boost
worth the kinks it'll likely introduce in the existing qmail architecture?
I'm not sure...

-- 
Adrian HoTinker, Drifter, Fixer, Bum   [EMAIL PROTECTED]
ListArchive: 
Useful URLs:  
  

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 11:20:37AM -0400, Philip Mak wrote:
[snip]
> I'm also considering putting in my .qmail-default file:
> 
> |forward `database-lookup $RECIPIENT`
> 
> where database-lookup is a simple C program that connects to MySQL, looks
> up the recipient in the database, and prints it to standard output.


You may want to look at dteq (www.dataloss.nl/software/dteq/) in that
case, which is a commandline mysql query tool


However, using `` allows you no way to detect errors in the query. If
the mysqld is down, where does the mail go?

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

> My rough guess (see grep | wc above) is 7000 deliveries per hour.

About 2 a second, that's not huge, but it's starting to get busy when
you have to invoke multiple programs and establish a socket each time
to your database.

> I see "status: local 10/10" a lot. It goes back down in a few seconds,
> then can come back up in a few seconds too. (Should I increase
> concurrencylocal?)

No. quite the opposite if anything. Think of concurrencylocal as a
peak load that you want qmail to impose on your database - or your
local file system for that matter.

Do you really want to have more than 10 concurrent connections to your
database? Better to keep that number safely within the capabilities of
your system.

I suspect that 10 is a good starting point for your delivery rates and
it's much better to have an occassional local delivery delayed than to
grind your database into dust.

> |forward `database-lookup $RECIPIENT`
> 
> where database-lookup is a simple C program that connects to MySQL, looks
> up the recipient in the database, and prints it to standard output.
> 
> This would be easy to write. It might not be as efficient as fastforward
> due to having to open a new connection to MySQL every time, but if it's
> "efficient enough" I think it's better because then (1) the database is
> always in perfect synch and (2) I don't have to worry about cron jobs to
> synchronize the fastforward db with the MySQL db. I'll have to try it and
> see what happens.

That sounds like a good plan.


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

> > The qmail architecture does not lend itself well to this though does
> > it? qmail-remote is the only code that knows how to remotely deliver a
> > message and qmail-smtpd would have to be (extensively) modified to
> > call that instead of qmail-queue.
> 
> You are missing the point. We are just saying that a program invoked
> by qmail-local should have a way to communicate back to qmail 'change
> the address to blah', instead of having to reinject it. This would
> then still happen for every recipient like it does now.

Ug. That's even harder and it saves less than half of your queuing
costs!

That approach means that the message changes from a local to a remote
delivery - the queue structure does not lend itself to making this
change easily without incurring most of the cost of a queue
injection. It also likely that the length of the recipient address
will change - again the queue structure is poorly suited to this for
multiple recipient emails as recipients are stored as a series of \0
terminated strings.

It'll be interesting to see how you propose to atomically make such
queue changes while incurring a worthwhile queueing cost saving.


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 02:50:15PM +, MarkD wrote:
[snip]
> At this stage, periodic rebuilding of a fastforward file sure sounds
> easiest - perhaps triggered by database changes.

A 'select * from ...' followed by a fastforward cdb rebuild should
pose no interesting load when executed, say, every 5 minutes,
depending on volume. For 24.000 aliases (just assuming for now that
you have about as much aliases as users), every minute can be
feasible.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 02:44:08PM +, MarkD wrote:
> > That's not what he means. This still reads the message and reinjects
> > it. His proposal (which I have been pondering about for months already
> > :) means that a program can tell qmail 'send this mail you are trying
> > to give to me, to this address' without reinjection. This could save a
> > lot of disk bandwidth, IMHO.
> 
> The qmail architecture does not lend itself well to this though does
> it? qmail-remote is the only code that knows how to remotely deliver a
> message and qmail-smtpd would have to be (extensively) modified to
> call that instead of qmail-queue.

You are missing the point. We are just saying that a program invoked
by qmail-local should have a way to communicate back to qmail 'change
the address to blah', instead of having to reinject it. This would
then still happen for every recipient like it does now.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

> 24000 total users on a Pentium III 850MHz with 768 MB of RAM (not sure
> how many are active though...at least a couple thousand).

By volume I meant how many emails per hour. Number of users is largely
irrelevant.

> Upon activating this system, the load average of the machine has increased
> from 1-2 to 10! I suspect most of the time is being spent compiling the
> perl script and connecting to the MySQL database, though. If I switch to

If you're doing this per delivery, I'm not surprised. But it should be
easy to measure for sure with vmstat/top/acct, etc.

> fastforward (or if I rewrite the script in C, and use a persistent
> database connection handle somehow, maybe by storing it in an flock'd
> file) maybe the load average will drop back down to normal.

Maintaining a persistent connection across multiple local deliveries
is possible with some skull-hackery and a cooperating peer process,
but it's not easy, it's not possible using flock and it does raise the
issue of multiple deliveries using the same connection at the same
instant.

Tell us more about the deliveries? How many per hour, what is your
concurrencylocal? Are the deliveries keeping up? An unadulterated
snapshot of your qmail log would tell us a lot.

At this stage, periodic rebuilding of a fastforward file sure sounds
easiest - perhaps triggered by database changes.


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

> That's not what he means. This still reads the message and reinjects
> it. His proposal (which I have been pondering about for months already
> :) means that a program can tell qmail 'send this mail you are trying
> to give to me, to this address' without reinjection. This could save a
> lot of disk bandwidth, IMHO.

The qmail architecture does not lend itself well to this though does
it? qmail-remote is the only code that knows how to remotely deliver a
message and qmail-smtpd would have to be (extensively) modified to
call that instead of qmail-queue.

It would have been a cute touch if DjB had made the interface to
qmail-remote the same as qmail-queue. In fact, one wonders whether all
the inter-program delivery of mail in qmail should use some sort of
common protocol such as that used by qmail-remote. Better yet would be
to universally use QMTP/QMQP between programs.

Anyway, even overcoming the interface obstacles, you have the nasty
problem of inbound multiple recipients to deal with. qmail-remote only
handles multiple recipients if they all happen to be going to the same
domain.

You could simply punt to qmail-queue of course if there is more than
one recipient, but now it's starting to get messy as your delivery
paths will be substantially different for the same recipient simply
depending on whether they are part of an inbound multiple recipient
mail or not.


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 10:29:31AM -0400, Philip Mak wrote:
[snip]
> Upon activating this system, the load average of the machine has increased
> from 1-2 to 10! I suspect most of the time is being spent compiling the
> perl script and connecting to the MySQL database, though. If I switch to
> fastforward (or if I rewrite the script in C, and use a persistent
> database connection handle somehow, maybe by storing it in an flock'd
> file) maybe the load average will drop back down to normal.

You can't store persistent database connection handles in flock'd
files.

There are two ways to persistent database connections:
- do your stuff from within qmail (probably not the right spot)
- have your perl/C program connect to a daemon that has a couple of
  persistent connections

But I don't think this will help anything - most of your time is
probably spent compiling perl.

My recommendation is to use fastforward.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[Philip Mak]

On 28 Jul 2001, MarkD wrote:

> To answer Philip's question: Yes, that overhead is un[avoidable] as
> there is no standard qmail solution for redirecting mail without it
> going thru the queue at least once.
>
> Having said that your concern about overhead may be misplaced. What
> sort of volume are you expecting on what sort of system?

24000 total users on a Pentium III 850MHz with 768 MB of RAM (not sure
how many are active though...at least a couple thousand).

I currently use .qmail-default to run a perl script which connects to a
MySQL database, performs a lookup on the alias, then opens a pipe to
sendmail to deliver the message.

Upon activating this system, the load average of the machine has increased
from 1-2 to 10! I suspect most of the time is being spent compiling the
perl script and connecting to the MySQL database, though. If I switch to
fastforward (or if I rewrite the script in C, and use a persistent
database connection handle somehow, maybe by storing it in an flock'd
file) maybe the load average will drop back down to normal.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho wrote:
> On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote:
> > I wonder if in the future, they'll make an "alias delivery" option in
> > qmail; that is, it calls an external program, but instead of sending the
> > entire message to the program, it just sends the "RCPT TO:" address to the
> > program and the program returns to it which mailbox(es) should be
> > delivered to.
> 
> That's trivially done today -- no extra options required:
> 
>   |forward `my-redirector $RECIPIENT`

That's not what he means. This still reads the message and reinjects
it. His proposal (which I have been pondering about for months already
:) means that a program can tell qmail 'send this mail you are trying
to give to me, to this address' without reinjection. This could save a
lot of disk bandwidth, IMHO.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Re: Fastforward question (was Re: Mail Forwarding Service)

[MarkD]

On Sat, Jul 28, 2001 at 09:35:32PM +0800, Adrian Ho allegedly wrote:
> On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote:
> > I wonder if in the future, they'll make an "alias delivery" option in
> > qmail; that is, it calls an external program, but instead of sending the
> > entire message to the program, it just sends the "RCPT TO:" address to the
> > program and the program returns to it which mailbox(es) should be
> > delivered to.
> 
> That's trivially done today -- no extra options required:
>
>   |forward `my-redirector $RECIPIENT`

Nope.

You cut too much out of the original posting. He said:

> So it would still have the overhead of having to read a message from
> qmail, and then write that message back to qmail. That overhead would be
> unavoidable if I'm doing program delivery, I guess.

In other words he doesn't want each mail to go thru the queue twice as
your solution implies.

To answer Philip's question: Yes, that overhead is unavailable as
there is no standard qmail solution for redirecting mail without it
going thru the queue at least once.

Having said that your concern about overhead may be misplaced. What
sort of volume are you expecting on what sort of system?


Regards.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Adrian Ho]

On Sat, Jul 28, 2001 at 07:28:04AM -0400, Philip Mak wrote:
> I wonder if in the future, they'll make an "alias delivery" option in
> qmail; that is, it calls an external program, but instead of sending the
> entire message to the program, it just sends the "RCPT TO:" address to the
> program and the program returns to it which mailbox(es) should be
> delivered to.

That's trivially done today -- no extra options required:

  |forward `my-redirector $RECIPIENT`

Sounds like you haven't read "The Big Qmail Picture" by Andre Oppermann
.  Three years old and only 4 pages long, but
still very useful for illuminating the little-known corners of qmail.

-- 
Adrian HoTinker, Drifter, Fixer, Bum   [EMAIL PROTECTED]
ListArchive: 
Useful URLs:  
  

Re: Fastforward question (was Re: Mail Forwarding Service)

[Philip Mak]

On Sat, 28 Jul 2001, Adrian Ho wrote:

> > Hmm, looks like it could work. The "Speed tests" section of
> > http://cr.yp.to/fastforward.html says that it takes only 6 seconds to
> > regenerate an alias db with 5 entries. I could run a cron job every
> > two hours to regenerate the cdb.
>
> I'd be more worried about speed of delivery than speed of DB regeneration.
> Note that it's still a program delivery, albeit done through a more
> efficient program than your existing perlDB script.

Oh, I see now; fastforward is a program that I specify to be called in
.qmail-default. I thought it was a patch to be applied to qmail.

So it would still have the overhead of having to read a message from
qmail, and then write that message back to qmail. That overhead would be
unavoidable if I'm doing program delivery, I guess.

I wonder if in the future, they'll make an "alias delivery" option in
qmail; that is, it calls an external program, but instead of sending the
entire message to the program, it just sends the "RCPT TO:" address to the
program and the program returns to it which mailbox(es) should be
delivered to. Then again, this could turn out to be an ugly piece of
'feature creep'.

Re: Fastforward question (was Re: Mail Forwarding Service)

[Adrian Ho]

On Sat, Jul 28, 2001 at 05:34:59AM -0400, Philip Mak wrote:
> Hmm, looks like it could work. The "Speed tests" section of
> http://cr.yp.to/fastforward.html says that it takes only 6 seconds to
> regenerate an alias db with 5 entries. I could run a cron job every
> two hours to regenerate the cdb.

I'd be more worried about speed of delivery than speed of DB regeneration.
Note that it's still a program delivery, albeit done through a more
efficient program than your existing perlDB script.

As an aside, has anyone done any performance comparisons between
fastforward and .qmail-forwarding for large numbers of aliases (>10,000)?

> My question about fastforward is: Will my existing .qmail-* files stop
> working?

fastforward is traditionally invoked in ~alias/.qmail-default, so unless
you have some other delivery instructions already in that file, everything
else should still work.

-- 
Adrian HoTinker, Drifter, Fixer, Bum   [EMAIL PROTECTED]
ListArchive: 
Useful URLs:  
  

Re: Fastforward question (was Re: Mail Forwarding Service)

[Peter van Dijk]

On Sat, Jul 28, 2001 at 05:34:59AM -0400, Philip Mak wrote:
[snip]
> My question about fastforward is: Will my existing .qmail-* files stop
> working? If so, how can I make the ezmlm aliases still work? e.g. one of

Yes, they will work. fastforward will just sit in .qmail-default and
handle anything that's not being handled by their own .qmail file.

Greetz, Peter
-- 
Against Free Sex!   http://www.dataloss.nl/Megahard_en.html

Fastforward question (was Re: Mail Forwarding Service)

[Philip Mak]

On 28 Jul 2001, Frank D. Cringle wrote:

> Use fastforward - http://cr.yp.to/fastforward.html
>
> Periodically dump the relevant parts of your MySQL database into the
> cdb that fastforward uses.

Hmm, looks like it could work. The "Speed tests" section of
http://cr.yp.to/fastforward.html says that it takes only 6 seconds to
regenerate an alias db with 5 entries. I could run a cron job every
two hours to regenerate the cdb.

My question about fastforward is: Will my existing .qmail-* files stop
working? If so, how can I make the ezmlm aliases still work? e.g. one of
my .qmail files for posting to an announcement list says:

|egrep -i "^From:.*([EMAIL PROTECTED])" || (echo "Permission denied."; exit 100)
|/usr/local/bin/ezmlm/ezmlm-reject
|/usr/local/bin/ezmlm/ezmlm-send '/home/ptscb/lists/buildreferrals'
|/usr/local/bin/ezmlm/ezmlm-warn '/home/ptscb/lists/buildreferrals' || exit 0

I don't think this would work in /etc/aliases, which is supposed to be one
entry per line.