>>> Short of a threaded qmail-remote (or qmail-send/qmail-remote hybrid), a
>>> pre-forked pool of qmail-remotes waiting on a common socket would probably
be
>>> a significant improvement. In short, Apache-style process management.
>
>Well, remember that Apache is pre-forking the *LISTENERS*, not originators.
>So the exact setup wouldn't exactly be relavent. What you really want to
>be able to do is keep an existing session open to a mail server and shove
>additional messages down it.
The model is exactly the same. In both cases you have a bunch of equivalent
independent processes waiting for work from a single queue; and in fact
qmail-send is *listening* for work from qmail-send (this makes more sense when
you consider pre-forked workers). As far as pipelining is concerned, that's
HTTP/1.1.
But Apache is not really the issue here...
>I realize that DJB has numbers which say it's faster to open 10 independant
>connections than to stream 10 messages over 1 connection. If you're
>already at concurrencyremote I suspect it's just a waste of time to
>shut down a connection when you're just going to open it again.
>
>Also, pre-forking is only really relavent when you have *ANTICIPATED*
>load. If you actually have more messages to deliver, it's not called
>pre-forking. ;-)
The pool doesn't have to be a static size. You define minimum, maximum, and
maximum spare (workers not in active use). If the rate of work increases and
you start consuming your pool of idle workers, you spawn more in response to
the increasing workload. And when you have too many idle workers, you kill
them off.
But certainly a component of this design would be to make the worker procesess
persistent, i.e., they handle more than one connection before exit().
The reason I'm in favor of this model is because (a) it's proven effective for
very similar applications, hence the Apache reference; and (b) I believe it's
possible to modify the qmail-send:qmail-remote interface to support this kind
of model without completely ripping out the guts of qmail.
I really *don't* want to write a completely new MTA, particularly when qmail
already does certain things very well. But there are definitely areas where
performance can be improved.
-Jeff
