Work on Postfix continues to make it more scalable, after the system
was made feature-complete in the past couple years. One recent
example is Postscreen, which focuses on the increasing botnet threat,
and keeping zombie SMTP clients away from the SMTP servers.
This note discusses an approach to improve before-queue content
inspection performance, and to streamline local mail submission.
The first is more important as it can allow a system to handle more
mail with the same memory resources.
Wietse
Non-persistent queues, speed-matching and fall-back
===================================================
Imagine a stream of data from some source, that needs to be processed
by some processing software (a sink). There are several scenarios
where it can be helpful to save the data stream to a non-persistent
buffer before handing that data over to the processing software.
Example 1: data arrives slowly, and we don't want to allocate
expensive processing resources before all the data has arrived.
In the case of the Postfix mail system, it may take a long time
to receive an email message over the network, compared to the
time to send that same message into a local before-queue content
filter. We don't want to tie up an entire content filter
instance while the data trickles in via the network, because
each filter instance consumes a significant amount of memory
and other resources. If we could collect the message in a
non-persistent buffer before invoking the content filter, then
we would be able to run fewer content filter instances at the
same time, and the filter instances would be better utilized.
Another example of speed-matching involves reputation-based
throttling, where remote SMTP clients with poor or no reputation
can send mail only at limited bit rates. As bitrates decrease,
sessions durations increase. We can reduce the number of mail
handling processes by collecting each rate-limited message to
a non-persistent buffer before handing that message to the rest
of the mail infrastructure.
Example 2: when the preferred service is unavailable, give the data
to a backup service.
In the case of Postfix, the postdrop program injects a mail
submission from a local user into the mail system. It would be
good for performance if postdrop could hand over a message
directly to the cleanup server. However, the postdrop program
would need to be able to save the message to the maildrop queue
when the Postfix system is down. Besides, it is undesirable
that postdrop ties up a cleanup process while receiving mail
from some process that takes a long time to complete. The
solution is that postdrop saves the entire mail submission to
a non-persistent buffer before it contacts the cleanup service.
Example 3: the source produces data faster than the sink can consume
it.
If the sink is slow, but it can still process the data before
the source times out, then there is no benefit from inserting
a non-persistent buffer between source and sink. It would
actually cause more timeout errors, because a non-persistent
buffer disables the flow control mechanism that is implemented
by the operating system.
If the sink takes a significant time to process the data, a
persistent buffer is preferable, to prevent the source from
timing out before the sink reports completion. In the case of
Postfix, this is implemented by the persistent queue in
combination with post-queue content inspection.
Implementation for postdrop/cleanup
===================================
The modified postdrop program flow is straightforward: save the
request to file, and when the input is complete, pipe the message
into a cleanup server. If the cleanup server is happy, delete the
file, otherwise leave the file in the maildrop queue and mark it
as ready.
This can be implemented at a relatively low level in the existing
mail_stream module, by adding a new API that takes both a server
name and a queue file name. The new API would combine features of
the existing mail_stream_service() and mail_stream_file() functions.
This would initially save the input to file, try to dump the file
to the server, and leave the file in the queue when the server is
unavailable.
Under the covers, it would be possible to buffer small messages in
memory and to use the disk only for overflow or persistence.
Implementation for before-queue content filter
==============================================
In this case, the implementation is a little trickier, because
the Postfix SMTP server connects to the before-queue filter over
SMTP, which is a chatty protocol.
In the present, unbuffered, case the replies from the before-queue
filter alternate with commands from the before-filter smtpd process.
filter banner
ehlo command
filter reply
...
data command
filter reply 3xx
header+body
.
filter reply
quit command
In the buffered case, the before-filter smtpd writes its SMTP
commands to a replay log, together with the expected before-queue
filter SMTP replies. Only when the replay log is complete, the
before-filter smtpd contacts the before-queue content filter. While
quickly replaying the log, the before-filter smtpd compares the
expected before-queue filter replies in the replay log, against the
actual filter replies, and declares a "queue file write error" when
there is a mis-match. Only the filter's reply to "." is sent back
to the remote client in real time.
This is what "nice dump" of the replay log would look like:
expect 2xx
send ehlo myhostname
expect 2xx
...
send data
expect 3xx
send text
send text
send text
send .
expect 2xx
send quit command
Implementation break-down
-------------------------
The smtpd_proxy_cmd() function already encapsulates the handling
of a before-filter smtpd request and the corresponding expected
before-queue filter reply. The smtpd_proxy module also provides an
API for email message content I/O. The non-persistent buffer can
therefore be added with only low-level changes.
The new functionality involves opening the replay log file and
unlinking it immediately, writing commands and expected replies to
the replay log, and the replayer that sends commands to the content
filter, and that reuses the existing code to parse filter replies.
There is some opportunity for optimization: the same unlinked file
can be used to buffer the content of successive email transactions
by the same smtpd process. In practice, the non-persistent buffer
will sit in the file system cache and never hit the disk, so the
disk overhead will be small.
One tweak is where to open the file; it does not have to live in
the same file system as the Postfix queue (but there would have to
be enough free space to save the largest allowed message).
Finally, small messages could be buffered entirely in memory.
This is an space/time trade-off that can be added later.
