Re: zero-fork delivery?
Miroslav Zubcic <[EMAIL PROTECTED]> writes: > I'm using this: (m4 for sendmail.cf). > > MAILER_DEFINITIONS > Mcyrus, P=[IPC], F=lsDFMnqA@/:|SmXz, E=\r\n, > S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, > A=FILE /var/imap/socket/lmtp > > deliver (that is, MAILER(`cyrus')dnl) is excluded from sendmail. > > Hope it helps... Yes, it does, thanks. I'm now using this: MAILER(local) define(`CYRUS_MAILER_PATH', `[IPC]') define(`CYRUS_MAILER_FLAGS', `A5h@/:|SuwmXz') define(`CYRUS_MAILER_ARGS', `FILE /var/imap/socket/lmtp') define(`CYRUS_BB_MAILER_FLAGS', `A5h@/:|SuwmXz') define(`CYRUS_BB_MAILER_ARGS', `FILE /var/imap/socket/lmtp') MAILER(cyrus) define(`confLOCAL_MAILER',`cyrus') and I'm down to 2 forks (sendmail + cyrus) per delivery. Cool. I added the `8' flag to instruct sendmail not to do automatic 8to7 MIME mapping (Cyrus should handle 8-bit mail, one would assume), but I wasn't able to trigger any difference in behaviour, so I reverted it. One could also add `6' to make sendmail strip 8-bit characters in headers, rather than Cyrus replacing them with `X'. (Also, IMHO there should be a CYRUS_BB_MAILER_PATH -- you get a nonsensical configuration if you tweak CYRUS_MAILER* without also tweaking CYRUS_BB_MAILER*.)
Re: zero-fork delivery?
Lawrence Greenfield <[EMAIL PROTECTED]> writes: > Right you are; running Sendmail/deliver/lmtpd is an extremely > inefficient way to deliver mail. Long message follows. Very informative, thanks! > * Don't run an MTA on the Cyrus server > > Using LMTP over TCP and LMTP AUTH, you can completely avoid running an > MTA on your Cyrus server. When we finish the next round of upgrades, > we'll hopefully be doing this. This way you only pay for the cost of > the lmtpd but not queue manangement. Yes, this will probably be what I'll be doing if my server become too loaded. Or buy more hardware...
Re: zero-fork delivery?
Right you are; running Sendmail/deliver/lmtpd is an extremely inefficient way to deliver mail. Long message follows. * Normal configuration The way I normally expect people to configure Cyrus is to have their MTA talk LMTP directly to lmtpd. Smarter MTAs will reuse the connection. For instance, during a queue run Sendmail will only open one LMTP connection (= 1 lmtpd process) and shove as much mail down it as possible. You can see how to configure Sendmail and LMTP in the cyrus/doc/cyrusv2.mc file. * Default Sendmail inefficiences By default, Sendmail runs in DeliveryMode=background. When the load gets very large, it turns out that just the Sendmail fork() plus the lmtpd fork() for each message (the default thing Sendmail tries first) is quite expensive. Here's a scenario: mx.andrew.cmu.edu accepts a message and attempts delivery to cyrus.andrew.cmu.edu. - sendmail LISTENER fork()s a sendmail process to talk SMTP to mx.andrew. - sendmail SMTP gets EHLO, MAIL FROM:<> and fork()s - sendmail SMTP2 gets RCPT TO:<>, RCPT TO:<>, ..., DATA. - sendmail SMTP2 fork()s for delivery attempt - sendmail SMTP3 attempts to connect to /var/imap/socket/lmtpd - cyrus MASTER fork()s - cyrus CHILD exec()s lmtpd, lmtpd accepts the connection - SMTP3 <-> CHILD talk LMTP to deliver the message. (Note that this is still considerably better than using "deliver".) Sendmail 8.12 should reduce the number of forks considerably. * Our solution Since our volume of mail is large enough that interactive delivery (described above) pounds on the machine, we run Sendmail in DeliverMode=queue, as in: /usr/sbin/sendmail -O DeliveryMode=q -O QueueDirectory=/var/spool/mqueue/fast -bd -q1m All messages are placed into the queue and delivery is attempted by a queue run every minute. Every five minutes, any message older than five minutes in /var/spool/mqueue/fast is moved to /var/spool/mqueue/slow. This is done using the "re-mqueue.pl" script distributed with Sendmail (though using "qtool.pl" is probably a better idea). Every 1/2 hour, a different sendmail tries to deliver /var/spool/mqueue/slow. (It mostly contains messages for users who are over quota.) This way, an LMTP is fork()d for each queue run (1 per minute + 1 per 1/2 hour) and the amount of forking Sendmail does when attempting to deliver the message is greatly reduced. * Don't run an MTA on the Cyrus server Using LMTP over TCP and LMTP AUTH, you can completely avoid running an MTA on your Cyrus server. When we finish the next round of upgrades, we'll hopefully be doing this. This way you only pay for the cost of the lmtpd but not queue manangement. * Other possibilities Postfix may make much smarter use of LMTP connection caching so that messages can be tried immediately without running the system into the ground. * Future work for Cyrus lmtpd could stick around after one client disconnects and listen for another client; in fact, the "master" architecture was designed to make this possible, but I haven't done the work in lmtpd to make it clean up when one client disconnects. Larry From: Simon Josefsson <[EMAIL PROTECTED]> Date: 16 Feb 2001 22:41:43 +0100 I'm using sendmail 8.11.2 and cyrus imapd 2.0.11, currently using "deliver" as the delivery agent. This forks two processes ("deliver" and "lmtpd") for incoming mail, right? (assuming prefork=0) As far as I can tell sendmail doesn't keep the LMTP connection up between incoming connections either. Switching to FEATURE(local_lmtp) in sendmail seem to make sendmail use "mail.local" instead. It doesn't seem to reduce fork()s. Am I misstaken? It doesn't seem very difficult to achieve zero-fork delivery. Without investigating it further, it feels as if it might reduce load on a busy system. Opinions?
Re: zero-fork delivery?
Simon Josefsson <[EMAIL PROTECTED]> writes: > I'm using sendmail 8.11.2 and cyrus imapd 2.0.11, currently using > "deliver" as the delivery agent. This forks two processes ("deliver" > and "lmtpd") for incoming mail, right? (assuming prefork=0) As far as > I can tell sendmail doesn't keep the LMTP connection up between > incoming connections either. Use only lmtpd without deliver. > Switching to FEATURE(local_lmtp) in sendmail seem to make sendmail use > "mail.local" instead. It doesn't seem to reduce fork()s. Am I > misstaken? I'm using this: (m4 for sendmail.cf). MAILER_DEFINITIONS Mcyrus, P=[IPC], F=lsDFMnqA@/:|SmXz, E=\r\n, S=EnvFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, A=FILE /var/imap/socket/lmtp deliver (that is, MAILER(`cyrus')dnl) is excluded from sendmail. Hope it helps... -- This signature intentionally left blank
zero-fork delivery?
I'm using sendmail 8.11.2 and cyrus imapd 2.0.11, currently using "deliver" as the delivery agent. This forks two processes ("deliver" and "lmtpd") for incoming mail, right? (assuming prefork=0) As far as I can tell sendmail doesn't keep the LMTP connection up between incoming connections either. Switching to FEATURE(local_lmtp) in sendmail seem to make sendmail use "mail.local" instead. It doesn't seem to reduce fork()s. Am I misstaken? It doesn't seem very difficult to achieve zero-fork delivery. Without investigating it further, it feels as if it might reduce load on a busy system. Opinions?