Re: Conditional relayhost based on message size
Demi M. Obenour: > >> I am not sure whether adding such a feature motivated by just the > >> presented use-case is warranted. > > > > My take: not if it only solves only one problem. > > I agree. In particular, we should avoid a pile of features with very > narrow use-cases. I think that this narrow problem can/should be solved with a combination of smtp_delivery_status_filter that changes 5XX message size errors into 4XX, plus an smtp_fallback_relay with more generous size limits. Or just use that fallback relay all the time. Wuetse
Re: Conditional relayhost based on message size
On 1/16/21 5:12 PM, Wietse Venema wrote: > Viktor Dukhovni: >> On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote: >> >>> On Sat, Jan 16, 2021 at 08:14:34AM +, Alexander wrote: >>> My goal is to conditionally select the relayhost based on the total size of the outgoing message. The rationale is that I'm using Amazon AWS SES for the most part. Alas, SES only accepts messages up to 10 MB in size (this includes images and attachments that are part of the message), and when the outgoing message exceeds that size, it will be bounced. Any tips for filters or tools that would help me define another relayhost based on the outgoing message size would be greatly appreciated! >>> >>> Since the queue manager has no mechanism for size-dependent selection of >>> a default transport (with the help of trivial-rewrite), the only way >>> handle this deterministically in Postfix is to shunt large messages into >>> a separate instance, which can be done in a number of ways, but is not >>> particularly simple or elegant. >> >> For this to be naturally supported in Postfix, we'd need a new feature, >> namely a variant of "FILTER" that overrides *just* the default_transport, >> making it possible to use more general criteria than "sender-dependent" >> for default_transport selection. >> >> Possible design: >> >> 1. New access(5) verb taking an associated transport value that >> triggers such an override. >> 2. New queue-file record type holding the default-transport >> override. >> 3. Queue manager loads this record into memory in the same way >> as "FILTER". If both are present, "FILTER" wins. >> 4. Queue manager sends its value along with all trivial-rewrite >> "resolve" requests. >> 5. trivial-rewrite uses this default transport rather than >> $default_transport when the destination domain is not >> in any of the more specific address classes, and there >> is no explicit transport(5) mapping for the recipient. >> >> This would be primarily useful only when using a relayhost, subject to >> secondary criteria to divert some messages to a different transport >> and/or nexthop based on ad-hoc message features detected during >> message reception. The difference from "FILTER" is that messages >> to local recipients are not affected. >> >> I am not sure whether adding such a feature motivated by just the >> presented use-case is warranted. > > My take: not if it only solves only one problem. I agree. In particular, we should avoid a pile of features with very narrow use-cases. > For example, I have several notes on making the scheduling (within > a delivery transport) dependent on a concept of 'cost', which could > be one "number of delivery requests with similar sender or domain", > "message size", "number of delivery requests with similar receiver > or domain", or some weighted combination. Just compute a cost and > schedule as appropriate for that cost bucket. > > If we make routing dependent on message properties, we could take > a page from trivial-rewrite which has two flavors of routing, one > for probes and one for non-probes. We could have N flavors of > routing and select the flavor based on the weighted computation of > something. > > Wietse Personally, I would like to be able to control this via a plugin of some sort. Declarative configuration is awesome, but it eventually reaches its limits. Would the existing milter system be a good fit for this? If not, I wonder if an embedded Lua interpreter would be a good idea. Lua is *tiny* (much smaller than Postfix itself), extremely portable, and reasonably fast for an interpreted language. There is also LuaJIT, which is extremely fast, while still being smaller than Postfix. Embedding an interpreter would allow users to express their business rules in a full programming language, rather than being limited to what Postfix supports natively. Sincerely, Demi OpenPGP_0xB288B55FFF9C22C1.asc Description: application/pgp-keys OpenPGP_signature Description: OpenPGP digital signature
Re: Conditional relayhost based on message size
On 16 Jan 2021, at 02:48, Viktor Dukhovni wrote: > Since the queue manager has no mechanism for size-dependent selection of > a default transport Postfix does, however, know the size of the message based if it is over or under the message_size_limit, and it must know that before the message hits the question manager, right? Mightn't it be possible to hook into that size check pre-queue without. Great deal of faff? -- C code. C code run. Run, code, run.
Re: Conditional relayhost based on message size
Viktor Dukhovni: > On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote: > > > On Sat, Jan 16, 2021 at 08:14:34AM +, Alexander wrote: > > > > > My goal is to conditionally select the relayhost based on the total size > > > of the outgoing message. The rationale is that I'm using Amazon AWS SES > > > for the most part. Alas, SES only accepts messages up to 10 MB in size > > > (this includes images and attachments that are part of the message), and > > > when the outgoing message exceeds that size, it will be bounced. > > > > > > Any tips for filters or tools that would help me define another > > > relayhost based on the outgoing message size would be greatly > > > appreciated! > > > > Since the queue manager has no mechanism for size-dependent selection of > > a default transport (with the help of trivial-rewrite), the only way > > handle this deterministically in Postfix is to shunt large messages into > > a separate instance, which can be done in a number of ways, but is not > > particularly simple or elegant. > > For this to be naturally supported in Postfix, we'd need a new feature, > namely a variant of "FILTER" that overrides *just* the default_transport, > making it possible to use more general criteria than "sender-dependent" > for default_transport selection. > > Possible design: > > 1. New access(5) verb taking an associated transport value that > triggers such an override. > 2. New queue-file record type holding the default-transport > override. > 3. Queue manager loads this record into memory in the same way > as "FILTER". If both are present, "FILTER" wins. > 4. Queue manager sends its value along with all trivial-rewrite > "resolve" requests. > 5. trivial-rewrite uses this default transport rather than > $default_transport when the destination domain is not > in any of the more specific address classes, and there > is no explicit transport(5) mapping for the recipient. > > This would be primarily useful only when using a relayhost, subject to > secondary criteria to divert some messages to a different transport > and/or nexthop based on ad-hoc message features detected during > message reception. The difference from "FILTER" is that messages > to local recipients are not affected. > > I am not sure whether adding such a feature motivated by just the > presented use-case is warranted. My take: not if it only solves only one problem. For example, I have several notes on making the scheduling (within a delivery transport) dependent on a concept of 'cost', which could be one "number of delivery requests with similar sender or domain", "message size", "number of delivery requests with similar receiver or domain", or some weighted combination. Just compute a cost and schedule as appropriate for that cost bucket. If we make routing dependent on message properties, we could take a page from trivial-rewrite which has two flavors of routing, one for probes and one for non-probes. We could have N flavors of routing and select the flavor based on the weighted computation of something. Wietse
Re: Conditional relayhost based on message size
On Sat, Jan 16, 2021 at 04:48:22AM -0500, Viktor Dukhovni wrote: > On Sat, Jan 16, 2021 at 08:14:34AM +, Alexander wrote: > > > My goal is to conditionally select the relayhost based on the total size > > of the outgoing message. The rationale is that I'm using Amazon AWS SES > > for the most part. Alas, SES only accepts messages up to 10 MB in size > > (this includes images and attachments that are part of the message), and > > when the outgoing message exceeds that size, it will be bounced. > > > > Any tips for filters or tools that would help me define another > > relayhost based on the outgoing message size would be greatly > > appreciated! > > Since the queue manager has no mechanism for size-dependent selection of > a default transport (with the help of trivial-rewrite), the only way > handle this deterministically in Postfix is to shunt large messages into > a separate instance, which can be done in a number of ways, but is not > particularly simple or elegant. For this to be naturally supported in Postfix, we'd need a new feature, namely a variant of "FILTER" that overrides *just* the default_transport, making it possible to use more general criteria than "sender-dependent" for default_transport selection. Possible design: 1. New access(5) verb taking an associated transport value that triggers such an override. 2. New queue-file record type holding the default-transport override. 3. Queue manager loads this record into memory in the same way as "FILTER". If both are present, "FILTER" wins. 4. Queue manager sends its value along with all trivial-rewrite "resolve" requests. 5. trivial-rewrite uses this default transport rather than $default_transport when the destination domain is not in any of the more specific address classes, and there is no explicit transport(5) mapping for the recipient. This would be primarily useful only when using a relayhost, subject to secondary criteria to divert some messages to a different transport and/or nexthop based on ad-hoc message features detected during message reception. The difference from "FILTER" is that messages to local recipients are not affected. I am not sure whether adding such a feature motivated by just the presented use-case is warranted. -- Viktor.
Re: Conditional relayhost based on message size
On Sat, Jan 16, 2021 at 08:14:34AM +, Alexander wrote: > My goal is to conditionally select the relayhost based on the total size > of the outgoing message. The rationale is that I'm using Amazon AWS SES > for the most part. Alas, SES only accepts messages up to 10 MB in size > (this includes images and attachments that are part of the message), and > when the outgoing message exceeds that size, it will be bounced. > > Any tips for filters or tools that would help me define another > relayhost based on the outgoing message size would be greatly > appreciated! Since the queue manager has no mechanism for size-dependent selection of a default transport (with the help of trivial-rewrite), the only way handle this deterministically in Postfix is to shunt large messages into a separate instance, which can be done in a number of ways, but is not particularly simple or elegant. The simplest is perhaps to allow the message to route to AWS, where it would typically rejected in response to "MAIL FROM" when Postfix announces the message size, and to map that response to a tempfail, at which point the message would go the SMTP fallback relay for the transport, but this can be complex if amazon publishes many IP addresses for their submission service, since Postfix would try those first and may give up after a small number of connections. You'd therefore need to also ensure that the AWS relay resolves to just 1 IP address. That could be done with Python hooks in unbound, or a static IP (bypassing live DNS) that's updated out of band. If you want to take the second instance route, then it may be simplest to place large messages on hold, and within the same filesystem, move messages from the hold queue to the incoming queue of the second instance. Alternatively, an SMTP content_filter could dynamically choose the reinjection port based on message size, but you'd need to implement such an SMTP proxy. Finally, you could always try Exim. For sufficiently exotic requirements, though I'm generally loathe to recomment a complex monolithic MTA with a spotty security track record, it does offer a lot more conditional logic in its mail processing pipeline. You might even use Exim as a content_filter for Postfix, and have Postfix ultimately deliver the outbound message, with Exim routing the message to one of two Postfix instances. Though it is rather overkill if all you want is a size-dependent SMTP switch. -- Viktor.
Conditional relayhost based on message size
Hello, My goal is to conditionally select the relayhost based on the total size of the outgoing message. The rationale is that I'm using Amazon AWS SES for the most part. Alas, SES only accepts messages up to 10 MB in size (this includes images and attachments that are part of the message), and when the outgoing message exceeds that size, it will be bounced. Any tips for filters or tools that would help me define another relayhost based on the outgoing message size would be greatly appreciated! Alex
