Re: Overriding transport_maps with sender_dependent
On 8/12/21 14:41, Gerard E. Seibert wrote: Have you made any attempt to get your IP 'whitelisted' with Microsoft? Several attempts. If you know of a decent contact I can pursue it further, but even after fighting with their usual ticket people and getting "mitigation" turned on for the sending IP things are still insta-spam-boxed. The MailOps list is filled with people in similar boats, and at least a few have given up and just relay to Microsoft as well. Matt
Re: Overriding transport_maps with sender_dependent
On Thu, 12 Aug 2021 11:56:36 -0400, Matt Corallo stated: >On 8/12/21 09:37, Wietse Venema wrote: >> Matt Corallo: >>> I tried variations of this but never could get it to work - as far >>> as I could tell the nexthop is fully resolved by the time we get to >>> the smtp daemon, so there aren't any relevant settings to override >>> or otherwise set the default on the nexthop there. >> >> In the FILTER command you specify transport AND nexthop. There is >> nothing to be resolved BEFORE the SMTP client. > >Ah, thanks. Sadly I'm not sure this solves the immediate issue either >as it seems to override local domain delivery as well. i.e. it results >in any of the source addresses which would need proxying to external >domains being proxied to local domains as well. > >For context, this doesn't feel like a crazy setup - some users have >external addresses they want to be able to send mail as, which we >(obviously) need to relay via their external providers' smtp with >authentication. For local-domain mails, there are some providers >(*cough* Microsoft *cough*) which treat all mail from low-volume IPs >as spam, no matter what best-practices you comply with, so we want to >relay anything to Microsoft domains out via a third-party provider. > >Its currently working with a second postfix instance and a simple >socketmap program in transport_maps to lookup if a domain's MX is >*.outlook.com. > >Matt Have you made any attempt to get your IP 'whitelisted' with Microsoft? -- Gerard
Re: Overriding transport_maps with sender_dependent
On 8/12/21 09:37, Wietse Venema wrote: Matt Corallo: I tried variations of this but never could get it to work - as far as I could tell the nexthop is fully resolved by the time we get to the smtp daemon, so there aren't any relevant settings to override or otherwise set the default on the nexthop there. In the FILTER command you specify transport AND nexthop. There is nothing to be resolved BEFORE the SMTP client. Ah, thanks. Sadly I'm not sure this solves the immediate issue either as it seems to override local domain delivery as well. i.e. it results in any of the source addresses which would need proxying to external domains being proxied to local domains as well. For context, this doesn't feel like a crazy setup - some users have external addresses they want to be able to send mail as, which we (obviously) need to relay via their external providers' smtp with authentication. For local-domain mails, there are some providers (*cough* Microsoft *cough*) which treat all mail from low-volume IPs as spam, no matter what best-practices you comply with, so we want to relay anything to Microsoft domains out via a third-party provider. Its currently working with a second postfix instance and a simple socketmap program in transport_maps to lookup if a domain's MX is *.outlook.com. Matt
Re: Overriding transport_maps with sender_dependent
Matt Corallo: > I tried variations of this but never could get it to work - as far as I could > tell the nexthop is fully resolved by the > time we get to the smtp daemon, so there aren't any relevant settings to > override or otherwise set the default on the > nexthop there. In the FILTER command you specify transport AND nexthop. There is nothing to be resolved BEFORE the SMTP client. Wietse > Thanks, > Matt > > On 8/11/21 17:37, Wietse Venema wrote: > > Matt Corallo: > >> > >> > >> On 8/11/21 16:52, Wietse Venema wrote: > >> > If the sender address can override the routing, even if the recipient > >> > would otherwise be delivered locally, then that would be a recipe > >> > for mailer loops with the potential for mail explosions. This is > >> > why we have sender_dependent overrides for default transports and > >> > relay hosts, and avoid such stability problems. > >> > >> Ah! Understood, indeed, the setup I've had to fall back to has some risk > >> of routing loops, though with some care to > >> hopefully ensure it can't ever actually be hit. I guess the only solution > >> is multi-key lookups, which would be nice, but > >> understood that its likely very nontrivial to add :). > > > > Would this do the job: > > > > /etc/postfix/main.cf: > > smtpd_sender_restrictions = hash:/etc/postfix/sender_access > > > > /etc/postfix/sender_access > > example.com filter smtp-example-com:relay-for-example-com > > ... > > > > /etc/postfix/master.cf: > > smtp-example-com .. .. .. .. .. .. smtp > > ... > > > > It avoids the need for another instance. Postfix should break a > > mailer loop that delivers to itself. > > > > Wietse > > >
Re: Overriding transport_maps with sender_dependent
I tried variations of this but never could get it to work - as far as I could tell the nexthop is fully resolved by the time we get to the smtp daemon, so there aren't any relevant settings to override or otherwise set the default on the nexthop there. Thanks, Matt On 8/11/21 17:37, Wietse Venema wrote: Matt Corallo: On 8/11/21 16:52, Wietse Venema wrote: > If the sender address can override the routing, even if the recipient > would otherwise be delivered locally, then that would be a recipe > for mailer loops with the potential for mail explosions. This is > why we have sender_dependent overrides for default transports and > relay hosts, and avoid such stability problems. Ah! Understood, indeed, the setup I've had to fall back to has some risk of routing loops, though with some care to hopefully ensure it can't ever actually be hit. I guess the only solution is multi-key lookups, which would be nice, but understood that its likely very nontrivial to add :). Would this do the job: /etc/postfix/main.cf: smtpd_sender_restrictions = hash:/etc/postfix/sender_access /etc/postfix/sender_access example.com filter smtp-example-com:relay-for-example-com ... /etc/postfix/master.cf: smtp-example-com .. .. .. .. .. .. smtp ... It avoids the need for another instance. Postfix should break a mailer loop that delivers to itself. Wietse
Re: Overriding transport_maps with sender_dependent
Matt Corallo: > > > On 8/11/21 16:52, Wietse Venema wrote: > > If the sender address can override the routing, even if the recipient > > would otherwise be delivered locally, then that would be a recipe > > for mailer loops with the potential for mail explosions. This is > > why we have sender_dependent overrides for default transports and > > relay hosts, and avoid such stability problems. > > Ah! Understood, indeed, the setup I've had to fall back to has some risk of > routing loops, though with some care to > hopefully ensure it can't ever actually be hit. I guess the only solution is > multi-key lookups, which would be nice, but > understood that its likely very nontrivial to add :). Would this do the job: /etc/postfix/main.cf: smtpd_sender_restrictions = hash:/etc/postfix/sender_access /etc/postfix/sender_access example.com filter smtp-example-com:relay-for-example-com ... /etc/postfix/master.cf: smtp-example-com .. .. .. .. .. .. smtp ... It avoids the need for another instance. Postfix should break a mailer loop that delivers to itself. Wietse
Re: Overriding transport_maps with sender_dependent
On 8/11/21 16:52, Wietse Venema wrote: > If the sender address can override the routing, even if the recipient > would otherwise be delivered locally, then that would be a recipe > for mailer loops with the potential for mail explosions. This is > why we have sender_dependent overrides for default transports and > relay hosts, and avoid such stability problems. Ah! Understood, indeed, the setup I've had to fall back to has some risk of routing loops, though with some care to hopefully ensure it can't ever actually be hit. I guess the only solution is multi-key lookups, which would be nice, but understood that its likely very nontrivial to add :). On 8/11/21 16:49, post...@ptld.com wrote: I might be off course, but wouldn't a milter cover those requirements? I would love to find that it is, but its not my understanding that a milter can set the smtp nexthop without modifying the mail itself. Thanks, Matt
Re: Overriding transport_maps with sender_dependent
Matt Corallo: > > > On 8/11/21 13:54, Viktor Dukhovni wrote: > >> On 11 Aug 2021, at 11:00 am, Matt Corallo wrote: > >> > >> Hmm, well I suppose consider this a feature request for > >> sender_dependent_relay_transport_maps or sender_dependent_transport_maps :) > > > > No such feature fits into a single-key lookup design. > > > > You're looking to exempt specific sender (domains) from the recipient-based > > nexthop of specific recipient domains. This is a multi-key transport > > decision, and supporting this requires a radically different design. > > > > So the feature request is extremely unlikely to be actionable. > > Correct me if I'm wrong, but it should still be possible to have > a sender-dependent lookup that happens before transport_maps or > relay_transport_maps? Indeed, I understand that no multi-key lookup > can occur, but the decision could be made first by an optional > sender lookup, then by the recipient lookup, then the default, no? If the sender address can override the routing, even if the recipient would otherwise be delivered locally, then that would be a recipe for mailer loops with the potential for mail explosions. This is why we have sender_dependent overrides for default transports and relay hosts, and avoid such stability problems. Wietse > In any case, I'll likely go with a second instance of postfix for now, but > its a lot of additional complexity. > > Matt >
Re: Overriding transport_maps with sender_dependent
On 08-11-2021 4:41 pm, Matt Corallo wrote: Correct me if I'm wrong, but it should still be possible to have a sender-dependent lookup that happens before transport_maps or relay_transport_maps? Indeed, I understand that no multi-key lookup can occur, but the decision could be made first by an optional sender lookup, then by the recipient lookup, then the default, no? In any case, I'll likely go with a second instance of postfix for now, but its a lot of additional complexity. I might be off course, but wouldn't a milter cover those requirements?
Re: Overriding transport_maps with sender_dependent
On 8/11/21 13:54, Viktor Dukhovni wrote: On 11 Aug 2021, at 11:00 am, Matt Corallo wrote: Hmm, well I suppose consider this a feature request for sender_dependent_relay_transport_maps or sender_dependent_transport_maps :) No such feature fits into a single-key lookup design. You're looking to exempt specific sender (domains) from the recipient-based nexthop of specific recipient domains. This is a multi-key transport decision, and supporting this requires a radically different design. So the feature request is extremely unlikely to be actionable. Correct me if I'm wrong, but it should still be possible to have a sender-dependent lookup that happens before transport_maps or relay_transport_maps? Indeed, I understand that no multi-key lookup can occur, but the decision could be made first by an optional sender lookup, then by the recipient lookup, then the default, no? In any case, I'll likely go with a second instance of postfix for now, but its a lot of additional complexity. Matt
Re: Overriding transport_maps with sender_dependent
Viktor Dukhovni: > > On 11 Aug 2021, at 11:00 am, Matt Corallo wrote: > > > > Hmm, well I suppose consider this a feature request for > > sender_dependent_relay_transport_maps or sender_dependent_transport_maps :) > > No such feature fits into a single-key lookup design. > > You're looking to exempt specific sender (domains) from the recipient-based > nexthop of specific recipient domains. This is a multi-key transport > decision, and supporting this requires a radically different design. > > So the feature request is extremely unlikely to be actionable. We solved a multi-criteria decision problem with check_policy_service in Postfix access maps, where a policy server can make a ruling based on multiple attributes. Whet helped here was that many smtpd processes make policy requests in parallel, so that individual response latencies do not add up. Routing decisions are different - they are requested by a single queue manager process, so that individual response latencies will add up unless that part of the scheduler is also parallized (just the handling of delivery requests is parallelized). But then who would implement the server that makes routing decisions? Maybe time to revive the vintage-1997 ideas for a configurable trivial-rewrite service. I abandoned that because it started to look like sendmail.cf. Wietse
Re: Overriding transport_maps with sender_dependent
> On 11 Aug 2021, at 11:00 am, Matt Corallo wrote: > > Hmm, well I suppose consider this a feature request for > sender_dependent_relay_transport_maps or sender_dependent_transport_maps :) No such feature fits into a single-key lookup design. You're looking to exempt specific sender (domains) from the recipient-based nexthop of specific recipient domains. This is a multi-key transport decision, and supporting this requires a radically different design. So the feature request is extremely unlikely to be actionable. -- Viktor.
Re: Overriding transport_maps with sender_dependent
Hmm, well I suppose consider this a feature request for sender_dependent_relay_transport_maps or sender_dependent_transport_maps :) Matt > On Aug 10, 2021, at 23:01, Viktor Dukhovni wrote: > > On Tue, Aug 10, 2021 at 10:34:52PM -0400, Matt Corallo wrote: > >> I have a need to map some destination domains to a specific smtp >> nexthop, but need to override that nexthop on a sender_dependent >> basis. I've tried a few things and all with no luck: > > Sorry, that's not directly possible. Sender-dependent transport > selection overrides only the *default* transport. Anything else would > require looking at both the sender and recipient at the same time, and > Postfix does not presently have support for multi-key tables. > >> Am I missing something or is there some way to do a custom rewrite >> engine? > > It may be possible to rewrite the recipients in question to replace the > domain part with the desired nexthop, so that the default transport > would take care of delivery to the correct nexthop, and this case, > sender-dependent overrides can kick in. You might then need to reverse > the rewrites via smtp_generic_maps. > > Another option is a multi-instance pipeline, in which the recipient > determines the downstream Postfix instance, and then in some special > downstream instances the nexthop in question is the default transport, > and again sender-based overrides can take place. > >> On Tue, Aug 10, 2021 at 10:43:46PM -0400, Matt Corallo wrote: >> Oh, and if its possible, is it also possible to specify the original >> domains as "any domain with an MX of $REGEX" instead of only "any >> recipient domain of $REGEX"? > > No, the queue manager has no access to MX host lookups which happen > during delivery, not transport selection. > > -- > Viktor.
Re: Overriding transport_maps with sender_dependent
On Tue, Aug 10, 2021 at 10:34:52PM -0400, Matt Corallo wrote: > I have a need to map some destination domains to a specific smtp > nexthop, but need to override that nexthop on a sender_dependent > basis. I've tried a few things and all with no luck: Sorry, that's not directly possible. Sender-dependent transport selection overrides only the *default* transport. Anything else would require looking at both the sender and recipient at the same time, and Postfix does not presently have support for multi-key tables. > Am I missing something or is there some way to do a custom rewrite > engine? It may be possible to rewrite the recipients in question to replace the domain part with the desired nexthop, so that the default transport would take care of delivery to the correct nexthop, and this case, sender-dependent overrides can kick in. You might then need to reverse the rewrites via smtp_generic_maps. Another option is a multi-instance pipeline, in which the recipient determines the downstream Postfix instance, and then in some special downstream instances the nexthop in question is the default transport, and again sender-based overrides can take place. On Tue, Aug 10, 2021 at 10:43:46PM -0400, Matt Corallo wrote: > Oh, and if its possible, is it also possible to specify the original > domains as "any domain with an MX of $REGEX" instead of only "any > recipient domain of $REGEX"? No, the queue manager has no access to MX host lookups which happen during delivery, not transport selection. -- Viktor.
Re: Overriding transport_maps with sender_dependent
Oh, and if its possible, is it also possible to specify the original domains as "any domain with an MX of $REGEX" instead of only "any recipient domain of $REGEX"? Thanks, Matt On 8/10/21 22:34, Matt Corallo wrote: I have a need to map some destination domains to a specific smtp nexthop, but need to override that nexthop on a sender_dependent basis. I've tried a few things and all with no luck: * transport_maps specifying the nexthop can't be overridden at all, it seems (and doesn't support sender_dependent matching), * putting the specified destination domains in relay_domains (and stopping relay to them via smtpd_relay_restrictions) but there doesn't seem to be a way to override relay_transport on a sender_dependent way * relayhost-based stuff doesn't seem to be an option as I don't want to change the delivery of mail that isn't either from one of the sender_dependent rules or to one of the specified domains. Am I missing something or is there some way to do a custom rewrite engine? Thanks, Matt
Overriding transport_maps with sender_dependent
I have a need to map some destination domains to a specific smtp nexthop, but need to override that nexthop on a sender_dependent basis. I've tried a few things and all with no luck: * transport_maps specifying the nexthop can't be overridden at all, it seems (and doesn't support sender_dependent matching), * putting the specified destination domains in relay_domains (and stopping relay to them via smtpd_relay_restrictions) but there doesn't seem to be a way to override relay_transport on a sender_dependent way * relayhost-based stuff doesn't seem to be an option as I don't want to change the delivery of mail that isn't either from one of the sender_dependent rules or to one of the specified domains. Am I missing something or is there some way to do a custom rewrite engine? Thanks, Matt
