Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[deoren]

On 5/10/2016 2:07 PM, Viktor Dukhovni wrote:

On Tue, May 10, 2016 at 01:46:57PM -0500, deoren wrote:


# match on ALL Gmail accounts.
@gmail.com autosieve


I would not expect the messages generated by Sieve to be sent from
"gmail.com".  These auto-generated messages are created on your
machine, and should be sent by some user on your machine (envelope
sender, not rfc822 "From:" header).  Figure out which user sends
Sieve-generated mail.


The incoming Google Calendar notifications from Google's servers are what
I'm attempting to match against. Once I get that working I would enable the
local Sieve script.


That won't work, because "sender_dependent_default_transport_maps"
only modifies the "default" transport, and will almost never affect
the delivery of inbound mail.

I thought you wanted to strip "Auto-Submitted" from outbound mail.
If you need this to happen upstream of inbound delivery to Sieve,
you need a different solution.


Please correct me if I am wrong, but even if I send to the local 
delivery agent that's still not "outbound" from the perspective of 
Postfix, so that wouldn't work either, right?





My goal is to match on the incoming notifications from a remote server,
strip out the Auto-Submitted header so a local Sieve rule will agree to
match against those emails and then generate a new custom notification email
(not using any of the existing text, headers, etc).


I think you need to convince Sieve to selectively violate RFC3834,
perhaps there's a way to do that in Sieve.



If I can't convince Pigeonhole to allow me to violate that RFC, then I 
suppose I would have to do what Wietse mentioned and use a content 
filter. Perhaps accept the whole email, strip the specific header and 
then send back to the submitting email address?


Thanks for your help with this.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Viktor Dukhovni]

On Tue, May 10, 2016 at 01:46:57PM -0500, deoren wrote:

> >># match on ALL Gmail accounts.
> >>@gmail.com autosieve
> >
> >I would not expect the messages generated by Sieve to be sent from
> >"gmail.com".  These auto-generated messages are created on your
> >machine, and should be sent by some user on your machine (envelope
> >sender, not rfc822 "From:" header).  Figure out which user sends
> >Sieve-generated mail.
> 
> The incoming Google Calendar notifications from Google's servers are what
> I'm attempting to match against. Once I get that working I would enable the
> local Sieve script.

That won't work, because "sender_dependent_default_transport_maps"
only modifies the "default" transport, and will almost never affect
the delivery of inbound mail.

I thought you wanted to strip "Auto-Submitted" from outbound mail.
If you need this to happen upstream of inbound delivery to Sieve,
you need a different solution.

> "postfix sender_dependent_default_transport_maps not working"
> 
> and ended up on a conversation you had with Russell Jones about four years
> ago:
> 
> http://postfix.1071664.n5.nabble.com/sender-dependent-default-transport-maps-difficulties-td3402.html#a3409
> 
> Your comment here got me thinking:
> 
> If the delivery is local (via LMTP), would
> sender_dependent_default_transport_maps or even transport_maps even apply?

No.

> My goal is to match on the incoming notifications from a remote server,
> strip out the Auto-Submitted header so a local Sieve rule will agree to
> match against those emails and then generate a new custom notification email
> (not using any of the existing text, headers, etc).

I think you need to convince Sieve to selectively violate RFC3834,
perhaps there's a way to do that in Sieve.

-- 
Viktor.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[deoren]

On 5/10/2016 11:07 AM, Viktor Dukhovni wrote:

On Tue, May 10, 2016 at 01:15:59AM -0500, deoren wrote:


# sender_dependent_default_transport_maps.conf
#
# This entry is matching against gmail.com because Google Calendar
# delivers notifications to your associated Gmail account. To
# receive them on your outside email account you have to configure
# Gmail to forward email. When it does, the Calendar notification
# appears to end up with an envelope sender address of your
# Gmail account. For testing purposes, I am attempting to just
# match on ALL Gmail accounts.
@gmail.com autosieve


I would not expect the messages generated by Sieve to be sent from
"gmail.com".  These auto-generated messages are created on your
machine, and should be sent by some user on your machine (envelope
sender, not rfc822 "From:" header).  Figure out which user sends
Sieve-generated mail.


The incoming Google Calendar notifications from Google's servers are 
what I'm attempting to match against. Once I get that working I would 
enable the local Sieve script.


For my testing I used a Gmail account and sent an email directly from it 
to my primary account on the system. My expectation was that I would see 
the new autosieve transport use logged in the mail log to prove that the 
matching was at least occurring. I didn't see any clues which suggested 
that the transport was being used though.





autosieve  unix  -   -   -   -   -   smtp
  -o syslog_name=postfix/autosieve
  -o smtp_header_checks=$autosieve_header_checks


This is chrooted, which is OK if that works for you, but can be
a source of some problems (though not related to this thread).



Thanks for the heads up. I don't really know what I'm doing there, I 
just copied the existing line from master.cf (Postfix 2.11.x, Ubuntu 
14.04 LTS) and modified the name in an attempt to reflect your example.


As I got ready to do send this I turned to Google and looked for others 
who ran into the same problem:


"postfix sender_dependent_default_transport_maps not working"

and ended up on a conversation you had with Russell Jones about four 
years ago:


http://postfix.1071664.n5.nabble.com/sender-dependent-default-transport-maps-difficulties-td3402.html#a3409

Your comment here got me thinking:

> The sender_dependent_default_transport_maps is a mechanism
in Postfix, that overrides the default transport:nexthop for outbound
mail based on the sender address.

If the delivery is local (via LMTP), would 
sender_dependent_default_transport_maps or even transport_maps even apply?


My goal is to match on the incoming notifications from a remote server, 
strip out the Auto-Submitted header so a local Sieve rule will agree to 
match against those emails and then generate a new custom notification 
email (not using any of the existing text, headers, etc). That custom 
email notification would then be delivered through an email-to-text 
message gateway that the cellular phone service provider allows others 
to use (ex: Verizon's vtext.com).


In other words, the notification that will be generated by the Sieve 
rule will not be coming back to the system unless the remote 
email-to-text gateway server bounces it. If it does, my goal is to have 
the Postfix setup not match on those. This should allow Pigeonhole to 
see the Auto-Submitted header and then refuse to process it, hopefully 
preventing a loop from occurring.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Viktor Dukhovni]

On Tue, May 10, 2016 at 01:15:59AM -0500, deoren wrote:

> # sender_dependent_default_transport_maps.conf
> #
> # This entry is matching against gmail.com because Google Calendar
> # delivers notifications to your associated Gmail account. To
> # receive them on your outside email account you have to configure
> # Gmail to forward email. When it does, the Calendar notification
> # appears to end up with an envelope sender address of your
> # Gmail account. For testing purposes, I am attempting to just
> # match on ALL Gmail accounts.
> @gmail.com autosieve

I would not expect the messages generated by Sieve to be sent from
"gmail.com".  These auto-generated messages are created on your
machine, and should be sent by some user on your machine (envelope
sender, not rfc822 "From:" header).  Figure out which user sends
Sieve-generated mail.

> autosieve  unix  -   -   -   -   -   smtp
>   -o syslog_name=postfix/autosieve
>   -o smtp_header_checks=$autosieve_header_checks

This is chrooted, which is OK if that works for you, but can be
a source of some problems (though not related to this thread).

-- 
Viktor.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[deoren]

On 5/7/2016 2:18 PM, Viktor Dukhovni wrote:
> 
>> On May 7, 2016, at 8:31 AM, Wietse Venema  wrote:
>>
>>> What I'm currently trying to get working is a service entry in master.cf
>>> that has its own header checks conf file. This service would only be
>>> used for specific sender addresses in order to limit the header removal
>>> to just those email notifications generated by Google Calendar.
>>
>> Postfix can't do this on the receiving end, but you can do it on the
>> delivering end.
> 
> Well, Postfix can do this on the sending end via:
> 
> main.cf:
>   # Resolve sieve auto-forwarder sender address to "autosieve" transport
>   sender_dependent_default_transport_maps = ...
>   autosieve_header_checks = pcre:${config_directory}/autosieve.pcre
> 
> master.cf:
>   autosieve unix ... smtp
>   -o smtp_header_checks=$autosieve_header_checks
> 
> autosieve.pcre:
>   /^Auto-Submitted:/  IGNORE
> 
> 
> http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps
> 
> http://www.postfix.org/postconf.5.html#empty_address_default_transport_maps_lookup_key
> http://www.postfix.org/postconf.5.html#smtp_header_checks
> 
> Log inspection will reveal what envelope sender address sieve notices
> are sent by.
> 
> However, dropping "Auto-Submitted:" headers rather risks mail loops
> and need to be done with great care, if at all.  The OP needs to
> be quite sure that he's not generating an email in response to email
> that is a response to a previously generated email ...
> 
> So, in general don't do this.  In specific cases, only after careful
> analysis of the resulting mail flow making sure that the Sieve rule
> can never fire in response to any message that might come back as
> a result of the generated and no longer manifestly "Auto-Submitted"
> notice.
> 
> Read RFC 3834 with care.
> 

Hi Viktor,

Thank you for the detailed reply. Would the following be correct?

# master.cf (two lines)
autosieve  unix  -   -   -   -   -   smtp
  -o smtp_header_checks=$autosieve_header_checks

# main.cf (two lines)
sender_dependent_default_transport_maps =
hash:${config_directory}/sender_dependent_default_transport_maps.conf
autosieve_header_checks = pcre:${config_directory}/autosieve.pcre

# sender_dependent_default_transport_maps.conf
#
# This entry is matching against gmail.com because Google Calendar
# delivers notifications to your associated Gmail account. To
# receive them on your outside email account you have to configure
# Gmail to forward email. When it does, the Calendar notification
# appears to end up with an envelope sender address of your
# Gmail account. For testing purposes, I am attempting to just
# match on ALL Gmail accounts.
@gmail.com autosieve

After not getting any results, I went back an added a test Gmail account
and sent from there to my primary account. No entries in the mail log
any different from a standard delivery. I went back and modified the
master.cf entry like so:

# master.cf (three lines)
autosieve  unix  -   -   -   -   -   smtp
  -o syslog_name=postfix/autosieve
  -o smtp_header_checks=$autosieve_header_checks

At this point I suspect I missed something obvious.

I have the postconf -n and postconf -Mf output ready to post if you
don't spot my mistake above.

Thanks for your help.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Christian Kivalo]

On 2016-05-07 19:16, /dev/rob0 wrote:

On Sat, May 07, 2016 at 11:05:07AM +0200, Christian Kivalo wrote:

On 2016-05-07 10:27, Viktor Dukhovni wrote:
>On Sat, May 07, 2016 at 09:59:00AM +0200, Christian Kivalo wrote:
>
>>>Looking at those emails, I see that
>>>calendar-notificat...@google.com is the source address. I added
>>>the entry to my check_sender_access table with an action of
>>>'FILTER strip-automated-headers:'
>>
>>You could probably get it working with a restriction class.
>>
>>check_sender_access hash:/path/to/sender_access
>>
>>smtpd_restriction_classes: strip_automated_headers
>>
>>strip_automated_headers =
>>  header_checks = pcre:/path/to/strip_automated_headers.pcre
>
>No.  This can't work.
Ok, so i'm wrong. Could you explain why?


Restriction classes are groups of smtpd(8) restrictions.

header_checks(5) is not a smtpd restriction.  In fact it's not even
implemented in smtpd.  See Wietse's reply in this thread.

Thanks, will do.

--
 Christian Kivalo

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[/dev/rob0]

On Sat, May 07, 2016 at 11:05:07AM +0200, Christian Kivalo wrote:
> On 2016-05-07 10:27, Viktor Dukhovni wrote:
> >On Sat, May 07, 2016 at 09:59:00AM +0200, Christian Kivalo wrote:
> >
> >>>Looking at those emails, I see that 
> >>>calendar-notificat...@google.com is the source address. I added 
> >>>the entry to my check_sender_access table with an action of 
> >>>'FILTER strip-automated-headers:'
> >>
> >>You could probably get it working with a restriction class.
> >>
> >>check_sender_access hash:/path/to/sender_access
> >>
> >>smtpd_restriction_classes: strip_automated_headers
> >>
> >>strip_automated_headers =
> >>  header_checks = pcre:/path/to/strip_automated_headers.pcre
> >
> >No.  This can't work.
> Ok, so i'm wrong. Could you explain why?

Restriction classes are groups of smtpd(8) restrictions.

header_checks(5) is not a smtpd restriction.  In fact it's not even 
implemented in smtpd.  See Wietse's reply in this thread.
-- 
  http://rob0.nodns4.us/
  Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Viktor Dukhovni]

> On May 7, 2016, at 8:31 AM, Wietse Venema  wrote:
> 
>> What I'm currently trying to get working is a service entry in master.cf
>> that has its own header checks conf file. This service would only be
>> used for specific sender addresses in order to limit the header removal
>> to just those email notifications generated by Google Calendar.
> 
> Postfix can't do this on the receiving end, but you can do it on the
> delivering end.

Well, Postfix can do this on the sending end via:

main.cf:
# Resolve sieve auto-forwarder sender address to "autosieve" transport
sender_dependent_default_transport_maps = ...
autosieve_header_checks = pcre:${config_directory}/autosieve.pcre

master.cf:
autosieve unix ... smtp
-o smtp_header_checks=$autosieve_header_checks

autosieve.pcre:
/^Auto-Submitted:/  IGNORE


http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps

http://www.postfix.org/postconf.5.html#empty_address_default_transport_maps_lookup_key
http://www.postfix.org/postconf.5.html#smtp_header_checks

Log inspection will reveal what envelope sender address sieve notices
are sent by.

However, dropping "Auto-Submitted:" headers rather risks mail loops
and need to be done with great care, if at all.  The OP needs to
be quite sure that he's not generating an email in response to email
that is a response to a previously generated email ...

So, in general don't do this.  In specific cases, only after careful
analysis of the resulting mail flow making sure that the Sieve rule
can never fire in response to any message that might come back as
a result of the generated and no longer manifestly "Auto-Submitted"
notice.

Read RFC 3834 with care.

-- 
Viktor.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[deoren]

On 5/7/2016 7:31 AM, Wietse Venema wrote:
> deoren:
>> What I'm currently trying to get working is a service entry in master.cf
>> that has its own header checks conf file. This service would only be
>> used for specific sender addresses in order to limit the header removal
>> to just those email notifications generated by Google Calendar.
> 
> Postfix can't do this on the receiving end, but you can do it on the
> delivering end.
> 
> Postfix can do it with an external content filter.
> 
> Postfix can do it with header_checks in the SMTP/LMTP client.  You
> can use transport_maps to direct a specific recipient to a specific
> smtp (or lmtp) client instance in master.cf with its own
> smtp_header_checks (or lmtp_header_checks) in master.cf.
> 
>   Wietse

Thanks for the response and the suggestions. I had another task on my
TODO list that required adding a header based on final recipient; using
a transport map worked great for that task. In particular I could see
the log entries show that the new transport was being used as expected.

I tried a similar approach here and haven't had any luck yet. It's like
none of the recipient virtual aliases that I've listed on the lhs are
matching. Has virtual alias resolution already completed leaving me with
the final recipient? Is that the value I have to use on the lhs?


In master.cf, I added this since the final delivery is via LMTP:

strip-automated-headers unix  -   -   -   -   -   lmtp
  -o syslog_name=postfix/strip-automated-headers
  -o
lmtp_header_checks=regexp:/etc/postfix/strip_automated_headers.conf,regexp:/etc/postfix/header_checks.conf

which is a copy/paste of the existing lmtp transport with a new name and
two passed options (should be seen as three lines with leading spaces
for the options).

When I didn't get any activity in the mail log for that transport I
tried it like so just to be complete:

strip-automated-headers unix  -   -   -   -   -   smtp
  -o syslog_name=postfix/strip-automated-headers
  -o
smtp_header_checks=regexp:/etc/postfix/strip_automated_headers.conf,regexp:/etc/postfix/header_checks.conf

Anything obviously wrong with what I've tried?

Thanks.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Wietse Venema]

deoren:
> What I'm currently trying to get working is a service entry in master.cf
> that has its own header checks conf file. This service would only be
> used for specific sender addresses in order to limit the header removal
> to just those email notifications generated by Google Calendar.

Postfix can't do this on the receiving end, but you can do it on the
delivering end.

Postfix can do it with an external content filter.

Postfix can do it with header_checks in the SMTP/LMTP client.  You
can use transport_maps to direct a specific recipient to a specific
smtp (or lmtp) client instance in master.cf with its own
smtp_header_checks (or lmtp_header_checks) in master.cf.

Wietse

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Christian Kivalo]

On 2016-05-07 10:27, Viktor Dukhovni wrote:

On Sat, May 07, 2016 at 09:59:00AM +0200, Christian Kivalo wrote:


>Looking at those emails, I see that calendar-notificat...@google.com is
>the source address. I added the entry to my check_sender_access table
>with an action of 'FILTER strip-automated-headers:'

You could probably get it working with a restriction class.

check_sender_access hash:/path/to/sender_access

smtpd_restriction_classes: strip_automated_headers

strip_automated_headers =
  header_checks = pcre:/path/to/strip_automated_headers.pcre



No.  This can't work.

Ok, so i'm wrong. Could you explain why?

Thanks
--
 Christian Kivalo

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Viktor Dukhovni]

On Sat, May 07, 2016 at 09:59:00AM +0200, Christian Kivalo wrote:

> >Looking at those emails, I see that calendar-notificat...@google.com is
> >the source address. I added the entry to my check_sender_access table
> >with an action of 'FILTER strip-automated-headers:'
> 
> You could probably get it working with a restriction class.
> 
> check_sender_access hash:/path/to/sender_access
> 
> smtpd_restriction_classes: strip_automated_headers
> 
> strip_automated_headers =
>   header_checks = pcre:/path/to/strip_automated_headers.pcre


No.  This can't work.

-- 
Viktor.

Re: Goal: Setup transport that runs a header check to strip out specific header for emails from specific source

[Christian Kivalo]

Since Postfix (2.11) is my MTA and it supports removing headers, I'd
like to have it strip this one. I found that I can include this line in
the same file I use for the other header checks applied to ALL mail:

/^Auto-Submitted:/IGNORE

but then it will affect all mail and I'd rather not do that.

What I'm currently trying to get working is a service entry in 
master.cf

that has its own header checks conf file. This service would only be
used for specific sender addresses in order to limit the header removal
to just those email notifications generated by Google Calendar.

Looking at those emails, I see that calendar-notificat...@google.com is
the source address. I added the entry to my check_sender_access table
with an action of 'FILTER strip-automated-headers:'


You could probably get it working with a restriction class.

check_sender_access hash:/path/to/sender_access


smtpd_restriction_classes: strip_automated_headers

strip_automated_headers =
  header_checks = pcre:/path/to/strip_automated_headers.pcre



sender_access:
calendar-notificat...@google.com   strip_automated_headers


strip_automated_headers.pcre
/^Auto-Submitted:/IGNORE

Haven't come around to test it tough...


http://www.postfix.org/RESTRICTION_CLASS_README.html




Thanks in advance for your help!


--
 Christian Kivalo