Re: Duplicate email issue with opendkim milter

[Viktor Dukhovni]

On Fri, Jan 08, 2016 at 03:08:33PM -0800, Quanah Gibson-Mount wrote:

> >I usually solve this differently.  I avoid self-referential
> >expansions!
> >
> >virtual:
> > use...@example.com  
> > use...@mailstore-name.example.com,
> > use...@mailstore-name.example.com,
> 
> Thanks, that makes a lot of sense.  We have a concept of an internal home
> mailbox server, so if we modified things to use it inside our ldap map for
> virtual_alias_maps, I think it'd resolve the problem

That's the idea anyway.

> However, I think there is an underlying flaw in our virtual_alias_maps ldap
> mapping:
> 
> query_filter =
>   (&(|(zimbraMailDeliveryAddress=%s)
>   (zimbraMailAlias=%s)
>   (zimbraMailCatchAllAddress=%s))
> (zimbraMailStatus=enabled))
> result_attribute = zimbraMailDeliveryAddress,
>zimbraMailForwardingAddress,
>zimbraPrefMailForwardingAddress,
>zimbraMailCatchAllForwardingAddress
> 
> Because we include zimbraMailDeliveryAddress as both a search key and a
> result, it will continually get expanded.  It'd make more sense then, I
> believe to be something like:

Indeed, in my implementations, the delivery domains are never
virtual, and the ldap tables are configured to not query said
domains:

# Virtual domains for ldap lookup
domain = example.com, ...

with the input mail addresses in the virtual domains subject to
virtual alias expansion, but the (leaf) outputs generally not
landing in those domains.  Only when one virtual address is an
alias for another (rather delivers for itself to a mailbox domain)
are the lookups recursive.

This also means that I avoid returning the query attribute, the
multi-valued forwarding address attribute also returns the input
address, when the forward is a Cc, rather than a "redirect".

The simplest version of this is:

domain = example.com
query = (mail=%s)
resultAttribute = maildrop

# Example LDIF:
mail: ju...@example.com
maildrop: ju...@imap.example.com

Everything else is just elaboration on the theme, with
special_result_attribute indirection, leaf and terminal attributes,
...  I used the lot at Morgan Stanley, where I developed those
features because I needed them.

-- 
Viktor.

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Tuesday, December 01, 2015 12:34 AM + Viktor Dukhovni 
 wrote:



I usually solve this differently.  I avoid self-referential
expansions!

virtual:
use...@example.com  
use...@mailstore-name.example.com,
use...@mailstore-name.example.com,


Hi Viktor,

Thanks, that makes a lot of sense.  We have a concept of an internal home 
mailbox server, so if we modified things to use it inside our ldap map for 
virtual_alias_maps, I think it'd resolve the problem


However, I think there is an underlying flaw in our virtual_alias_maps ldap 
mapping:


query_filter = 
(&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled))
result_attribute = 
zimbraMailDeliveryAddress,zimbraMailForwardingAddress,zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress


Because we include zimbraMailDeliveryAddress as both a search key and a 
result, it will continually get expanded.  It'd make more sense then, I 
believe to be something like:


query_filter = 
(&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled))
result_attribute = 
zimbraMailHomeDeliveryAddress,zimbraMailForwardingAddress,zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress


where zimbraMailHomeDeliveryAddress would be: user@mailbox.domain
and zimbraMailDeliveryAddress is: user@domain

The forwarding is in zimbraMailForwardingAddress

This would block multiple expansions.  Does that seem correct?

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Tuesday, November 24, 2015 12:42 AM + Viktor Dukhovni 
 wrote:




> Since this was implemented, we've had an issue where when emails with
> a  large number of recipients are processed, the result is that the
> recipients  get duplicates of the email.  We found one workaround to
> this was to  default_destination_recipient_limit to large value.

Why did that make a difference?


Well, of course I would change only "smtp-amavis_recipient_limit",
but the difference is that otherwise the default limit delivers at
most 50 recipients at a time to the content filter.  This reduces
opportunities for duplicate elimination (across lists), especially
with the default "enable_original_recipient = yes".

I've always (since ~2001) used large recipient limits with filter
transports, this also improves efficiency, no need to scan the same
content multiple times.


Hi Viktor,

Thanks for the reply!  I've been on vacation so catching up on email.  It 
sounds like the better solution then is to add:


-o default_destination_recipient_limit=5000

to the content filter definitions? ;)

--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount 
 wrote:



I've always (since ~2001) used large recipient limits with filter
transports, this also improves efficiency, no need to scan the same
content multiple times.


Hi Viktor,

Thanks for the reply!  I've been on vacation so catching up on email.  It
sounds like the better solution then is to add:

-o default_destination_recipient_limit=5000

to the content filter definitions? ;)


I've tried variations on this with no success.  My master.cf eventually 
being the following, but still getting duplicates.  I added "-o 
receive_override_options=no_address_mappings" to the [127.0.0.1]:10030 
block, and after that, no more duplicates.  However, the original 
suggestion from Viktor still seems cleaner, although I'm not seeing it work 
so far. :/


smtp  inet  n   -   n   -   1   postscreen
tlsproxy  unix  -   -   n   -   0   tlsproxy
dnsblog   unix  -   -   n   -   0   dnsblog
smtpd pass  -   -   n   -   -   smtpd
   -o smtpd_tls_security_level=may
   -o content_filter=scan:[127.0.0.1]:10030
   -o default_destination_recipient_limit=100
465inet  n   -   n   -   -   smtpd
   -o content_filter=scan:[127.0.0.1]:10030
   -o default_destination_recipient_limit=100
   -o smtpd_tls_wrappermode=yes
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=
   -o smtpd_data_restrictions=
   -o smtpd_helo_restrictions=
   -o smtpd_recipient_restrictions=
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
   -o syslog_name=postfix/smtps
   -o milter_macro_daemon_name=ORIGINATING
submission inet n  -   n   -   -   smtpd
   -o content_filter=scan:[127.0.0.1]:10030
   -o default_destination_recipient_limit=100
   -o smtpd_etrn_restrictions=reject
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_tls_security_level=may
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o smtpd_data_restrictions=
   -o smtpd_helo_restrictions=
   -o smtpd_recipient_restrictions=
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
   -o syslog_name=postfix/submission
   -o milter_macro_daemon_name=ORIGINATING
scan  unix  -   -   n   -   10  smtp
   -o smtp_send_xforward_command=yes
   -o disable_mime_output_conversion=yes
   -o smtp_generic_maps=
pickupunix  n   -   n   60  1   pickup
cleanup   unix  n   -   n   -   0   cleanup
qmgr  unix  n   -   n   300 1   qmgr
tlsmgrunix  -   -   n   1000?   1   tlsmgr
rewrite   unix  -   -   n   -   -   trivial-rewrite
bounceunix  -   -   n   -   0   bounce
defer unix  -   -   n   -   0   bounce
trace unix  -   -   n   -   0   bounce
verifyunix  -   -   n   -   1   verify
flush unix  n   -   n   1000?   0   flush
proxymap  unix  -   -   n   -   -   proxymap
smtp  unix  -   -   n   -   -   smtp
relay unix  -   -   n   -   -   smtp
showq unix  n   -   n   -   -   showq
error unix  -   -   n   -   -   error
retry unix  -   -   n   -   -   error
discard   unix  -   -   n   -   -   discard
local unix  -   n   n   -   -   local
virtual   unix  -   n   n   -   -   virtual
lmtp  unix  -   -   n   -   -   lmtp
anvil unix  -   -   n   -   1   anvil
scacheunix  -   -   n   -   1   scache
maildrop  unix  -   n   n   -   -   pipe
 flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
old-cyrus unix  -   n   n   -   -   pipe
 flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
cyrus unix  -   n   n   -   -   pipe
 user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
uucp  unix  -   n   n   -   -   pipe
 flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail 
($recipient)

ifmailunix  -   n   n   -   -   pipe
 flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix  -   n   n   -   -   pipe
 flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop 
$recipient

smtp-amavis unix -  -   n   -   10  smtp
   -o smtp_data_done_timeout=1200
   -o smtp_send_xforward_command=yes
   -o disable_dns_lookups=yes
   -o default_destination_recipient_limit=100
   -o max_use=20
[127.0.0.1]:10025 inet n  -   n   -   -  smtpd
   -o content_filter=
   -o default_destination_recipient_limit=100
   -o 

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Monday, November 30, 2015 3:01 PM -0800 Quanah Gibson-Mount 
 wrote:



--On Monday, November 30, 2015 2:56 PM -0800 Quanah Gibson-Mount
 wrote:


--On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount
 wrote:


I've always (since ~2001) used large recipient limits with filter
transports, this also improves efficiency, no need to scan the same
content multiple times.


Hi Viktor,

Thanks for the reply!  I've been on vacation so catching up on email.
It sounds like the better solution then is to add:

-o default_destination_recipient_limit=5000

to the content filter definitions? ;)


I've tried variations on this with no success.  My master.cf eventually
being the following, but still getting duplicates.  I added "-o
receive_override_options=no_address_mappings" to the [127.0.0.1]:10030
block, and after that, no more duplicates.  However, the original
suggestion from Viktor still seems cleaner, although I'm not seeing it
work so far. :/


I also set:

smtp-amavis_destination_recipient_limit = 100

via postconf, with no change in behavior.

Either default_destination_recipient_limit has to be bumped up via
postconf, or -o receive_override_options=no_address_mappings requires
setting to resolve this so far.


and for completion, setting:

smtp-amavis_recipient_limit = 100

had no change in behavior either. ;)

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Viktor Dukhovni]

On Mon, Nov 30, 2015 at 07:00:08AM -0800, Quanah Gibson-Mount wrote:

> >I've always (since ~2001) used large recipient limits with filter
> >transports, this also improves efficiency, no need to scan the same
> >content multiple times.
> 
> Hi Viktor,
> 
> Thanks for the reply!  I've been on vacation so catching up on email.  It
> sounds like the better solution then is to add:
> 
> -o default_destination_recipient_limit=5000
> 
> to the content filter definitions? ;)

That won't work.  The recipient limits are used in queue manager,
not the transport.

_destination_recipient_limit

And this would be an unwise global default, and the number is much
too high.  You get dimishing returns especially beyond whatever
you've set for the virtual expansion limits.

-- 
Viktor.

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Tuesday, December 01, 2015 12:03 AM + Viktor Dukhovni 
 wrote:



On Mon, Nov 30, 2015 at 03:11:39PM -0800, Quanah Gibson-Mount wrote:


--On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount
 wrote:

>> Either default_destination_recipient_limit has to be bumped up via
>> postconf, or -o receive_override_options=no_address_mappings requires
>> setting to resolve this so far.

Hm, so according to our clients setting
default_destination_recipient_limit to a size larger than the list size
fixed the issue for them.  However, I just tested that as well, and it
had no effect either.  So really, the only way that I stop seeing
duplicates is to set -o
receive_override_options=no_address_mappings


The main effect of higher recipient limits is to reduce the CPU
cost of processing multiple copies of the same message.  Any effect
on duplicate suppression is largely coincidental.

Duplicates arise when multiple lists have common recipients, and
whether these lead to multiple deliveries or not depends mostly on
enable_original_recipient.


That's not really what we're seeing.  What we are seeing is:

A list has 51 or more members (users1 through users55)

One of those list members sets their email to forward somewhere else (local 
or remote, doesn't matter) (say, user50 forwards to user70).


The user who has a forward set will receive two copies of the email at 
their forward location unless no_address_mappings is specified.


--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Viktor Dukhovni]

On Mon, Nov 30, 2015 at 04:11:15PM -0800, Quanah Gibson-Mount wrote:

> >Duplicates arise when multiple lists have common recipients, and
> >whether these lead to multiple deliveries or not depends mostly on
> >enable_original_recipient.
> 
> That's not really what we're seeing.  What we are seeing is:
> 
> A list has 51 or more members (users1 through users55)
> 
> One of those list members sets their email to forward somewhere else (local
> or remote, doesn't matter) (say, user50 forwards to user70).

That is two lists with common members, one list is user50 (expands
to user70), the other list is user70 (no expansion).  Your real
problem however is the self-referencing user50:

user50: user70, user50

If you expand this twice, you get:

user50 -> user70, user50 > user70, user50
\
 > user70

Hence the duplication.  At the second hop, the two expansions have
different original recipient values.  This is why FILTER_README
describes doing virtual expansion either before or after the filter,
but not both.

I usually solve this differently.  I avoid self-referential
expansions!

virtual:
use...@example.com  
use...@mailstore-name.example.com,
use...@mailstore-name.example.com,

That is, I ensured that the input keys for virtual aliases are
*always* in virtual alias domains (this is not required, but a good
idea).  And the outputs are *never* in virtual alias domains.  I
used LDAP with a "domain = example.com" setting to not even bother
to do lookups on the output domains (big latency saving since
virtual alias expansion is recursive).

I had an environment with multiple mailstores under a single virtual
domain.  Multiple Microsoft Exchange deployments, some IMAP
mailstores, and some cloud mailstores.  All expansions were:

mail -> maildrop

not

mail -> mail

The domainpart of "mail" was always a virtual alias domain.  The
domainpart of "maildrop" is always the routing information to the
user's mailstore.  The transport table was per maildrop domain,
not per user.

-- 
Viktor.

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount 
 wrote:



Either default_destination_recipient_limit has to be bumped up via
postconf, or -o receive_override_options=no_address_mappings requires
setting to resolve this so far.


Hm, so according to our clients setting default_destination_recipient_limit 
to a size larger than the list size fixed the issue for them.  However, I 
just tested that as well, and it had no effect either.  So really, the only 
way that I stop seeing duplicates is to set -o 
receive_override_options=no_address_mappings


--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

--On Monday, November 30, 2015 2:56 PM -0800 Quanah Gibson-Mount 
 wrote:



--On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount
 wrote:


I've always (since ~2001) used large recipient limits with filter
transports, this also improves efficiency, no need to scan the same
content multiple times.


Hi Viktor,

Thanks for the reply!  I've been on vacation so catching up on email.  It
sounds like the better solution then is to add:

-o default_destination_recipient_limit=5000

to the content filter definitions? ;)


I've tried variations on this with no success.  My master.cf eventually
being the following, but still getting duplicates.  I added "-o
receive_override_options=no_address_mappings" to the [127.0.0.1]:10030
block, and after that, no more duplicates.  However, the original
suggestion from Viktor still seems cleaner, although I'm not seeing it
work so far. :/


I also set:

smtp-amavis_destination_recipient_limit = 100

via postconf, with no change in behavior.

Either default_destination_recipient_limit has to be bumped up via 
postconf, or -o receive_override_options=no_address_mappings requires 
setting to resolve this so far.


--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Viktor Dukhovni]

On Mon, Nov 30, 2015 at 03:11:39PM -0800, Quanah Gibson-Mount wrote:

> --On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount
>  wrote:
> 
> >>Either default_destination_recipient_limit has to be bumped up via
> >>postconf, or -o receive_override_options=no_address_mappings requires
> >>setting to resolve this so far.
> 
> Hm, so according to our clients setting default_destination_recipient_limit
> to a size larger than the list size fixed the issue for them.  However, I
> just tested that as well, and it had no effect either.  So really, the only
> way that I stop seeing duplicates is to set -o
> receive_override_options=no_address_mappings

The main effect of higher recipient limits is to reduce the CPU
cost of processing multiple copies of the same message.  Any effect
on duplicate suppression is largely coincidental.

Duplicates arise when multiple lists have common recipients, and
whether these lead to multiple deliveries or not depends mostly on
enable_original_recipient.

-- 
Viktor.

Duplicate email issue with opendkim milter

[Quanah Gibson-Mount]

I have the following configuration for passing email to OpenDKIM for 
processing:


[127.0.0.1]:10030 inet n - n - - smtpd
   -o local_recipient_maps=
   -o virtual_mailbox_maps=
   -o virtual_alias_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_delay_reject=no
   -o smtpd_milters=inet:localhost:8465
   -o smtpd_client_restrictions=permit_mynetworks,reject
   -o smtpd_sender_restrictions=
   -o smtpd_helo_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o smtpd_reject_unlisted_sender=no
   -o smtpd_relay_restrictions=
   -o smtpd_data_restrictions=
   -o smtpd_end_of_data_restrictions=
   -o syslog_name=postfix/dkimmilter
   -o content_filter=smtp-amavis:[127.0.0.1]:10032


Since this was implemented, we've had an issue where when emails with a 
large number of recipients are processed, the result is that the recipients 
get duplicates of the email.  We found one workaround to this was to 
default_destination_recipient_limit to large value.


Recently, one of our clients found that setting:

-o 
receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_address_mappings


Also resolved the problem.  I've been reading over the docs, and it seems 
that this is a bit aggressive.  However, it does seem that at least some 
set of these options should be set.  I believe it is only necessary to set 
"no_address_mappings".  Does that seem correct?


Thanks,
Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.

Zimbra ::  the leader in open source messaging and collaboration

Re: Duplicate email issue with opendkim milter

[Wietse Venema]

Quanah Gibson-Mount:
> I have the following configuration for passing email to OpenDKIM for 
> processing:
> 
> [127.0.0.1]:10030 inet n - n - - smtpd
> -o local_recipient_maps=
> -o virtual_mailbox_maps=
> -o virtual_alias_maps=
> -o relay_recipient_maps=
> -o smtpd_restriction_classes=
> -o smtpd_delay_reject=no
> -o smtpd_milters=inet:localhost:8465
> -o smtpd_client_restrictions=permit_mynetworks,reject
> -o smtpd_sender_restrictions=
> -o smtpd_helo_restrictions=
> -o smtpd_recipient_restrictions=permit_mynetworks,reject
> -o smtpd_reject_unlisted_sender=no
> -o smtpd_relay_restrictions=
> -o smtpd_data_restrictions=
> -o smtpd_end_of_data_restrictions=
> -o syslog_name=postfix/dkimmilter
> -o content_filter=smtp-amavis:[127.0.0.1]:10032
> 
> Since this was implemented, we've had an issue where when emails with a 
> large number of recipients are processed, the result is that the recipients 
> get duplicates of the email.  We found one workaround to this was to 
> default_destination_recipient_limit to large value.

Why did that make a difference?

Wietse

Re: Duplicate email issue with opendkim milter

[Viktor Dukhovni]

On Mon, Nov 23, 2015 at 07:29:18PM -0500, Wietse Venema wrote:

> > I have the following configuration for passing email to OpenDKIM for 
> > processing:
> > 
> > [127.0.0.1]:10030 inet n - n - - smtpd
> > -o local_recipient_maps=
> > -o virtual_mailbox_maps=
> > -o virtual_alias_maps=
> > -o relay_recipient_maps=
> > -o smtpd_restriction_classes=
> > -o smtpd_delay_reject=no
> > -o smtpd_milters=inet:localhost:8465
> > -o smtpd_client_restrictions=permit_mynetworks,reject
> > -o smtpd_sender_restrictions=
> > -o smtpd_helo_restrictions=
> > -o smtpd_recipient_restrictions=permit_mynetworks,reject
> > -o smtpd_reject_unlisted_sender=no
> > -o smtpd_relay_restrictions=
> > -o smtpd_data_restrictions=
> > -o smtpd_end_of_data_restrictions=
> > -o syslog_name=postfix/dkimmilter
> > -o content_filter=smtp-amavis:[127.0.0.1]:10032
> > 
> > Since this was implemented, we've had an issue where when emails with a 
> > large number of recipients are processed, the result is that the recipients 
> > get duplicates of the email.  We found one workaround to this was to 
> > default_destination_recipient_limit to large value.
> 
> Why did that make a difference?

Well, of course I would change only "smtp-amavis_recipient_limit",
but the difference is that otherwise the default limit delivers at
most 50 recipients at a time to the content filter.  This reduces
opportunities for duplicate elimination (across lists), especially
with the default "enable_original_recipient = yes".

I've always (since ~2001) used large recipient limits with filter
transports, this also improves efficiency, no need to scan the same
content multiple times.

-- 
Viktor.