Re: Duplicate email issue with opendkim milter
On Fri, Jan 08, 2016 at 03:08:33PM -0800, Quanah Gibson-Mount wrote: > >I usually solve this differently. I avoid self-referential > >expansions! > > > >virtual: > > use...@example.com > > use...@mailstore-name.example.com, > > use...@mailstore-name.example.com, > > Thanks, that makes a lot of sense. We have a concept of an internal home > mailbox server, so if we modified things to use it inside our ldap map for > virtual_alias_maps, I think it'd resolve the problem That's the idea anyway. > However, I think there is an underlying flaw in our virtual_alias_maps ldap > mapping: > > query_filter = > (&(|(zimbraMailDeliveryAddress=%s) > (zimbraMailAlias=%s) > (zimbraMailCatchAllAddress=%s)) > (zimbraMailStatus=enabled)) > result_attribute = zimbraMailDeliveryAddress, >zimbraMailForwardingAddress, >zimbraPrefMailForwardingAddress, >zimbraMailCatchAllForwardingAddress > > Because we include zimbraMailDeliveryAddress as both a search key and a > result, it will continually get expanded. It'd make more sense then, I > believe to be something like: Indeed, in my implementations, the delivery domains are never virtual, and the ldap tables are configured to not query said domains: # Virtual domains for ldap lookup domain = example.com, ... with the input mail addresses in the virtual domains subject to virtual alias expansion, but the (leaf) outputs generally not landing in those domains. Only when one virtual address is an alias for another (rather delivers for itself to a mailbox domain) are the lookups recursive. This also means that I avoid returning the query attribute, the multi-valued forwarding address attribute also returns the input address, when the forward is a Cc, rather than a "redirect". The simplest version of this is: domain = example.com query = (mail=%s) resultAttribute = maildrop # Example LDIF: mail: ju...@example.com maildrop: ju...@imap.example.com Everything else is just elaboration on the theme, with special_result_attribute indirection, leaf and terminal attributes, ... I used the lot at Morgan Stanley, where I developed those features because I needed them. -- Viktor.
Re: Duplicate email issue with opendkim milter
--On Tuesday, December 01, 2015 12:34 AM + Viktor Dukhovni wrote: I usually solve this differently. I avoid self-referential expansions! virtual: use...@example.com use...@mailstore-name.example.com, use...@mailstore-name.example.com, Hi Viktor, Thanks, that makes a lot of sense. We have a concept of an internal home mailbox server, so if we modified things to use it inside our ldap map for virtual_alias_maps, I think it'd resolve the problem However, I think there is an underlying flaw in our virtual_alias_maps ldap mapping: query_filter = (&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled)) result_attribute = zimbraMailDeliveryAddress,zimbraMailForwardingAddress,zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress Because we include zimbraMailDeliveryAddress as both a search key and a result, it will continually get expanded. It'd make more sense then, I believe to be something like: query_filter = (&(|(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled)) result_attribute = zimbraMailHomeDeliveryAddress,zimbraMailForwardingAddress,zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress where zimbraMailHomeDeliveryAddress would be: user@mailbox.domain and zimbraMailDeliveryAddress is: user@domain The forwarding is in zimbraMailForwardingAddress This would block multiple expansions. Does that seem correct? --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
On Mon, Nov 30, 2015 at 04:11:15PM -0800, Quanah Gibson-Mount wrote: > >Duplicates arise when multiple lists have common recipients, and > >whether these lead to multiple deliveries or not depends mostly on > >enable_original_recipient. > > That's not really what we're seeing. What we are seeing is: > > A list has 51 or more members (users1 through users55) > > One of those list members sets their email to forward somewhere else (local > or remote, doesn't matter) (say, user50 forwards to user70). That is two lists with common members, one list is user50 (expands to user70), the other list is user70 (no expansion). Your real problem however is the self-referencing user50: user50: user70, user50 If you expand this twice, you get: user50 -> user70, user50 > user70, user50 \ > user70 Hence the duplication. At the second hop, the two expansions have different original recipient values. This is why FILTER_README describes doing virtual expansion either before or after the filter, but not both. I usually solve this differently. I avoid self-referential expansions! virtual: use...@example.com use...@mailstore-name.example.com, use...@mailstore-name.example.com, That is, I ensured that the input keys for virtual aliases are *always* in virtual alias domains (this is not required, but a good idea). And the outputs are *never* in virtual alias domains. I used LDAP with a "domain = example.com" setting to not even bother to do lookups on the output domains (big latency saving since virtual alias expansion is recursive). I had an environment with multiple mailstores under a single virtual domain. Multiple Microsoft Exchange deployments, some IMAP mailstores, and some cloud mailstores. All expansions were: mail -> maildrop not mail -> mail The domainpart of "mail" was always a virtual alias domain. The domainpart of "maildrop" is always the routing information to the user's mailstore. The transport table was per maildrop domain, not per user. -- Viktor.
Re: Duplicate email issue with opendkim milter
--On Tuesday, December 01, 2015 12:03 AM + Viktor Dukhovni wrote: On Mon, Nov 30, 2015 at 03:11:39PM -0800, Quanah Gibson-Mount wrote: --On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount wrote: >> Either default_destination_recipient_limit has to be bumped up via >> postconf, or -o receive_override_options=no_address_mappings requires >> setting to resolve this so far. Hm, so according to our clients setting default_destination_recipient_limit to a size larger than the list size fixed the issue for them. However, I just tested that as well, and it had no effect either. So really, the only way that I stop seeing duplicates is to set -o receive_override_options=no_address_mappings The main effect of higher recipient limits is to reduce the CPU cost of processing multiple copies of the same message. Any effect on duplicate suppression is largely coincidental. Duplicates arise when multiple lists have common recipients, and whether these lead to multiple deliveries or not depends mostly on enable_original_recipient. That's not really what we're seeing. What we are seeing is: A list has 51 or more members (users1 through users55) One of those list members sets their email to forward somewhere else (local or remote, doesn't matter) (say, user50 forwards to user70). The user who has a forward set will receive two copies of the email at their forward location unless no_address_mappings is specified. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
On Mon, Nov 30, 2015 at 03:11:39PM -0800, Quanah Gibson-Mount wrote: > --On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount > wrote: > > >>Either default_destination_recipient_limit has to be bumped up via > >>postconf, or -o receive_override_options=no_address_mappings requires > >>setting to resolve this so far. > > Hm, so according to our clients setting default_destination_recipient_limit > to a size larger than the list size fixed the issue for them. However, I > just tested that as well, and it had no effect either. So really, the only > way that I stop seeing duplicates is to set -o > receive_override_options=no_address_mappings The main effect of higher recipient limits is to reduce the CPU cost of processing multiple copies of the same message. Any effect on duplicate suppression is largely coincidental. Duplicates arise when multiple lists have common recipients, and whether these lead to multiple deliveries or not depends mostly on enable_original_recipient. -- Viktor.
Re: Duplicate email issue with opendkim milter
On Mon, Nov 30, 2015 at 07:00:08AM -0800, Quanah Gibson-Mount wrote: > >I've always (since ~2001) used large recipient limits with filter > >transports, this also improves efficiency, no need to scan the same > >content multiple times. > > Hi Viktor, > > Thanks for the reply! I've been on vacation so catching up on email. It > sounds like the better solution then is to add: > > -o default_destination_recipient_limit=5000 > > to the content filter definitions? ;) That won't work. The recipient limits are used in queue manager, not the transport. _destination_recipient_limit And this would be an unwise global default, and the number is much too high. You get dimishing returns especially beyond whatever you've set for the virtual expansion limits. -- Viktor.
Re: Duplicate email issue with opendkim milter
--On Monday, November 30, 2015 3:08 PM -0800 Quanah Gibson-Mount wrote: Either default_destination_recipient_limit has to be bumped up via postconf, or -o receive_override_options=no_address_mappings requires setting to resolve this so far. Hm, so according to our clients setting default_destination_recipient_limit to a size larger than the list size fixed the issue for them. However, I just tested that as well, and it had no effect either. So really, the only way that I stop seeing duplicates is to set -o receive_override_options=no_address_mappings --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
--On Monday, November 30, 2015 3:01 PM -0800 Quanah Gibson-Mount wrote: --On Monday, November 30, 2015 2:56 PM -0800 Quanah Gibson-Mount wrote: --On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount wrote: I've always (since ~2001) used large recipient limits with filter transports, this also improves efficiency, no need to scan the same content multiple times. Hi Viktor, Thanks for the reply! I've been on vacation so catching up on email. It sounds like the better solution then is to add: -o default_destination_recipient_limit=5000 to the content filter definitions? ;) I've tried variations on this with no success. My master.cf eventually being the following, but still getting duplicates. I added "-o receive_override_options=no_address_mappings" to the [127.0.0.1]:10030 block, and after that, no more duplicates. However, the original suggestion from Viktor still seems cleaner, although I'm not seeing it work so far. :/ I also set: smtp-amavis_destination_recipient_limit = 100 via postconf, with no change in behavior. Either default_destination_recipient_limit has to be bumped up via postconf, or -o receive_override_options=no_address_mappings requires setting to resolve this so far. and for completion, setting: smtp-amavis_recipient_limit = 100 had no change in behavior either. ;) --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
--On Monday, November 30, 2015 2:56 PM -0800 Quanah Gibson-Mount wrote: --On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount wrote: I've always (since ~2001) used large recipient limits with filter transports, this also improves efficiency, no need to scan the same content multiple times. Hi Viktor, Thanks for the reply! I've been on vacation so catching up on email. It sounds like the better solution then is to add: -o default_destination_recipient_limit=5000 to the content filter definitions? ;) I've tried variations on this with no success. My master.cf eventually being the following, but still getting duplicates. I added "-o receive_override_options=no_address_mappings" to the [127.0.0.1]:10030 block, and after that, no more duplicates. However, the original suggestion from Viktor still seems cleaner, although I'm not seeing it work so far. :/ I also set: smtp-amavis_destination_recipient_limit = 100 via postconf, with no change in behavior. Either default_destination_recipient_limit has to be bumped up via postconf, or -o receive_override_options=no_address_mappings requires setting to resolve this so far. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
--On Monday, November 30, 2015 7:00 AM -0800 Quanah Gibson-Mount wrote: I've always (since ~2001) used large recipient limits with filter transports, this also improves efficiency, no need to scan the same content multiple times. Hi Viktor, Thanks for the reply! I've been on vacation so catching up on email. It sounds like the better solution then is to add: -o default_destination_recipient_limit=5000 to the content filter definitions? ;) I've tried variations on this with no success. My master.cf eventually being the following, but still getting duplicates. I added "-o receive_override_options=no_address_mappings" to the [127.0.0.1]:10030 block, and after that, no more duplicates. However, the original suggestion from Viktor still seems cleaner, although I'm not seeing it work so far. :/ smtp inet n - n - 1 postscreen tlsproxy unix - - n - 0 tlsproxy dnsblog unix - - n - 0 dnsblog smtpd pass - - n - - smtpd -o smtpd_tls_security_level=may -o content_filter=scan:[127.0.0.1]:10030 -o default_destination_recipient_limit=100 465inet n - n - - smtpd -o content_filter=scan:[127.0.0.1]:10030 -o default_destination_recipient_limit=100 -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes -o smtpd_client_restrictions= -o smtpd_data_restrictions= -o smtpd_helo_restrictions= -o smtpd_recipient_restrictions= -o smtpd_relay_restrictions=permit_sasl_authenticated,reject -o syslog_name=postfix/smtps -o milter_macro_daemon_name=ORIGINATING submission inet n - n - - smtpd -o content_filter=scan:[127.0.0.1]:10030 -o default_destination_recipient_limit=100 -o smtpd_etrn_restrictions=reject -o smtpd_sasl_auth_enable=yes -o smtpd_tls_security_level=may -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_data_restrictions= -o smtpd_helo_restrictions= -o smtpd_recipient_restrictions= -o smtpd_relay_restrictions=permit_sasl_authenticated,reject -o syslog_name=postfix/submission -o milter_macro_daemon_name=ORIGINATING scan unix - - n - 10 smtp -o smtp_send_xforward_command=yes -o disable_mime_output_conversion=yes -o smtp_generic_maps= pickupunix n - n 60 1 pickup cleanup unix n - n - 0 cleanup qmgr unix n - n 300 1 qmgr tlsmgrunix - - n 1000? 1 tlsmgr rewrite unix - - n - - trivial-rewrite bounceunix - - n - 0 bounce defer unix - - n - 0 bounce trace unix - - n - 0 bounce verifyunix - - n - 1 verify flush unix n - n 1000? 0 flush proxymap unix - - n - - proxymap smtp unix - - n - - smtp relay unix - - n - - smtp showq unix n - n - - showq error unix - - n - - error retry unix - - n - - error discard unix - - n - - discard local unix - n n - - local virtual unix - n n - - virtual lmtp unix - - n - - lmtp anvil unix - - n - 1 anvil scacheunix - - n - 1 scache maildrop unix - n n - - pipe flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient} old-cyrus unix - n n - - pipe flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user} cyrus unix - n n - - pipe user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user} uucp unix - n n - - pipe flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient) ifmailunix - n n - - pipe flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient) bsmtp unix - n n - - pipe flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient smtp-amavis unix - - n - 10 smtp -o smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o disable_dns_lookups=yes -o default_destination_recipient_limit=100 -o max_use=20 [127.0.0.1]:10025 inet n - n - - smtpd -o content_filter= -o default_destination_recipient_limit=100 -o local_recipient_maps=
Re: Duplicate email issue with opendkim milter
--On Tuesday, November 24, 2015 12:42 AM + Viktor Dukhovni wrote: > Since this was implemented, we've had an issue where when emails with > a large number of recipients are processed, the result is that the > recipients get duplicates of the email. We found one workaround to > this was to default_destination_recipient_limit to large value. Why did that make a difference? Well, of course I would change only "smtp-amavis_recipient_limit", but the difference is that otherwise the default limit delivers at most 50 recipients at a time to the content filter. This reduces opportunities for duplicate elimination (across lists), especially with the default "enable_original_recipient = yes". I've always (since ~2001) used large recipient limits with filter transports, this also improves efficiency, no need to scan the same content multiple times. Hi Viktor, Thanks for the reply! I've been on vacation so catching up on email. It sounds like the better solution then is to add: -o default_destination_recipient_limit=5000 to the content filter definitions? ;) --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. Zimbra :: the leader in open source messaging and collaboration
Re: Duplicate email issue with opendkim milter
On Mon, Nov 23, 2015 at 07:29:18PM -0500, Wietse Venema wrote: > > I have the following configuration for passing email to OpenDKIM for > > processing: > > > > [127.0.0.1]:10030 inet n - n - - smtpd > > -o local_recipient_maps= > > -o virtual_mailbox_maps= > > -o virtual_alias_maps= > > -o relay_recipient_maps= > > -o smtpd_restriction_classes= > > -o smtpd_delay_reject=no > > -o smtpd_milters=inet:localhost:8465 > > -o smtpd_client_restrictions=permit_mynetworks,reject > > -o smtpd_sender_restrictions= > > -o smtpd_helo_restrictions= > > -o smtpd_recipient_restrictions=permit_mynetworks,reject > > -o smtpd_reject_unlisted_sender=no > > -o smtpd_relay_restrictions= > > -o smtpd_data_restrictions= > > -o smtpd_end_of_data_restrictions= > > -o syslog_name=postfix/dkimmilter > > -o content_filter=smtp-amavis:[127.0.0.1]:10032 > > > > Since this was implemented, we've had an issue where when emails with a > > large number of recipients are processed, the result is that the recipients > > get duplicates of the email. We found one workaround to this was to > > default_destination_recipient_limit to large value. > > Why did that make a difference? Well, of course I would change only "smtp-amavis_recipient_limit", but the difference is that otherwise the default limit delivers at most 50 recipients at a time to the content filter. This reduces opportunities for duplicate elimination (across lists), especially with the default "enable_original_recipient = yes". I've always (since ~2001) used large recipient limits with filter transports, this also improves efficiency, no need to scan the same content multiple times. -- Viktor.
Re: Duplicate email issue with opendkim milter
Quanah Gibson-Mount: > I have the following configuration for passing email to OpenDKIM for > processing: > > [127.0.0.1]:10030 inet n - n - - smtpd > -o local_recipient_maps= > -o virtual_mailbox_maps= > -o virtual_alias_maps= > -o relay_recipient_maps= > -o smtpd_restriction_classes= > -o smtpd_delay_reject=no > -o smtpd_milters=inet:localhost:8465 > -o smtpd_client_restrictions=permit_mynetworks,reject > -o smtpd_sender_restrictions= > -o smtpd_helo_restrictions= > -o smtpd_recipient_restrictions=permit_mynetworks,reject > -o smtpd_reject_unlisted_sender=no > -o smtpd_relay_restrictions= > -o smtpd_data_restrictions= > -o smtpd_end_of_data_restrictions= > -o syslog_name=postfix/dkimmilter > -o content_filter=smtp-amavis:[127.0.0.1]:10032 > > Since this was implemented, we've had an issue where when emails with a > large number of recipients are processed, the result is that the recipients > get duplicates of the email. We found one workaround to this was to > default_destination_recipient_limit to large value. Why did that make a difference? Wietse