Re: sender login maps and alias domains
--On November 19, 2014 at 7:40:07 PM -0500 Wietse Venema wie...@porcupine.org wrote: Viktor Dukhovni: On Wed, Nov 19, 2014 at 03:07:19PM -0800, Quanah Gibson-Mount wrote: Sorry, the login table lookup must handle the original address as the lookup key via a single query. And clearly postfix is already designed in some cases to do multiple queries based on the results of previous queries. This is not a productive direction to pursue. The feature you're looking for is not there, and is not planned. You need to find a different solution. If Quanah is asking for the sender login map to automagically expand Postfix virtual alias maps, then don't count on that to ever happen. Not really, no. My point is there are three queries necessary to deliver this email. Postfix does two of them, and gets all of the data it needs via those two queries to do the third and final query, and thus deliver the email. I understand it is the prerogative of the postfix team to not further expand the queries, so that's that. If there was a way to use an lmdb table to rewrite the request for the domain prior to the LDAP lookup, that'd be another way to solve the issue. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
Quanah Gibson-Mount:
This is not a productive direction to pursue. The feature you're
looking for is not there, and is not planned. You need to find
a different solution.
If Quanah is asking for the sender login map to automagically
expand Postfix virtual alias maps, then don't count on that to
ever happen.
Not really, no.
Good. That would be a problem.
My point is there are three queries necessary to deliver this email.
Postfix does two of them, and gets all of the data it needs via those two
queries to do the third and final query, and thus deliver the email. I
understand it is the prerogative of the postfix team to not further expand
the queries, so that's that. If there was a way to use an lmdb table to
rewrite the request for the domain prior to the LDAP lookup, that'd be
another way to solve the issue.
Postfix 2.12 allows one lookup table to produce a query for another
lookup table. For example:
smtpd_sender_login_maps = pipemap:{lmdb:map1, ldap:map2}
This will transform the sender with the LMDB table, before searching
the LDAP table. Note: when the LMDB table returns not found then
there will be no LDAP query.
I expect that pipemap will be the start of a family of simple
database primitives that canbe combined to implement powerful
queries.
Wietse
Re: sender login maps and alias domains
--On November 21, 2014 at 2:30:18 PM -0500 Wietse Venema
wie...@porcupine.org wrote:
Postfix 2.12 allows one lookup table to produce a query for another
lookup table. For example:
smtpd_sender_login_maps = pipemap:{lmdb:map1, ldap:map2}
This will transform the sender with the LMDB table, before searching
the LDAP table. Note: when the LMDB table returns not found then
there will be no LDAP query.
I expect that pipemap will be the start of a family of simple
database primitives that canbe combined to implement powerful
queries.
That sounds like an awesome feature. So theoretically then, I'd assume I
could then just use two ldap maps... One to do the domain rewrite, the
second to look up where to deliver.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
li...@rhsoft.net wrote: but LDAP is a very limited database missing things like joins, views and a lot of other things you can easily do with a SQL query While you're right you don't recommend using joins for productive use, do you? - don't blame the messenger and just setup a wrapper fetching the data from LDAP and present them in a SQL database for postfix Hmm, wouldn't it be better to sync the data with a local DB then? In this case you could transform the data in any lookup form needed which should get much better performance. did something similar recently for a inbound mail-gateway by fetching data from a infinite count of final destination servers each with it's own mysql configuration and present them on the gateway server including the transport table with the target machines that's finally a cronjob running every 5 minutes - such data don't change every second Hmm, running a CRON job every 5 minutes which does almost nothing is a waste of resources when data sets are big. In case of OpenLDAP as data source one could write a syncrepl client. Food for thought: Following DN reference attributes with additional LDAP lookups is not very performant either. With OpenLDAP's slapo-deref postfix could use the deref extended control [1] to fetch referenced entries with one query. Ciao, Michael. [1] http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob_plain;f=doc/drafts/draft-masarati-ldap-deref-xx.txt;hb=HEAD smime.p7s Description: S/MIME Cryptographic Signature
Re: sender login maps and alias domains
Am 20.11.2014 um 09:10 schrieb Michael Ströder:
li...@rhsoft.net wrote:
but LDAP is a very limited database missing things like joins, views and a lot
of other things you can easily do with a SQL query
While you're right you don't recommend using joins for productive use, do you?
surely - if i work only with a flat table i don't need SQL at all
CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY
DEFINER VIEW `dbma_recipients` AS select `dbmail_aliases`.`alias` AS
`alias` from `dbmail_aliases` where (not((`dbmail_aliases`.`alias` like
'%@'))) union (select concat('@',`dbma_mta`.`mydestination`) AS `alias`
from `dbma_mta` where (`dbma_mta`.`transport`
'dbmail-lmtp:127.0.0.1:24')) union (select
`dbma_recipient_relay`.`mydestination` AS `alias` from
`dbma_recipient_relay`) union (select `dbma_aliases_global`.`alias` AS
`alias` from `dbma_aliases_global`)
- don't blame the messenger
and just setup a wrapper fetching the data from LDAP and present them in a SQL
database for postfix
Hmm, wouldn't it be better to sync the data with a local DB then?
In this case you could transform the data in any lookup form needed which
should get much better performance.
depends what you want to achieve
our inbound MX has hash-tables re-created on demand every 5 minutes from
the sql databases, the backend servers just use happily views, joins and
a lot of SQL
so the views above are only part of the game for the 5% messages making
it through the first server
did something similar recently for a inbound mail-gateway by fetching data
from a infinite count of final destination servers each with it's own mysql
configuration and present them on the gateway server including the transport
table with the target machines
that's finally a cronjob running every 5 minutes - such data don't change
every second
Hmm, running a CRON job every 5 minutes which does almost nothing is a waste
of resources when data sets are big.
on a machine running SpamAssassin and ClamAV that waste of ressources
is meaningless
In case of OpenLDAP as data source one could write a syncrepl client.
Food for thought:
Following DN reference attributes with additional LDAP lookups is not very
performant either. With OpenLDAP's slapo-deref postfix could use the deref
extended control [1] to fetch referenced entries with one query
Re: sender login maps and alias domains
On Wed, Nov 19, 2014 at 01:31:59PM -0800, Quanah Gibson-Mount wrote: server_host = ldap://zre-ldap003.eng.zimbra.com:389 server_port = 389 search_base = query_filter = ((|(uid=%s)(zimbraAllowFromAddress=%s)(zimbraMailDeliveryAddress=%s)(zimbraMailAlias=%s)(zimbraMailCatchAllAddress=%s))(zimbraMailStatus=enabled)) result_format = %u, %s result_attribute = uid,zimbraMailDeliveryAddress,zimbraMailForwardingAddress,zimbraPrefMailForwardingAddress,zimbraMailCatchAllForwardingAddress,zimbraMailAlias,zimbraAllowFromAddress version = 3 start_tls = yes tls_ca_cert_dir = /opt/zimbra/conf/ca bind = yes bind_dn = uid=zmpostfix,cn=appaccts,cn=zimbra bind_pw = B7fPB4Deo timeout = 30 So in this instance, I have a user (testuser1) in the domain zre-ldap003.eng.zimbra.com. I've created an alias domain of zre-ldap002.eng.zimbra.com. I'm unable to send out email using testus...@zre-ldap002.eng.zimbra.com as the MAIL FROM: address for this specific case, even if I (successfully) log in using that username. For each user list all their mail addresses as additional values of a suitable multi-valued attribute that holds the users various valid email addresses. For example, Microsoft Exchange stores: proxyAddresses = smtp:address1 proxyAddresses = smtp:address2 ... proxyAddresses = smtp:addressN Is there some way to tell postfix to look for the user under the actual domain? The actual domain is the domain in the user address on the wire, sorry no wildcards. Wildcards should be avoided, but soap-box aside, they are not supported in this context. -- Viktor.
Re: sender login maps and alias domains
--On November 19, 2014 at 9:39:26 PM + Viktor Dukhovni postfix-us...@dukhovni.org wrote: For each user list all their mail addresses as additional values of a suitable multi-valued attribute that holds the users various valid email addresses. For example, Microsoft Exchange stores: For sites with 10 million + users (or even 1 million +), I don't think they'll want to have to go through and list those for every user (we do this for account aliases within a domain currently, but they have to explicitly set those for the user), and then have to do this every time they add new accounts as well. In addition, some sites have multiple alias domains. I have a test dataset from a real customer that is a hosting provider, where many of their customers own 20-30 domains, so there is 1 primary domain, and 19-29 alias domains. Is there some way to tell postfix to look for the user under the actual domain? The actual domain is the domain in the user address on the wire, sorry no wildcards. Wildcards should be avoided, but soap-box aside, they are not supported in this context. I was thinking more of an additional lookup, given that the real domain is returned back as part of one of the queries. I.e., we have the real domain from: zimbraMailCatchAllForwardingAddress: @zre-ldap003.eng.zimbra.com so re-run this search: Nov 19 15:24:53 zre-ldap003 slapd[23266]: conn=7277 op=9 SRCH base= scope=2 deref=0 filter=((|(uid=testus...@zre-ldap002.eng.zimbra.com)(zimbraAllowFromAddress=testus...@zre-ldap002.eng.zimbra.com)(zimbraMailDeliveryAddress=testus...@zre-ldap002.eng.zimbra.com)(zimbraMailAlias=testus...@zre-ldap002.eng.zimbra.com)(zimbraMailCatchAllAddress=testus...@zre-ldap002.eng.zimbra.com))(zimbraMailStatus=enabled)) Nov 19 15:24:53 zre-ldap003 slapd[23266]: conn=7277 op=9 SRCH attr=uid zimbraMailDeliveryAddress zimbraMailForwardingAddress zimbraPrefMailForwardingAddress zimbraMailCatchAllForwardingAddress zimbraMailAlias zimbraAllowFromAddress using the domain returned above, which will have a successful result, and allow things to proceed. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
Quanah Gibson-Mount: When implementing sender login maps, we've run into an issue where people cannot send out email using a from address of the alias domain. We store Postfix alias expansion happens after the sender authorization check. At the risk of asking an obvious question: have you tried adding the authorized email addresses in sender login maps? Have you tried automating the account management procedure so that it does that for you? Alias expansions are one-to-many, and alias expansions are recursive. Suppose that foo is aliased to bar and baz. When a client sends mail from foo, the Postfix SMTP server does not know what addresses foo is aliased to. And even if it did, what address should it look up in the sender login map? foo? bar? baz? All three of them? And what if the client sends mail from bar? Wietse
Re: sender login maps and alias domains
Am 19.11.2014 um 22:57 schrieb Quanah Gibson-Mount: --On November 19, 2014 at 9:39:26 PM + Viktor Dukhovni postfix-us...@dukhovni.org wrote: For each user list all their mail addresses as additional values of a suitable multi-valued attribute that holds the users various valid email addresses. For example, Microsoft Exchange stores: For sites with 10 million + users (or even 1 million +), I don't think they'll want to have to go through and list those for every user (we do this for account aliases within a domain currently, but they have to explicitly set those for the user), and then have to do this every time they add new accounts as well that' a weak excuse in case of such sites you have *databases* used directly for MTA queries or generate config files from databases i personally hate this weak excuses because every day the are absued millions of times where some idiot provider allows outgoing mail from hacked accounts with one of our domains never have to go out from there at all
Re: sender login maps and alias domains
On Wed, Nov 19, 2014 at 01:57:48PM -0800, Quanah Gibson-Mount wrote: For each user list all their mail addresses as additional values of a suitable multi-valued attribute that holds the users various valid email addresses. For example, Microsoft Exchange stores: For sites with 10 million + users (or even 1 million +), I don't think they'll want to have to go through and list those for every user (we do this for account aliases within a domain currently, but they have to explicitly set those for the user), and then have to do this every time they add new accounts as well. Avoiding wildcards gives you the opportunity to not burden every user with legacy domain names just because some users need them. The update in question as part of rolling out a new domain for 10 million users is not a major burden. One might also suggest that SQL is better suited to such situations than LDAP, because of the richer query language that can do joins. In addition, some sites have multiple alias domains. I have a test dataset from a real customer that is a hosting provider, where many of their customers own 20-30 domains, so there is 1 primary domain, and 19-29 alias domains. That's fine, just list 20 addresses for some of the users in question. Worked fine for ~80,000 users at previous employer. Sorry, the login table lookup must handle the original address as the lookup key via a single query. -- Viktor.
Re: sender login maps and alias domains
--On November 19, 2014 at 11:02:05 PM +0100 li...@rhsoft.net wrote: in case of such sites you have *databases* used directly for MTA queries or generate config files from databases LDAP is a database. It already contains all of the necessary data to get the work done. What's missing is the postfix piece to utilize LDAP. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
--On November 19, 2014 at 10:04:07 PM + Viktor Dukhovni postfix-us...@dukhovni.org wrote: Avoiding wildcards gives you the opportunity to not burden every user with legacy domain names just because some users need them. Who said anything about legacy domain names? Sorry, the login table lookup must handle the original address as the lookup key via a single query. And clearly postfix is already designed in some cases to do multiple queries based on the results of previous queries. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
--On November 19, 2014 at 4:57:50 PM -0500 Wietse Venema wie...@porcupine.org wrote: Alias expansions are one-to-many, and alias expansions are recursive. Suppose that foo is aliased to bar and baz. When a client sends mail from foo, the Postfix SMTP server does not know what addresses foo is aliased to. And even if it did, what address should it look up in the sender login map? foo? bar? baz? All three of them? And what if the client sends mail from bar? They're required to use user@domain for the login. There won't be an issue of foo vs bar vs baz. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
On Wed, Nov 19, 2014 at 03:06:07PM -0800, Quanah Gibson-Mount wrote: in case of such sites you have *databases* used directly for MTA queries or generate config files from databases LDAP is a database. It already contains all of the necessary data to get the work done. What's missing is the postfix piece to utilize LDAP. LDAP is a denormalized star schema. It does not support fancy multi table queries, and clients should not need to compensate for this. To handle objects with multiple lookup keys in LDAP you either embed all these in a single multi-valued attribute in the target entry, or use LDAP member DNs to link an LDAP alias entry to the target entry. Postfix has no LDAP-specific schemas for implementing lookup features tables in an LDAP-specific way. LDAP is used as a key-value store. The key is the envelope sender address, the result is a list of SASL logins. You need to implement the appropriate key-value mapping by some suitable means. This lookup is not recursive. -- Viktor.
Re: sender login maps and alias domains
On Wed, Nov 19, 2014 at 03:07:19PM -0800, Quanah Gibson-Mount wrote: Sorry, the login table lookup must handle the original address as the lookup key via a single query. And clearly postfix is already designed in some cases to do multiple queries based on the results of previous queries. This is not a productive direction to pursue. The feature you're looking for is not there, and is not planned. You need to find a different solution. -- Viktor.
Re: sender login maps and alias domains
--On November 19, 2014 at 11:14:49 PM + Viktor Dukhovni postfix-us...@dukhovni.org wrote: On Wed, Nov 19, 2014 at 03:07:19PM -0800, Quanah Gibson-Mount wrote: Sorry, the login table lookup must handle the original address as the lookup key via a single query. And clearly postfix is already designed in some cases to do multiple queries based on the results of previous queries. This is not a productive direction to pursue. The feature you're looking for is not there, and is not planned. You need to find a different solution. I've already replied to the bug filed by our customers informing them they will need to create account aliases if they wish to use the feature, as postfix will not support it. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc Zimbra :: the leader in open source messaging and collaboration
Re: sender login maps and alias domains
Viktor Dukhovni: On Wed, Nov 19, 2014 at 03:07:19PM -0800, Quanah Gibson-Mount wrote: Sorry, the login table lookup must handle the original address as the lookup key via a single query. And clearly postfix is already designed in some cases to do multiple queries based on the results of previous queries. This is not a productive direction to pursue. The feature you're looking for is not there, and is not planned. You need to find a different solution. If Quanah is asking for the sender login map to automagically expand Postfix virtual alias maps, then don't count on that to ever happen. Wietse
Re: sender login maps and alias domains
Am 20.11.2014 um 00:06 schrieb Quanah Gibson-Mount: --On November 19, 2014 at 11:02:05 PM +0100 li...@rhsoft.net wrote: in case of such sites you have *databases* used directly for MTA queries or generate config files from databases LDAP is a database. It already contains all of the necessary data to get the work done. What's missing is the postfix piece to utilize LDAP. but LDAP is a very limited database missing things like joins, views and a lot of other things you can easily do with a SQL query - don't blame the messenger and just setup a wrapper fetching the data from LDAP and present them in a SQL database for postfix you need to realize that postfix works agnostic of the backend, hence what you do in case of SQL is fetch the needed data and present it similar to a hash lookable table - LDAP can't do this and that's not postfix's fault did something similar recently for a inbound mail-gateway by fetching data from a infinite count of final destination servers each with it's own mysql configuration and present them on the gateway server including the transport table with the target machines that's finally a cronjob running every 5 minutes - such data don't change every second
