Ldap queries optimization
Hello,
We are using ldap maps in a relay server. Ldap maps are for address
validation (valid users and alias) and a relocated map.
But now, we are having problems with our ldap servers. Problems are not
directly related with postfix servers, but I've been investigating if
postfix could do things better.
My config is:
virtual_alias_maps = hash:/etc/postfix/alu-aliases,
hash:/etc/postfix/dif-aliases, proxy:ldap:/etc/postfix/ldap-sysaliases.cf
relay_recipient_maps = hash:/etc/postfix/relaydomains,
hash:/etc/postfix/alu-aliases, hash:/etc/postfix/dif-aliases,
proxy:ldap:/etc/postfix/ldap-vmail.cf,
proxy:ldap:/etc/postfix/ldap-sysaliases.cf
relocated_maps = proxy:ldap:/etc/postfix/ldap-relocated.cf
proxy_read_maps = $local_recipient_maps $mydestination
$virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps
$virtual_mailbox_domains $relay_recipient_maps $relay_domains
$canonical_maps $sender_canonical_maps $recipient_canonical_maps
$relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps
Ldap maps are:
* ldap-sysaliases.cf: This is a map for alias destinations. Although
this is a relay server, I'm resolving alias because I relay internal
mail by lmtp instead of smtp
* ldap-vmail.cf: This is the map for real users.
* ldap-relocated.cf: This a relocated map (for users who change their
email address).
In my tests I have found that during a smtp transaction the next
searches are done:
* When the "mail from:" is received, if this mail from is from a ldap
domain, a search in the relocated map is done for this mail from address.
* When the "rcpt to:" is received, postfix makes 4 searches for the
recipient address in this order:
1. In the relocated map
2. In the alias map
3. In the vmail map
4. In the alias map again
* When the data command is finished, then it makes the searches:
1. In the relocated map for the mail from address.
2. In the relocated map for the recipient address.
3. In the alias map for the recipient address
4. In the alias map again for the recipient address
5. In the relocated map (again) for the recipient address
With a total of 10 searches. I repeated the test with the same from and
recipient and almost all searches are done again. In fact, the only
searches it hasn't done are searches 1 and 2 after data command.
My question are:
* is it normal this behaviour? I mean is it normal all these searches?
Or I have something with a wrong configuration?
* Is there any way to cache these queries? In a normal transaction I
have only 4 different searches of a total of 8 (or 10) and if I repeat
the mail, all searches are done again? Is there any way to cache these
results so there was no need to research again all the information?
I have attached the output of postconf -n, ldap maps config files and
the logs at ldap server for connections from the postfix server (I have
replace the final DN where I have the information and sender and
recipient address used)
--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información _o)
y las Comunicaciones Aplicadas (ATICA) / \\
http://www.um.es/atica_(___V
Tfo: 868887590
Fax: 86337
address_verify_map = btree:${data_directory}/verify
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_at_myorigin = yes
append_dot_mydomain = yes
body_checks = pcre:/etc/postfix/body_checks.pcre
bounce_size_limit = 10240
broken_sasl_auth_clients = yes
canonical_maps = hash:/etc/postfix/canonical
command_directory = /usr/sbin
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
daemon_directory = /usr/lib/postfix
default_privs = nobody
disable_vrfy_command = yes
header_checks = pcre:/etc/postfix/header_checks.pcre
html_directory = /usr/share/doc/postfix/html
inet_interfaces = all
lmtp_destination_concurrency_limit = 5
lmtp_destination_recipient_limit = 10
mail_owner = postfix
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
message_size_limit = 2560
mime_header_checks = pcre:/etc/postfix/mime_header_checks.pcre
mydestination = $myhostname, localhost.\$mydomain, localhost
mydomain = um.es
myhostname = xenon11.um.es
mynetworks = 127.0.0.0/8, 155.54.0.0/16, 10.54.0.0/16, 10.56.0.0/16,
10.64.0.0/28, 172.19.0.0/16, 155.54.212.160/28
myorigin = um.es
nested_header_checks = pcre:/etc/postfix/nested_header_checks.pcre
newaliases_path = /usr/bin/newaliases
notify_classes = resource, software
parent_domain_matches_subdomains = smtpd_access_maps
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains
$relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps
$recipient_canonical_maps $relocated_maps $transport_maps $mynetworks
$smtpd_sender_login_maps
queue_directory = /var/spool/postfix
queue_minfree = 3840
rbl_reply_maps = hash:/etc/postfix/rb
Re: Ldap queries optimization
On Thu, Feb 16, 2012 at 10:49:10AM +0100, Angel L. Mateo wrote: > My config is: > > virtual_alias_maps = hash:/etc/postfix/alu-aliases, > hash:/etc/postfix/dif-aliases, > proxy:ldap:/etc/postfix/ldap-sysaliases.cf > > relay_recipient_maps = hash:/etc/postfix/relaydomains, > hash:/etc/postfix/alu-aliases, > hash:/etc/postfix/dif-aliases, > proxy:ldap:/etc/postfix/ldap-vmail.cf, > proxy:ldap:/etc/postfix/ldap-sysaliases.cf There is no need to list virtual alias lookup tables in relay_recipient_maps. Postfix performs that lookup automatically, therefore, the relay_recipient_maps setting should be just: relay_recipient_maps = hash:/etc/postfix/relaydomains, proxy:ldap:/etc/postfix/ldap-vmail.cf > In my tests I have found that during a smtp transaction the next > searches are done: If your LDAP tables contain no "bare" (just the local part) address lookup keys, you may consider using "%u@%d" instead of "%s" in the query definition. That could also avoid some unneeded lookups, otherwise Postfix performs the lookups it needs to, and unless you've failed to index your LDAP attributes appropriately, Postfix is unlikely to be a significant burden on LDAP, nor is LDAP likely to noticeably slow down Postfix. > mydestination = $myhostname, localhost.\$mydomain, localhost That "\" is unlikely to be what you want. > mynetworks = 127.0.0.0/8, 155.54.0.0/16, 10.54.0.0/16, 10.56.0.0/16, > 10.64.0.0/28, 172.19.0.0/16, 155.54.212.160/28 With the entire class 155.54/16 listed, no need for the final /28. > smtpd_banner = $myhostname NO UCE ESMTP Don't, or at least make it "$myhostname ESMTP NO UCE", that ESMTP is not semantically valid unless it immediately follows the hostname. > smtpd_client_restrictions = > reject_rbl_client rbl.um.es, > permit_sasl_authenticated, > check_client_access hash:/etc/postfix/whitelist_um, > reject_unknown_reverse_client_hostname, > check_client_access cidr:/etc/postfix/client_checks.cidr, > smtpd_data_restrictions = reject_unauth_pipelining, permit > smtpd_end_of_data_restrictions = $(smtpdEndOfDataRestrictions) > smtpd_helo_restrictions = > permit_mynetworks, > check_helo_access hash:/etc/postfix/helo_checks > smtpd_recipient_restrictions = > reject_non_fqdn_recipient, > reject_unknown_recipient_domain, > check_recipient_access pcre:/etc/postfix/recipient_checks.pcre, > check_recipient_access hash:/etc/postfix/verified_recipient_checks, > permit_mynetworks, > permit_sasl_authenticated, > reject_unauth_destination, > check_recipient_maps, > permit You don't need these last two, they are implicit. > smtpd_sender_restrictions = reject_non_fqdn_sender, > reject_unknown_sender_domain, > check_sender_access pcre:/etc/postfix/sender_checks.pcre Otherwise nothing else to do in Postfix, make sure your LDAP tables are properly indexed. -- Viktor.
Re: Ldap queries optimization
El 16/02/12 16:35, Viktor Dukhovni escribió: On Thu, Feb 16, 2012 at 10:49:10AM +0100, Angel L. Mateo wrote: My config is: virtual_alias_maps = hash:/etc/postfix/alu-aliases, hash:/etc/postfix/dif-aliases, proxy:ldap:/etc/postfix/ldap-sysaliases.cf relay_recipient_maps = hash:/etc/postfix/relaydomains, hash:/etc/postfix/alu-aliases, hash:/etc/postfix/dif-aliases, proxy:ldap:/etc/postfix/ldap-vmail.cf, proxy:ldap:/etc/postfix/ldap-sysaliases.cf There is no need to list virtual alias lookup tables in relay_recipient_maps. Postfix performs that lookup automatically, therefore, the relay_recipient_maps setting should be just: relay_recipient_maps = hash:/etc/postfix/relaydomains, proxy:ldap:/etc/postfix/ldap-vmail.cf In my tests I have found that during a smtp transaction the next searches are done: If your LDAP tables contain no "bare" (just the local part) address lookup keys, you may consider using "%u@%d" instead of "%s" in the query definition. That could also avoid some unneeded lookups, otherwise Postfix performs the lookups it needs to, and unless you've failed to index your LDAP attributes appropriately, Postfix is unlikely to be a significant burden on LDAP, nor is LDAP likely to noticeably slow down Postfix. mydestination = $myhostname, localhost.\$mydomain, localhost That "\" is unlikely to be what you want. mynetworks = 127.0.0.0/8, 155.54.0.0/16, 10.54.0.0/16, 10.56.0.0/16, 10.64.0.0/28, 172.19.0.0/16, 155.54.212.160/28 With the entire class 155.54/16 listed, no need for the final /28. smtpd_banner = $myhostname NO UCE ESMTP Don't, or at least make it "$myhostname ESMTP NO UCE", that ESMTP is not semantically valid unless it immediately follows the hostname. smtpd_client_restrictions = reject_rbl_client rbl.um.es, permit_sasl_authenticated, check_client_access hash:/etc/postfix/whitelist_um, reject_unknown_reverse_client_hostname, check_client_access cidr:/etc/postfix/client_checks.cidr, smtpd_data_restrictions = reject_unauth_pipelining, permit smtpd_end_of_data_restrictions = $(smtpdEndOfDataRestrictions) smtpd_helo_restrictions = permit_mynetworks, check_helo_access hash:/etc/postfix/helo_checks smtpd_recipient_restrictions = reject_non_fqdn_recipient, reject_unknown_recipient_domain, check_recipient_access pcre:/etc/postfix/recipient_checks.pcre, check_recipient_access hash:/etc/postfix/verified_recipient_checks, permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, check_recipient_maps, permit You don't need these last two, they are implicit. smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain, check_sender_access pcre:/etc/postfix/sender_checks.pcre Otherwise nothing else to do in Postfix, make sure your LDAP tables are properly indexed. Although I could refine this configuration changes, problem is not ldap indexes. I have all indexes created, openldap is answering all the queries postfix makes. The problem I'm trying to fix is that postfix is making lot of repeated queries. In the transactions I sent in my first email, in one transaction it makes 8 queries, 5 of then was the same query. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica_(___V Tfo: 868887590 Fax: 86337
Re: Ldap queries optimization
On Fri, Feb 17, 2012 at 08:41:31AM +0100, Angel L. Mateo wrote: > El 16/02/12 16:35, Viktor Dukhovni escribi?: > >On Thu, Feb 16, 2012 at 10:49:10AM +0100, Angel L. Mateo wrote: > > > >If your LDAP tables contain no "bare" (just the local part) address > >lookup keys, you may consider using "%u@%d" instead of "%s" in the > >query definition. That could also avoid some unneeded lookups, > >otherwise Postfix performs the lookups it needs to, and unless > >you've failed to index your LDAP attributes appropriately, Postfix > >is unlikely to be a significant burden on LDAP, nor is LDAP likely > >to noticeably slow down Postfix. > > Although I could refine this configuration changes, problem is not > ldap indexes. I have all indexes created, openldap is answering all > the queries postfix makes. The problem I'm trying to fix is that > postfix is making lot of repeated queries. In the transactions I > sent in my first email, in one transaction it makes 8 queries, 5 of > then was the same query. Some queries are repeated, this should have no significant impact. Focus on overalll performance rather than optimizing non-critical paths. Postfix is not monolithic, so cleanup(8) repeats some queries made by smtpd(8), and in smtpd(8) some queries are repeated because this makes the code more modular. This works well enough for a large number of Postfix sites, it will work for you too if you "don't sweat the small stuff". -- Viktor. http://idioms.thefreedictionary.com/don%27t+sweat+the+small+stuff
Re: Ldap queries optimization
El 17/02/12 17:18, Viktor Dukhovni escribió: Some queries are repeated, this should have no significant impact. Focus on overalll performance rather than optimizing non-critical paths. Postfix is not monolithic, so cleanup(8) repeats some queries made by smtpd(8), and in smtpd(8) some queries are repeated because this makes the code more modular. This works well enough for a large number of Postfix sites, it will work for you too if you "don't sweat the small stuff". I know it needs to repeat queries. But, in this case, why not implementing a cache for such maps? If I remember well in previous versions it did. And it did for other maps, like the one for address_verify_map. And I know I should focus on the real problem with my ldap server. But I would like postfix to make things better (and it doesn't mean it doesn't do well) -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica_(___V Tfo: 868887590 Fax: 86337
Re: Ldap queries optimization
Angel L. Mateo: > > Some queries are repeated, this should have no significant impact. > > Focus on overalll performance rather than optimizing non-critical > > paths. Postfix is not monolithic, so cleanup(8) repeats some queries > > made by smtpd(8), and in smtpd(8) some queries are repeated because > > this makes the code more modular. This works well enough for a > > large number of Postfix sites, it will work for you too if you > > "don't sweat the small stuff". > > > I know it needs to repeat queries. But, in this case, why not > implementing a cache for such maps? If I remember well in previous If you believe this is a problem, contribute a solution. Wietse
Re: Ldap queries optimization
On Wed, Feb 22, 2012 at 09:17:24AM +0100, Angel L. Mateo wrote: > >Postfix is not monolithic, so cleanup(8) repeats some queries > >made by smtpd(8), and in smtpd(8) some queries are repeated because > >this makes the code more modular. > > I know it needs to repeat queries. But, in this case, why not > implementing a cache for such maps? Additional complexity is not free, someone has to write, test and maintain the code. Code improvements are focused on useful outcomes. Optimization of non-critical paths is not sufficiently useful IMHO. > If I remember well in previous versions it did. There has never been a working LDAP query cache in the OpenLDAP library. Postfix support for the non-working query cache was withdrawn soon after the discovery that the OpenLDAP query cache was disfunctional and deprecated. Don't confuse caching of address verification probe results, or postscreen client test results, ... with caching of table lookup results. The former in fact use uncached table lookups. The latter cache information about the outside world (that is impossible or impractical to obtain with every lookup) not the result of a table lookup. Postfix has never had caching at the table-type driver level, and probably never should, such caching would cause plenty of WTF when table updates are not reflected in the behaviour of the MTA in a timely manner. People already complain from time to time about the tiny 1-element cache in the transport lookup code that is re-discovered from time to time in test environments when all mail is sent to a single address... There can be more caching in specific lookup contexts by various Postfix services, e.g. smtpd(8) could cache a few more lookups either for the lifetime of a client session, or LRU cache a small number of key/value tuples for a particular type of lookup for the server's lifetime. However, the cost-benefit analysis has to make sense. To reduce high query load, consider postscreen which typically eliminates a decent fraction of botnet SMTP sessions without performing any LDAP queries, ... just DNSBL lookups. In practice, the query load is very rarely high enough to give a well tuned LDAP or SQL server any trouble. -- Viktor.
