Sorry, I didn't send this to the list; resending First of all, thanks everybody for overwhelming me with helpful pointers!
A quick update: We traced the unusually high response times in the fwd-rcpt-to phase to a side effect of our deployment: we were swamping the LDAP directory with hundreds of queries per second, and the back-end SMTP (which btw is not Postfix but a commercial one, my bad for forgetting to write that (but the hints did apply!)) was trying to access the same LDAP directory to verify local usernames). Basically, we DoS'ed our own directory, thus severely hampering our own SMTP backend, which showed up in the Amavisd timing as fwd-rcpt-to. As a brutal workaround we exported the list of spamlovers into a text file and stopped querying the LDAP directories. Another team is setting up a couple of additional LDAP slaves for our queries. Also, I noticed we run LDAP queries for *every* recipient, even those not belonging to our 5 domains. I didn't find any clue in the documentation on how to disable this, but will keep looking. Also, following hints in the list and on available literature, we moved the postfix instances away on 5 Solaris machines which were previously hosting the front-end custom SMTP instances and this is definitely giving us a more predictable behaviour: no more hitting swapfiles or I/O trashing and a much better load levelling on the 14 boxes (previously the balancer only saw the postfix instances, which were very quick at accepting and queueing mail, so all the servers appeared to be equally responsive and the load was not adaptive. Now, with an Amavis virtualhost spreading traffic to the 14 10024 ports, the balancer has much better info about system load and every new session is opened against the least loaded server.). The new setup is giving us some postfix issues (apparently we need some tuning there as well, we temporarily turned off logging via Solaris' Syslog which was killing the performances) but we are taking care of it. On the Amavis side, we brought up $max_servers and $max_requests (at 20 atm, we are experimenting other values to find the sweet spot for our setup) and have upgraded 6 machines to 2GB to run a comparison. The timing breakdown in the logs is now much more reasonable, total processing time averages 400ms and is never over 2000 ms (except for a few huge mails, but that's expected) and most of the time is now accounted to SA check and AV-scan as we expected. We are running a few load and stress tests but we'll probably have to bring 16 more servers in the pool, since our peak load can be 24000 mails per minute. We will keep tweaking the Amavis configuration because even with 30 servers this will amount to 13 emails/sec sustained and the servers might not be able to service all requests satisfactorily. Thanks everybody on the list for your suggestions, and please if you have more don't hesitate to let me know :) ------------------------------------------------------- SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf _______________________________________________ AMaViS-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/amavis-user AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3 AMaViS-HowTos:http://www.amavis.org/howto/
