Re: Connection caching/SMTP piggybacking and postfix?
On Wed, 27 Feb 2013, Wietse Venema wrote: Well, how does one migrate **AN OPEN TLS SESSION** from one process into the other? I am not aware an OpenSSL API for doing that. Then just don’t do that… keep it in the other process. (I think OpenSSH does that, though probably not standard TLS.) bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 54881-314 HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Sebastian Mancke
Re: Connection caching/SMTP piggybacking and postfix?
Am 28.02.2013 10:38, schrieb Thorsten Glaser: On Wed, 27 Feb 2013, Wietse Venema wrote: Well, how does one migrate **AN OPEN TLS SESSION** from one process into the other? I am not aware an OpenSSL API for doing that. Then just don’t do that… keep it in the other process and how do you imagine that? did you try to understand how postfix works? i guess no signature.asc Description: OpenPGP digital signature
Re: Connection caching/SMTP piggybacking and postfix?
On Wed, 27 Feb 2013, Viktor Dukhovni wrote: No, connection caching has nothing to do with concurrency. Whether connections are cached, or not, Postfix will attempt parallel delivery up to the configured concurrency limit: Hrm. Okay, lowering default_destination_concurrency_limit is certainly a viable workaround. I’m not the Postfix expert here, and he didn’t suggest it to me. of the MX hosts for a destination is unreachable. The load on the remote server is identical with and without connection caching. That’s untrue because it certainly makes a difference whether you send mails in parallel or serial, especially when the receiving host then spawns an LDAP connection, a shell command and an MSA to forward the mail further. Sendmail lacks a queue-manager, if you really want to avoid Sendmail melt-downs under load, deploy Postfix. Sendmail does not “melt down”, it just refuses connections until the load gets lower again. I sure hope Postfix does the same if the system load is high? (The load is not necessarily from sendmail; this is a spamfilter setup which is tightly coupled to several system components, one of which is sendmail, but the LDAP search and two sed invocations as well as calling bmf are the CPU hogs here.) Connection caching without the ability to re-use the connection in a different delivery-agent process *is* abusive of the remote system, since you're keeping a remote process occupied and idle, while making new connections. This will not be implemented. You can just close them immediately if there’s no mail in the queue for the remote system any more. Or after a second. When disgruntled, especially due to problems with system A, don't rant on the mailing list for system B. My problem here is with system B as system A is working just fine and by the specs. bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 54881-314 HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Sebastian Mancke
Re: Connection caching/SMTP piggybacking and postfix?
On Thu, 28 Feb 2013, Reindl Harald wrote: Then just don’t do that… keep it in the other process did you try to understand how postfix works? i guess no If the way something works makes it impossible to do something that is a requirement it says something about the way that particular something works, not about the requirement. But Viktor provided me with a viable workaround I did not know (considering I’m not the Postfix expert here). bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 54881-314 HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Sebastian Mancke
Re: Connection caching/SMTP piggybacking and postfix?
Am 28.02.2013 10:45, schrieb Thorsten Glaser: On Thu, 28 Feb 2013, Reindl Harald wrote: Then just don’t do that… keep it in the other process did you try to understand how postfix works? i guess no If the way something works makes it impossible to do something that is a requirement it says something about the way that particular something works, not about the requirement after years on this mailing-list in 99 out of 100 cases the opposite is true because mostly people are coming up with a solution not working as expected instead to describe or understand the problem if you have a VERY HIGH amount of outgoing mails to send you can not expect that this works in burst with only one outgoing server - nothing will change this signature.asc Description: OpenPGP digital signature
Re: Connection caching/SMTP piggybacking and postfix?
On Thu, 28 Feb 2013, Reindl Harald wrote: if you have a VERY HIGH amount of outgoing mails to send you can not expect that this works in burst with only one outgoing server - nothing will change this Yes well, we have three, but I’m running into firewall admins currently so in practice there’s only one at the moment. I hope that this part of the issue will also resolve eventually… I set the concurrency limit to 4 for now. bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 54881-314 HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Sebastian Mancke
Re: Connection caching/SMTP piggybacking and postfix?
Thorsten Glaser: On Wed, 27 Feb 2013, Wietse Venema wrote: Well, how does one migrate **AN OPEN TLS SESSION** from one process into the other? I am not aware an OpenSSL API for doing that. Then just don?t do that? keep it in the other process. (I think OpenSSH does that, though probably not standard TLS.) I welcome you to propose a design that is as scalable as the rest of Postfix. A centralized process that proxies all TLS sessions for Postfix does not meet that requirement. Wietse
Re: Connection caching/SMTP piggybacking and postfix?
Thorsten Glaser: [ Charset UTF-8 unsupported, converting... ] On Wed, 27 Feb 2013, Viktor Dukhovni wrote: No, connection caching has nothing to do with concurrency. Whether connections are cached, or not, Postfix will attempt parallel delivery up to the configured concurrency limit: Hrm. Okay, lowering default_destination_concurrency_limit is certainly a viable workaround. I?m not the Postfix expert here, and he didn?t suggest it to me. of the MX hosts for a destination is unreachable. The load on the remote server is identical with and without connection caching. That?s untrue because it certainly makes a difference whether you send mails in parallel or serial, especially when the receiving Connection caching has nothing to do with concurrency. Sheesh. Wietse
Re: Connection caching/SMTP piggybacking and postfix?
On Thu, 28 Feb 2013, Wietse Venema wrote: That?s untrue because it certainly makes a difference whether you send mails in parallel or serial, especially when the receiving Connection caching has nothing to do with concurrency. Sheesh. I was assuming that, when you already have a connection to a receiving MX, you’d not open another one but reuse the existing one, so it would, of course, have something to do with that. If not… well, that’s the qmail model. I welcome you to propose a design that is as scalable as the rest I am *not* interested in Postfix. I’m not even primarily working with Postfix. I came here because I was suffering, on the receiving side, from perceived Postfix misbehaviour. I’ve got a configuration item with which I can reduce the possibility of the behaviour becoming an issue, which has helped me. That was it. bye, //mirabilos -- tarent solutions GmbH Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/ Tel: +49 228 54881-393 • Fax: +49 228 54881-314 HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941 Geschäftsführer: Boris Esser, Sebastian Mancke
Re: Connection caching/SMTP piggybacking and postfix?
Thorsten Glaser: On Thu, 28 Feb 2013, Wietse Venema wrote: That?s untrue because it certainly makes a difference whether you send mails in parallel or serial, especially when the receiving Connection caching has nothing to do with concurrency. Sheesh. I was assuming that, when you already have a connection to a receiving MX, you?d not open another one but reuse the existing Concurrency is the number of simultaneous connections. It has nothing to do with connection caching, which is using one connection for more than one transaction. Both are configurable with Postfix. Wietse
Re: Connection caching/SMTP piggybacking and postfix?
On Thu, Feb 28, 2013 at 01:16:26PM +0100, Thorsten Glaser wrote: Connection caching has nothing to do with concurrency. Sheesh. I was assuming that, when you already have a connection to a receiving MX, you'd not open another one but reuse the existing one, so it would, of course, have something to do with that. If not? well, that's the qmail model. One can't immediately re-use an existing connection when some other process is currently using it! To do that, one would have to always wait for the other process to relinquish the connection, which forces concurrency = 1. This has nothing to do with qmail, which uses a recipient limit of 1 which leads to more deliveries with multi-recipient messages than with Postfix or Sendmail. Postfix reuses *idle* existing connections, but creates new ones in *parallel* up to the concurrency limit to avoid severely limiting throughput to destinations with high capacity in the face of high single message delivery latency. Sendmail has no concurrency limits at all, since each process runs in isolation (no queue-manager), it just hiccups at random when the CPU load is high, which is not a meaningful signal that the MTA should pause deliveries or refuse new connections. You mentioned a Sendmail QueueLA or RefuseLA of 8, this is a meagerly limit on modern multi-cpu server hardware. ~10 years ago, I was tuning multi-CPU Sendmail servers of that era with a QueueLA of 64. I welcome you to propose a design that is as scalable as the rest I am *not* interested in Postfix. I'm not even primarily working with Postfix. You're also not interested in any form of critical thinking before you spout non sequiturs, or even a modicum of humility which would suggest asking how to improve latency in your two MTA setup, nor do you care to do any research, rather you barge in here in response to an 11-year old post blaming Postfix for Sendmail design flaws. I smell a troll, please don't come back. -- Viktor.
Re: Connection caching/SMTP piggybacking and postfix?
Wietse Venema wietse at porcupine.org writes: deliveries. Proper SMTP connection caching is not done by the SMTP clients but by a separate process that is queried by SMTP clients. If you don’t manage to do that with TLS, this statement is plainly wrong. Connection caching is a matter of also being nice to the recipient, who doesn’t have to spawn many sendmail processes in parallel to handle the roughly 500 mails that piled up on the Postfix side, right now. If Postfix were able to do proper connection reuse (WITH TLS, of course!) the mail delivery wouldn’t stop after the first couple of messages was submitted (because sendmail stops listening when the load reaches 8) and wouldn’t need to be retried (manually using postqueue -f once the sendmail load is down again, because the sending box is too busy to ever empty this queue with the default retry times). So I request of Postfix to implement connection reuse, supporting TLS (that means with*out* the separate dæmon unless you manage to be able to do that and still use TLS), by default. Thanks. Sorry for my tone, I’m a tad disgruntled right now *and* under pressure from my cow-orkers to get the pile of mails delivered.
Re: Connection caching/SMTP piggybacking and postfix?
On Wed, Feb 27, 2013 at 01:45:04PM +, Thorsten Glaser wrote: deliveries. Proper SMTP connection caching is not done by the SMTP clients but by a separate process that is queried by SMTP clients. If you don't manage to do that with TLS, this statement is plainly wrong. If you don't understand what you're talking about, you're liable to say things that are plainly wrong. Connection caching is a matter of also being nice to the recipient, who doesn't have to spawn many sendmail processes in parallel to handle the roughly 500 mails that piled up on the Postfix side, right now. No, connection caching has nothing to do with concurrency. Whether connections are cached, or not, Postfix will attempt parallel delivery up to the configured concurrency limit: http://www.postfix.org/postconf.5.html#transport_destination_concurrency_limit http://www.postfix.org/postconf.5.html#smtp_destination_concurrency_limit http://www.postfix.org/postconf.5.html#relay_destination_concurrency_limit ... Connection caching is a latency reduction optimization on the SMTP client to reduce the impact of TCP connect timeouts when a subset of the MX hosts for a destination is unreachable. The load on the remote server is identical with and without connection caching. If Postfix were able to do proper connection reuse (WITH TLS, of course!) the mail delivery wouldn't stop after the first couple of messages was submitted (because sendmail stops listening when the load reaches 8) Don't blame Sendmail design flaws on Postfix. The RefuseLA and QueueLA features are fundamentally broken. Set these sufficiently high that you never hit these in practice, an MTA should never suspend delivery under high load, that just leads to even higher load later. Sendmail lacks a queue-manager, if you really want to avoid Sendmail melt-downs under load, deploy Postfix. So I request of Postfix to implement connection reuse, supporting TLS (that means with*out* the separate d?mon unless you manage to be able to do that and still use TLS), by default. Connection caching without the ability to re-use the connection in a different delivery-agent process *is* abusive of the remote system, since you're keeping a remote process occupied and idle, while making new connections. This will not be implemented. Migrating TLS sessions between processes is non-trivial due to understandablel limitations in the OpenSSL API (it is not easy to completely serialize the cryptographic state of an established SSL connection). Thanks. Sorry for my tone, I'm a tad disgruntled right now *and* under pressure from my cow-orkers to get the pile of mails delivered. When disgruntled, especially due to problems with system A, don't rant on the mailing list for system B. -- Viktor.
Re: Connection caching/SMTP piggybacking and postfix?
Thorsten Glaser: Wietse Venema wietse at porcupine.org writes: deliveries. Proper SMTP connection caching is not done by the SMTP clients but by a separate process that is queried by SMTP clients. If you don?t manage to do that with TLS, this statement is plainly wrong. Well, how does one migrate **AN OPEN TLS SESSION** from one process into the other? I am not aware an OpenSSL API for doing that. Wietse
