Disable outbound smtp pipelining?
It seems we are running into an increased number of hosts that have broken smtp pipelining. We are able to resolve delivery issues to these hosts using the solution that Wieste recommended a couple weeks ago: http://www.mail-archive.com/postfix-users@postfix.org/msg44135.html However, I'm not sure I fully understand why this fixes the problem for outbound email. On outbound email, Postfix is acting as the client and therefore one might think that smtpd_discard_ehlo_keyword_address_maps would not have any affect, since this setting is supposed to suppress EHLO keywords in response to a client. Does smtpd_discard_ehlo_keyword_address_maps also ignore EHLO keyword responses it receives on outbound connections? I'm guessing so, because unless something else changed, the solution worked. In fact, the above mentioned thread that Wieste is responding to was also an outbound delivery issue, and so it seems he might be implying that smtpd_discard_ehlo_keyword_address_maps has an effect on outbound connections as well. Ok, so assuming this solution works for outbound connections, this solution would require that we create and maintain a list of perhaps thousands of servers that have this problem. Does anyone have any experience with just turning it off completely for all hosts? Which, I suppose would be done like this: smtpd_discard_ehlo_keywords = pipelining,silent-discard Aside from a little extra bandwidth, would this cause outbound deliveries to go horrendously slower on a busy mail server? And is there a way to only disable smtp pipelining for outbound connections only? I better stop asking questions now since I might be getting ahead of my self if my assumption about smtpd_discard_ehlo_keyword_address_maps does not actually apply to outbound email... Thanks, Curtis
Re: Disable outbound smtp pipelining?
On Tue, Jun 05, 2012 at 09:46:16PM -0600, Curtis wrote: > However, I'm not sure I fully understand why this fixes the problem > for outbound email. On outbound email, Postfix is acting as the > client and therefore one might think that > smtpd_discard_ehlo_keyword_address_maps would not have any effect, > since this setting is supposed to suppress EHLO keywords in response > to a client. Correct the Postfix SMTP server-specific setting has no effect on Postfix SMTP client behaviour. > Does smtpd_discard_ehlo_keyword_address_maps also > ignore EHLO keyword responses it receives on outbound connections? NO. > I'm guessing so, because unless something else changed, the solution > worked. The observations are erroneous. > Does anyone have any > experience with just turning it off completely for all hosts? Which, > I suppose would be done like this: I've never had to work-around remote PIPELINING issues at remote sites and I ran a fairly busy MTA plant delivering ~2M messages per day. Some people have reported a small number of sites for which they needed work-arounds, but you should not be seeing a lot of remote peers with the issue unless there is a problem with a firewall at your site. Fix that. -- Viktor.
Re: Disable outbound smtp pipelining?
* Curtis : > It seems we are running into an increased number of hosts that have > broken smtp pipelining. We are able to resolve delivery issues to > these hosts using the solution that Wieste recommended a couple weeks > ago: > > http://www.mail-archive.com/postfix-users@postfix.org/msg44135.html > > However, I'm not sure I fully understand why this fixes the problem > for outbound email. On outbound email, Postfix is acting as the > client and therefore one might think that > smtpd_discard_ehlo_keyword_address_maps would not have any affect, This links lists smtp_discard_ehlo_keyword_address_maps (no d) not smtpd_discard_ehlo_keyword_address_maps -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Disable outbound smtp pipelining?
On 6/5/2012 10:44 PM, Viktor Dukhovni wrote: On Tue, Jun 05, 2012 at 09:46:16PM -0600, Curtis wrote: However, I'm not sure I fully understand why this fixes the problem for outbound email. On outbound email, Postfix is acting as the client and therefore one might think that smtpd_discard_ehlo_keyword_address_maps would not have any effect, since this setting is supposed to suppress EHLO keywords in response to a client. Correct the Postfix SMTP server-specific setting has no effect on Postfix SMTP client behaviour. Does smtpd_discard_ehlo_keyword_address_maps also ignore EHLO keyword responses it receives on outbound connections? NO. I'm guessing so, because unless something else changed, the solution worked. The observations are erroneous. Actually it was my reference to smtpd_discard_ehlo_keyword_address_maps that was erroneous. (Thanks Ralf for pointing out my error!) The actual parameter name referenced in the solution that Wiestse provided in the thread I referenced was smtp_discard_ehlo_keyword_address_maps (no d). The fix I implemented was actually smtp_discard_ehlo_keyword_address_maps (no d) as well. When I went to look up the parameter in the documentation to see why it worked, I got my parameter names mixed up, which is why I was confused as to why the solution worked.Now that I'm looking at the right parameter name in the documentation, it is clear that smtp_discard_ehlo_keyword_address_maps does the job for outbound connections. I don't get into the Postfix configs often enough, as I had forgotten that there were several parameters that were so similarly named (which is actually a good thing once you understand the naming scheme). Does anyone have any experience with just turning it off completely for all hosts? Which, I suppose would be done like this: I've never had to work-around remote PIPELINING issues at remote sites and I ran a fairly busy MTA plant delivering ~2M messages per day. Some people have reported a small number of sites for which they needed work-arounds, but you should not be seeing a lot of remote peers with the issue unless there is a problem with a firewall at your site. Fix that. Our mail servers are busy here too. I think you'll start to see some issues as well, as Wietse clearly shows that there are some broken servers out there: http://www.mail-archive.com/postfix-users@postfix.org/msg44135.html The one referenced in the above thread is a Lunarpages server, which is where we first ran into the same problem, and now we've run into the issue with a Bluehost server as well. Bluehost claims to host millions of domains and I suppose Lunarpages isn't that far behind. I suspect they each have several hundred mail servers between them that would all need to be added to the exceptions list that Wietse recommended. I think these hosts broke something in May as we've just started get complaints of intermittent issues delivering to them. So again, I'll ask if anyone out there has resorted to turning off pipelining on outbound connections? ... smtp_discard_ehlo_keywords = pipelining,silent-discard (Again, this time without the d.) Aside from a little extra bandwidth, would this cause outbound deliveries to go horrendously slower on a busy mail server? Thanks, Curtis
Re: Disable outbound smtp pipelining?
Curtis: > smtp_discard_ehlo_keywords = pipelining,silent-discard > > (Again, this time without the d.) Aside from a little extra bandwidth, > would this cause outbound deliveries to go horrendously slower on a busy > mail server? The issue is NOT bandwidth (i.e. the width of a pipe). It's latency (i.e. the distance between pipe endpoints). You can increase the former, but you can't break the speed-of-light limit. Pipelining reduces the number of TCP round-trip times, At a minimum there will be 5 round-trip times (SYN-SYNACK, ACK-220, EHLO-EHLOREPLY, DATA-DATAREPLY, ENDOFDATA-ENDOFDATAREPLY; the SMTP client does not wait for final handshake completion). Disabling pipelining adds one round-trip time for each MAIL FROM and for each RCPT TO command, from 5 to 7 or more. Your SMTP server may spend 40% more time delivering mail, depending on the number of messages per destination (which reduces impact of DNS lookups), and on the number of recipients per message. Postfix has bug workarounds for CISCO PIX that are enabled automatically when mail has been queued for 500s ore more. Maybe some of the optimizations such as command pipelining could also be made time-dependent. Wietse
Re: Disable outbound smtp pipelining?
On 6/6/2012 8:45 AM, Wietse Venema wrote: Curtis: smtp_discard_ehlo_keywords = pipelining,silent-discard (Again, this time without the d.) Aside from a little extra bandwidth, would this cause outbound deliveries to go horrendously slower on a busy mail server? The issue is NOT bandwidth (i.e. the width of a pipe). It's latency Yeah, I realized that the bandwidth impact would be minimal. (Someone else noted in another thread a slight increase of bandwidth due to the extra TCP round-trips.) (i.e. the distance between pipe endpoints). You can increase the former, but you can't break the speed-of-light limit. Pipelining reduces the number of TCP round-trip times, At a minimum there will be 5 round-trip times (SYN-SYNACK, ACK-220, EHLO-EHLOREPLY, DATA-DATAREPLY, ENDOFDATA-ENDOFDATAREPLY; the SMTP client does not wait for final handshake completion). Disabling pipelining adds one round-trip time for each MAIL FROM and for each RCPT TO command, from 5 to 7 or more. Your SMTP server may spend 40% more time delivering mail, depending on the number of messages per destination (which reduces impact of DNS lookups), and on the number of recipients per message. Yeah, I figured it would be a pretty significant slow down for hosts that support it... I just wasn't sure what percentage of hosts support pipelining. If only 10% of hosts were using it in the first place, the 40% extra delivery time lost on those hosts might be managable... but, if you're saying that the majority of hosts do have pipelining enabled (it makes sense that they would), then a 40% slow down would probably be too expensive. Postfix has bug workarounds for CISCO PIX that are enabled automatically when mail has been queued for 500s ore more. Maybe some of the optimizations such as command pipelining could also be made time-dependent. That would be awesome. I will stay tuned for that. In the mean time, it sounds like we'll have to stick with the smtp_discard_ehlo_keyword_address_maps solution. We'll just have to set up something to monitor the logs and automatically add problem hosts. I love how there's almost always multiple ways to solve a problem with Postfix. Postfix rocks! :-) Curtis Wietse
Re: Disable outbound smtp pipelining?
Den 2012-06-06 05:46, Curtis skrev: [snip] I better stop asking questions now since I might be getting ahead of my self if my assumption about smtpd_discard_ehlo_keyword_address_maps does not actually apply to outbound email... back to basic: smtpd_* is incomming smtp_* is outgoing
Re: Disable outbound smtp pipelining?
Curtis: > Yeah, I figured it would be a pretty significant slow down for hosts > that support it... I just wasn't sure what percentage of hosts support > pipelining. If only 10% of hosts were using it in the first place, the > 40% extra delivery time lost on those hosts might be managable... but, > if you're saying that the majority of hosts do have pipelining enabled > (it makes sense that they would), then a 40% slow down would probably be > too expensive. I have no statistics for ESMTP PIPELINING deployment. Many major MTA implementation support this, but unfortunately some "security" "firewall" implementors still make basic mistakes, and that is probably what we are looking at. Wietse
Re: Disable outbound smtp pipelining?
On Wed, Jun 06, 2012 at 03:24:41PM -0400, Wietse Venema wrote: > Curtis: > > Yeah, I figured it would be a pretty significant slow down for hosts > > that support it... I just wasn't sure what percentage of hosts support > > pipelining. If only 10% of hosts were using it in the first place, the > > 40% extra delivery time lost on those hosts might be managable... but, > > if you're saying that the majority of hosts do have pipelining enabled > > (it makes sense that they would), then a 40% slow down would probably be > > too expensive. > > I have no statistics for ESMTP PIPELINING deployment. Many major > MTA implementation support this, but unfortunately some "security" > "firewall" implementors still make basic mistakes, and that is > probably what we are looking at. > > Wietse > Hi Wietse, I really like your idea about a time-based fallback to a no-pipelining send similar to the Cisco PIX work-around. Regards, Ken
Re: Disable outbound smtp pipelining?
On Wed, 06 Jun 2012 20:40:01 +0200 Benny Pedersen articulated: >Den 2012-06-06 05:46, Curtis skrev: > >[snip] >> I better stop asking questions now since I might be getting ahead of >> my self if my assumption about >> smtpd_discard_ehlo_keyword_address_maps >> does not actually apply to outbound email... > >back to basic: > >smtpd_* is incomming >smtp_* is outgoing I know it may sound stupid; however, I have seen this exact question posted on various forums before. Perhaps on the "http://www.postfix.com/postconf.5.html"; page along with the other documentation at the top of the page, this additional documentation could be added. Considering the number of times I have seen this question asked, I think it is something to consider. Just my 2¢ on the matter. -- Jerry ✌ postfix-u...@seibercom.net _ TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html It is difficult to soar with the eagles when you work with turkeys.
