Low performance when using mode http for Exchange-Outlook-Anywhere-RPC
Hello List, I placed haproxy in front of our exchange cluster for OutlookAnywhere Clients (that's just RPCoverHTTP, port 443). SSL is terminated by pound and forwards traffic on loopback to haproxy. Everything works but it's awfully slow when i use mode http; requests look like this: RPC_IN_DATA /rpc/rpcproxy.dll?[...] HTTP/1.1 HTTP/1.1 200 Success..Content-Type:application/rpc..Content-Length:1073741824 RPC_OUT_DATA /rpc/rpcproxy.dll?[..] HTTP/1.1 HTTP/1.1 200 Success..Content-Type:application/rpc..Content-Length:1073741824 (this is the nature of microsoft rpc I've been told, it's using two channels to make it duplex) and are held open in both cases (mode tcp and mode http) due to long configured timeouts (and no option httpclose for the http-mode) I can't see a big difference in how packets look, there's an awful lot of nearly empty packets with Syn and Push set, but that's in both cases. Packets reach 16k (that's the mtu of the loopback device) The only difference you can see in the Outlook Connection Info Window is the response-time: with mode tcp it's around 16-200ms while in http-mode it's above 800ms. Any hint? Or is mode-http of no use because I'll be unable to inject stuff into the session-cookie at all? Thx in advance Beni.
Re: Low performance when using mode http for Exchange-Outlook-Anywhere-RPC
Hello Benedikt, On Tue, May 08, 2012 at 05:33:46PM +0200, Benedikt Fraunhofer wrote: Hello List, I placed haproxy in front of our exchange cluster for OutlookAnywhere Clients (that's just RPCoverHTTP, port 443). SSL is terminated by pound and forwards traffic on loopback to haproxy. Everything works but it's awfully slow when i use mode http; requests look like this: RPC_IN_DATA /rpc/rpcproxy.dll?[...] HTTP/1.1 HTTP/1.1 200 Success..Content-Type:application/rpc..Content-Length:1073741824 RPC_OUT_DATA /rpc/rpcproxy.dll?[..] HTTP/1.1 HTTP/1.1 200 Success..Content-Type:application/rpc..Content-Length:1073741824 (this is the nature of microsoft rpc I've been told, it's using two channels to make it duplex) and are held open in both cases (mode tcp and mode http) due to long configured timeouts (and no option httpclose for the http-mode) I can't see a big difference in how packets look, there's an awful lot of nearly empty packets with Syn and Push set, but that's in both cases. Packets reach 16k (that's the mtu of the loopback device) The only difference you can see in the Outlook Connection Info Window is the response-time: with mode tcp it's around 16-200ms while in http-mode it's above 800ms. Any hint? Or is mode-http of no use because I'll be unable to inject stuff into the session-cookie at all? For such border-line uses, you need to enable option http-no-delay. By default, haproxy tries to merge as many TCP segments as possible. But in your case, the application is abusing the HTTP protocol by expecting that incomplete data will be immediately delivered to the other end (which is wrong but it's not the first time microsoft does crappy things with HTTP, see NTLM). Please note that such protocols will generally not work across caches or anti-virus proxies. With option http-no-delay, haproxy refrains from merging consecutive segments and forwards data as fast as they enter. This obviously leads to higher CPU and network usage due to the increase of small packets, but at least it will work as expected. Regards, Willy
Re: Low performance when using mode http for Exchange-Outlook-Anywhere-RPC
Hello Willy, 2012/5/8 Willy Tarreau w...@1wt.eu: For such border-line uses, you need to enable option http-no-delay. By great! that did it. default, haproxy tries to merge as many TCP segments as possible. But in your case, the application is abusing the HTTP protocol by expecting that Does haproxy even discard the PUSH Flag on tcp-packets? or is microsoft simply not sending it? wrong but it's not the first time microsoft does crappy things with HTTP, see NTLM). HTTP is such a versatile protocol, and, as already being sung by some some of them want to be abused :) Please note that such protocols will generally not work across caches or anti-virus proxies. Well. In this case, all proxies on the client side will only see https traffic; they should not be able to inspect that. With option http-no-delay, haproxy refrains from merging consecutive segments and forwards data as fast as they enter. This obviously leads to higher CPU and network usage due to the increase of small packets, but at least it will work as expected. I'm following the mailing-list and saw that you did something different for web-sockets? [...]because haproxy switches to tunnel mode when it sees the WS handshake and it keeps the connection open for as long as there is traffic.[...] or is tunnel mode something different and keeps the inner working of assembling and merging packets in the http-mode I dunno if that's important but maybe one should do that for Content-Type:application/rpc, too, but anyhow it easy to throw in the option and i'm more than happy that i can stay with my setup and have client-stickiness for dryout-purposes. And congrats to your new president :) Thx again and again Beni.
Re: Low performance when using mode http for Exchange-Outlook-Anywhere-RPC
On Tue, May 08, 2012 at 06:14:15PM +0200, Benedikt Fraunhofer wrote: For such border-line uses, you need to enable option http-no-delay. By great! that did it. Fine, thanks for your feedback. Does haproxy even discard the PUSH Flag on tcp-packets? or is microsoft simply not sending it? It cannot know if there was a PUSH since only the kernel knows it. Haproxy does not receive packets but data. However the http-no-delay option ensures that a PUSH is emitted on output segments so that none of them waits anywhere. wrong but it's not the first time microsoft does crappy things with HTTP, see NTLM). HTTP is such a versatile protocol, and, as already being sung by some some of them want to be abused :) :-) Anyway not respecting standards is what causes trouble in real life. Please note that such protocols will generally not work across caches or anti-virus proxies. Well. In this case, all proxies on the client side will only see https traffic; they should not be able to inspect that. Some of them will still be able to do it, many proxies do inspect HTTPS right now by spoofing certificates. And on the server side, you cannot easily offload SSL to a proxy or install a load balancer (you needed one with a specific option to keep the latency low). I'm following the mailing-list and saw that you did something different for web-sockets? [...]because haproxy switches to tunnel mode when it sees the WS handshake and it keeps the connection open for as long as there is traffic.[...] exactly. or is tunnel mode something different and keeps the inner working of assembling and merging packets in the http-mode Haproxy switches to a tunnel mode when it sees a CONNECT request succeed or a 101 response to an Upgrade request. When in tunnel mode, it basically falls back to TCP mode and forwards data as fast as possible between both sides without inspecting anything. I dunno if that's important but maybe one should do that for Content-Type:application/rpc, too, but anyhow it easy to throw in the option and i'm more than happy that i can stay with my setup and have client-stickiness for dryout-purposes. No, I really don't want to change the TCP behaviour based on a content-type. It's not normal at all, the lower layers have no reason to adapt to contents. It's somewhat a violation of the layering model that we must not perform at all. Regards, Willy
