Re: low load client payload intermittently dropped with a "cD" error (v1.7.3)
9.99.8000: Flags [F.], seq 1, ack 2, win 229, options [nop,nop,TS val 114824224 ecr 548626318], length 0 F 2017-06-04 16:59:24.119154 IP 99.99.99.99.8000 > 10.1.77.243.37266: Flags [R], seq 1520443120, win 0, length 0 -- lfs On Thu, Apr 13, 2017 at 1:58 AM, Willy Tarreau wrote: > Hi Lincoln, > > On Wed, Apr 12, 2017 at 08:24:41PM -0400, Lincoln Stern wrote: > (...) > > *haproxy finishes connecting to the server (SYNACK/ACK) (good)* > > 38.120527 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [S.], seq > > 4125907118, ack 35568, win 28960, options [mss 1460,sackOK,TS val > > 662461622 ecr 82055529,nop,wscale 8], length 0 > > 38.120619 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [.], ack 1, > > win 229, options [nop,nop,TS val 82055545 ecr 662461622], length 0 > > > > *And now there is nothing for 5 seconds when we should have seen a data > > packet from haproxy to the server with the "length 198" payload. * > > > > *It appears that haproxy never tried to send the data!?!? * > > Thanks for this capture! This is a regression introduced in 1.7.3 while > trying to fix a bug in 1.7.2, it was fixed later by this commit : > > commit 51f2336ef7738c945ccedf5f1179825ac401862c > Author: Willy Tarreau > Date: Sat Mar 18 17:11:37 2017 +0100 > > BUG/MAJOR: stream-int: do not depend on connection flags to detect > connectio > (...) > > Please upgrade to 1.7.5 and it will be OK. > > Regards, > Willy >
Re: low load client payload intermittently dropped with a "cD" error (v1.7.3)
Thanks Bryan, The problem I'm having is isolated to the first one second of the connection not the end Here is a summary of the tcp traffic. Hopefully it makes the example more clear. *client connects to haproxy: (all good)* 38.057127 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [S], seq 2113072542, win 43690, options [mss 65495,sackOK,TS val 82055529 ecr 0,nop,wscale 7], length 0 38.057156 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [S.], seq 3284611992, ack 2113072543, win 43690, options [mss 65495,sackOK,TS val 82055529 ecr 82055529,nop,wscale 7], length 0 38.057178 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [.], ack 1, win 342, options [nop,nop,TS val 82055529 ecr 82055529], length 0 *haproxy starts connecting to server (SYN) (good)* 38.057295 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [S], seq 35567, win 29200, options [mss 1460,sackOK,TS val 82055529 ecr 0,nop,wscale 7], length 0 *client sends 198 bytes to initiate SSL connection and we have an ACK (good)* 38.060539 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [P.], seq 1:199, ack 1, win 342, options [nop,nop,TS val 82055530 ecr 82055529], length 198 38.060598 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [.], ack 199, win 350, options [nop,nop,TS val 82055530 ecr 82055530], length 0 *haproxy finishes connecting to the server (SYNACK/ACK) (good)* 38.120527 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [S.], seq 4125907118, ack 35568, win 28960, options [mss 1460,sackOK,TS val 662461622 ecr 82055529,nop,wscale 8], length 0 38.120619 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [.], ack 1, win 229, options [nop,nop,TS val 82055545 ecr 662461622], length 0 *And now there is nothing for 5 seconds when we should have seen a data packet from haproxy to the server with the "length 198" payload. * *It appears that haproxy never tried to send the data!?!? * *The server then disconnects 5 seconds into the transaction since it got no data. (that is the way the server is suppose to behave)* 43.183207 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 662466683 ecr 82055545], length 0 On Mon, Apr 10, 2017 at 7:58 PM, Bryan Talbot wrote: > > On Apr 8, 2017, at Apr 8, 2:24 PM, Lincoln Stern > wrote: > > I'm not sure how to interpret this, but it appears that haproxy is dropping > client payload intermittently (1/100). I have included tcpdumps and logs > to > show what is happening. > > Am I doing something wrong? I have no idea what could be causing this or > how > to go about debugging it. I cannot reproduce it, but I do observe in > production ~2 times > a day across 20 instances and 2K connections. > > Any help or advice would be greatly appreciated. > > > > > You’re in TCP mode with 60 second timeouts. So, if the connection is idle > for that long then the proxy will disconnect. If you need idle connections > to stick around longer and mix http and tcp traffic then you probably want > to set “timeout tunnel” to however long you’re willing to let idle tcp > connections sit around and not impact http timeouts. If you only need > long-lived tcp “tunnel” connections, then you can instead just increase > both your “timeout client” and “timeout server” timeouts to cover your > requirements. > > -Bryan > > > > What I'm trying to accomplish is to provide HA availability over two routes > (i.e. internet providers). One acts as primary and I gave it a "static-rr" > "weight" of 256 and the other as backup and has a weight of "1". Backup > should only be used in case of primary failure. > > > log: > Apr 4 18:55:27 app055 haproxy[13666]: 127.0.0.1:42262 > [04/Apr/2017:18:54:41.585] ws-local servers/server1 1/86/45978 4503 5873 -- > 0/0/0/0/0 0/0 > Apr 4 22:46:37 app055 haproxy[13666]: 127.0.0.1:47130 > [04/Apr/2017:22:46:36.931] ws-local servers/server1 1/62/663 7979 517 -- > 0/0/0/0/0 0/0 > Apr 4 22:46:38 app055 haproxy[13666]: 127.0.0.1:32931 > [04/Apr/2017:22:46:37.698] ws-local servers/server1 1/55/405 3062 553 -- > 1/1/1/1/0 0/0 > Apr 4 22:46:43 app055 haproxy[13666]: 127.0.0.1:41748 > [04/Apr/2017:22:46:43.190] ws-local servers/server1 1/115/452 7979 517 -- > 2/2/2/2/0 0/0 > Apr 4 22:46:46 app055 haproxy[13666]: 127.0.0.1:57226 > [04/Apr/2017:22:46:43.576] ws-local servers/server1 1/76/3066 2921 538 -- > 1/1/1/1/0 0/0 > Apr 4 22:46:47 app055 haproxy[13666]: 127.0.0.1:39656 > [04/Apr/2017:22:46:47.072] ws-local servers/server1 1/67/460 8254 528 -- > 1/1/1/1/0 0/0 > Apr 4 22:47:38 app055 haproxy[13666]: 127.0.0.1:39888 > [04/Apr/2017:22:46:38.057] ws-local servers/server1 1/63/60001 0 0 cD > 0/0/0/0/0 0/0 > Apr 5 08:44:55 app055 haproxy[13666]: 127.0.0.1:42650 > [05/Apr/2017
low load client payload intermittently dropped with a "cD" error (v1.7.3)
I'm not sure how to interpret this, but it appears that haproxy is dropping client payload intermittently (1/100). I have included tcpdumps and logs to show what is happening. Am I doing something wrong? I have no idea what could be causing this or how to go about debugging it. I cannot reproduce it, but I do observe in production ~2 times a day across 20 instances and 2K connections. Any help or advice would be greatly appreciated. What I'm trying to accomplish is to provide HA availability over two routes (i.e. internet providers). One acts as primary and I gave it a "static-rr" "weight" of 256 and the other as backup and has a weight of "1". Backup should only be used in case of primary failure. log: Apr 4 18:55:27 app055 haproxy[13666]: 127.0.0.1:42262 [04/Apr/2017:18:54:41.585] ws-local servers/server1 1/86/45978 4503 5873 -- 0/0/0/0/0 0/0 Apr 4 22:46:37 app055 haproxy[13666]: 127.0.0.1:47130 [04/Apr/2017:22:46:36.931] ws-local servers/server1 1/62/663 7979 517 -- 0/0/0/0/0 0/0 Apr 4 22:46:38 app055 haproxy[13666]: 127.0.0.1:32931 [04/Apr/2017:22:46:37.698] ws-local servers/server1 1/55/405 3062 553 -- 1/1/1/1/0 0/0 Apr 4 22:46:43 app055 haproxy[13666]: 127.0.0.1:41748 [04/Apr/2017:22:46:43.190] ws-local servers/server1 1/115/452 7979 517 -- 2/2/2/2/0 0/0 Apr 4 22:46:46 app055 haproxy[13666]: 127.0.0.1:57226 [04/Apr/2017:22:46:43.576] ws-local servers/server1 1/76/3066 2921 538 -- 1/1/1/1/0 0/0 Apr 4 22:46:47 app055 haproxy[13666]: 127.0.0.1:39656 [04/Apr/2017:22:46:47.072] ws-local servers/server1 1/67/460 8254 528 -- 1/1/1/1/0 0/0 Apr 4 22:47:38 app055 haproxy[13666]: 127.0.0.1:39888 [04/Apr/2017:22:46:38.057] ws-local servers/server1 1/63/60001 0 0 cD 0/0/0/0/0 0/0 Apr 5 08:44:55 app055 haproxy[13666]: 127.0.0.1:42650 [05/Apr/2017:08:44:05.529] ws-local servers/server1 1/53/49645 4364 4113 -- 0/0/0/0/0 0/0 tcpdump: 22:46:38.057127 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [S], seq 2113072542, win 43690, options [mss 65495,sackOK,TS val 82055529 ecr 0,nop,wscale 7], length 0 22:46:38.057156 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [S.], seq 3284611992, ack 2113072543, win 43690, options [mss 65495,sackOK,TS val 82055529 ecr 82055529,nop,wscale 7], length 0 22:46:38.057178 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [.], ack 1, win 342, options [nop,nop,TS val 82055529 ecr 82055529], length 0 22:46:38.057295 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [S], seq 35567, win 29200, options [mss 1460,sackOK,TS val 82055529 ecr 0,nop,wscale 7], length 0 22:46:38.060539 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [P.], seq 1:199, ack 1, win 342, options [nop,nop,TS val 82055530 ecr 82055529], length 198 22:46:38.060598 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [.], ack 199, win 350, options [nop,nop,TS val 82055530 ecr 82055530], length 0 ... client payload acked ... 22:46:38.120527 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [S.], seq 4125907118, ack 35568, win 28960, options [mss 1460,sackOK,TS val 662461622 ecr 82055529,nop,wscale 8], length 0 22:46:38.120619 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [.], ack 1, win 229, options [nop,nop,TS val 82055545 ecr 662461622], length 0 ... idle timeout by server 5 seconds later... 22:46:43.183207 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [F.], seq 1, ack 1, win 114, options [nop,nop,TS val 662466683 ecr 82055545], length 0 22:46:43.183387 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [F.], seq 1, ack 199, win 350, options [nop,nop,TS val 82056810 ecr 82055530], length 0 22:46:43.184011 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [.], ack 2, win 229, options [nop,nop,TS val 82056811 ecr 662466683], length 0 22:46:43.184025 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [.], ack 2, win 342, options [nop,nop,TS val 82056811 ecr 82056810], length 0 22:46:43.184715 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [P.], seq 199:206, ack 2, win 342, options [nop,nop,TS val 82056811 ecr 82056810], length 7 22:46:43.184795 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [.], ack 206, win 350, options [nop,nop,TS val 82056811 ecr 82056811], length 0 22:46:43.184849 IP 127.0.0.1.39888 > 127.0.0.1.9011: Flags [F.], seq 206, ack 2, win 342, options [nop,nop,TS val 82056811 ecr 82056811], length 0 22:46:43.184877 IP 127.0.0.1.9011 > 127.0.0.1.39888: Flags [.], ack 207, win 350, options [nop,nop,TS val 82056811 ecr 82056811], length 0 22:47:38.058683 IP 10.10.10.10.34289 > 99.99.99.99.8000: Flags [F.], seq 1, ack 2, win 229, options [nop,nop,TS val 82070529 ecr 662466683], length 0 22:47:38.116336 IP 99.99.99.99.8000 > 10.10.10.10.34289: Flags [R], seq 4125907120, win 0, length 0 config: global daemon maxconn 10 log /dev/log local0 stats socket /dev/shm/haproxy.sock mode 666 level admin defaults log global option tcplog log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %U %ts %ac/%fc/%bc/%sc/%rc %sq/%bq" option log-health-checks option redispatch mode tcp retries 3 timeout check 900ms time
Re: Interest in patch for web interface to enable/disable servers
Hi All,
Is this something that is potentially going to make it in to mainline
haproxy? Also Cyril, is there an updated patch around anywhere? I think this
provides some interesting functionality.
Cheers,
Linc.
On 28/07/2010, at 12:44 AM, Cyril Bonté wrote:
> Hi all,
>
> Le mardi 27 juillet 2010 18:03:41, Judd Montgomery a écrit :
>> I like the patch. It works fine in lynx, which is important to
>> me for remote access and should be scriptable with lynx, curl, and
>> wget.
>
> Nice !
> Well, there was still a little bug which disturbs the tables when using a
> proxy with backend capabilities and no server. I should know that last minute
> change equals last minute bug :-)
>
>> Let me know if you need help with the admin level security.
>
> I took time to work on it tonight.
> Tests, feedbacks and code review are welcome !
>
> To test the code, apply the patch in attachment after the one sent previously.
>
> Then, by default the admin capabilities becomes disabled.
> To enable the admin level, you must specify a new "stats admin {if | unless}
> " keyword in the configuration (not allowed in defaults to prevent
> security issues)
>
> Example 1 :
> listen stats :80
>mode http
>option httpclose
>
>stats enable
>
>acl LOCAL src 127.0.0.1
>stats admin if LOCAL
>
> Example 2 :
> listen stats :80
>mode http
>option httpclose
>
>stats enable
>stats auth admin:admin
>
># always enable admin level because an authentication is required
>stats admin if TRUE
>
> --
> Cyril Bonté
>
Hot reconfig briefly stops listening?
Hi, I'm testing haproxy 1.3.18 on ubuntu 9.10, and I'm having an issue with hot reconfiguration - it seems to refuse new connections for a very short period of time (around 50ms it seems). The command I'm using is /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) , and the haproxy config can be seen at http://gist.github.com/362959 . I've also tried building 1.4.3 from source, and I'm seeing the same issue there. I'm using ab to generate the load, and the load is both direct and via nginx running as a proxy - it shows up in both places. Is this expected behaviour, or am I doing something wrong / is something wrong? Cheers, Linc. smime.p7s Description: S/MIME cryptographic signature
Re: weird tcp syn/ack problem
Hi Willy, thanks for all your help with this issue. I upgraded to ubuntu with a recent kernel and poof, the problem disappeared. Thanks, Lincoln On Thu, Dec 3, 2009 at 12:59 AM, Willy Tarreau wrote: > On Thu, Dec 03, 2009 at 12:29:55AM -0500, Lincoln wrote: > > Hi Willy, I agree it's pretty confusing. > > > > I should have been clearer - the problem does not happen every time, it's > > very random. But when it happens it always follows that exact pattern - > > that's what I meant to say. > > OK, that's what I understood first, but I wanted confirmation. > > > I actually have somaxconn set to 1 so I don't think that's the issue. > > indeed. > > > At this point I'm thinking about scrapping my EC2 instances and trying 2 > new > > ones - you never know. > > One large site I know about had problems with some instances that were > a lot slower than others, and looked like they were randomly losing a > lot of packets (probably sharing the same machine as others saturating > the bandwidth). When they switched to other instances, they discovered > that some of them were immediately receiving attacks, most likely > because they were abandonned by sites being attacked. It seems like > what works well is already used and what you can find unused is probably > bad... This site finally moved off there to solve their problems, which > were undebugable in virtualized environments. > > > Just in case you have any other insights here's the output from the 3 > > commands you mentioned. Thanks again for all your help! > > > > Lincoln > > > > r...@lb1:~$ uname -a > > Linux domU-12-31-39-0A-92-72 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 > > EST 2008 i686 i686 i386 GNU/Linux > > I don't know if it's the latest Xen kernel available, but 2.6.21 does not > sound like on of the best kernels to me, so maybe that can explain things, > though I'm not specifically aware of issues in it. Don't you have anything > more recent for these boxes ? This kernel was built almost 2 years ago, and > given the number of critical security vulnerabilities since, there must > have been updates. > > > r...@lb1:~$ netstat -i > > Kernel Interface table > > Iface MTU MetRX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP > > TX-OVR Flg > > eth0 1500 0 67999261 0 0 0 70299595 0 0 > > 0 BMRU > > lo16436 0 8045554 0 0 0 8045554 0 0 > > 0 LRU > > OK no drop here. > > > r...@lb1:~$ netstat -s > > Tcp: > > 15400091 active connections openings > > 1500044 passive connection openings > > 2110125 failed connection attempts > > Is it expected that you have that many failed connection > attempts ? Maybe one of your servers is down and it's just > the health checks count, but it looks large for a health > check. It's possible that we have the same problem on both > sides. > > > TcpExt: > > 2722 invalid SYN cookies received > > Do you have SYN cookies enabled ? If so, could you try disabling > them ? > > > 1922 resets received for embryonic SYN_RECV sockets > > 712136 TCP sockets finished time wait in fast timer > > That sounds a lot, how many connections per second do you get > in average ? And from a same IP address ? > > > 39530 passive connections rejected because of time stamp > > Troubling ! Looks like what you're experiencing. I don't know > under what condition it can happen. Maybe the sender's clock > is going backwards when it reuses a same connection ? > > Willy > >
Re: weird tcp syn/ack problem
Hi Willy, I agree it's pretty confusing. I should have been clearer - the problem does not happen every time, it's very random. But when it happens it always follows that exact pattern - that's what I meant to say. I actually have somaxconn set to 1 so I don't think that's the issue. At this point I'm thinking about scrapping my EC2 instances and trying 2 new ones - you never know. Just in case you have any other insights here's the output from the 3 commands you mentioned. Thanks again for all your help! Lincoln r...@lb1:~$ uname -a Linux domU-12-31-39-0A-92-72 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 i686 i386 GNU/Linux r...@lb1:~$ netstat -i Kernel Interface table Iface MTU MetRX-OK RX-ERR RX-DRP RX-OVRTX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 67999261 0 0 0 70299595 0 0 0 BMRU lo16436 0 8045554 0 0 0 8045554 0 0 0 LRU r...@lb1:~$ netstat -s Ip: 76004137 total packets received 2 with invalid addresses 0 forwarded 0 incoming packets discarded 76004135 incoming packets delivered 78424996 requests sent out Icmp: 1700441 ICMP messages received 6 input ICMP message failed. ICMP input histogram: destination unreachable: 1485599 echo requests: 74856 echo replies: 139986 1559234 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 1484378 echo replies: 74856 Tcp: 15400091 active connections openings 1500044 passive connection openings 2110125 failed connection attempts 646 connection resets received 1 connections established 72811429 segments received 74607063 segments send out 56735 segments retransmited 1510 bad segments received. 781257 resets sent Udp: 7887 packets received 1484378 packets to unknown port received. 0 packet receive errors 1781057 packets sent UdpLite: TcpExt: 2722 invalid SYN cookies received 1922 resets received for embryonic SYN_RECV sockets 712136 TCP sockets finished time wait in fast timer 22808 time wait sockets recycled by time stamp 851007 TCP sockets finished time wait in slow timer 39530 passive connections rejected because of time stamp 160 packets rejects in established connections because of timestamp 585822 delayed acks sent 23 delayed acks further delayed because of locked socket Quick ack mode was activated 6840 times 19917 packets directly queued to recvmsg prequeue. 338 packets directly received from prequeue 15636688 packets header predicted 26116808 acknowledgments not containing data received 936810 predicted acknowledgments 130 times recovered from packet loss due to fast retransmit 7603 times recovered from packet loss due to SACK data 3 bad SACKs received Detected reordering 22 times using FACK Detected reordering 6 times using SACK Detected reordering 13 times using reno fast retransmit Detected reordering 119 times using time stamp 117 congestion windows fully recovered 542 congestion windows partially recovered using Hoe heuristic TCPDSACKUndo: 43 14626 congestion windows recovered after partial ack 6847 TCP data loss events 60 timeouts after reno fast retransmit 1965 timeouts after SACK recovery 306 timeouts in loss state 12099 fast retransmits 3795 forward retransmits 9935 retransmits in slow start 23335 other TCP timeouts TCPRenoRecoveryFail: 74 739 sack retransmits failed 6890 DSACKs sent for old packets 3367 DSACKs received 19 DSACKs for out of order packets received 643 connections reset due to unexpected data 240 connections reset due to early user close 201 connections aborted due to timeout On Thu, Dec 3, 2009 at 12:16 AM, Willy Tarreau wrote: > On Wed, Dec 02, 2009 at 07:44:40PM -0500, Lincoln wrote: > > Thanks Willy for offering to help us out with this. > > > > We are running on an Amazon EC2 m1small instance which is very common for > a > > load balancer machine. > > > > I changed /proc/sys/net/ipv4/tcp_timestamps to 1 - unfortunately to no > > effect. > > OK. > > > Here are my iptables settings (nothing special here that I can see - I > > haven't modified anything): > > r...@lb1:~$ iptables -L > > Chain INPUT (policy ACCEPT) > > target prot opt source destination > > > > Chain FORWARD (policy ACCEPT) > > target prot opt source destination > > > > Chain OUTPUT (policy ACCEPT) > > target prot opt source destination > > OK so most likely it was not even loaded. > > > I would like to try accepting INVALIDs as you suggest - just to see if > that > > add
Re: weird tcp syn/ack problem
Thanks Willy for offering to help us out with this. We are running on an Amazon EC2 m1small instance which is very common for a load balancer machine. I changed /proc/sys/net/ipv4/tcp_timestamps to 1 - unfortunately to no effect. Here are my iptables settings (nothing special here that I can see - I haven't modified anything): r...@lb1:~$ iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination I would like to try accepting INVALIDs as you suggest - just to see if that addresses the problem before digging deeper. Unfortunately I'm not very familiar with iptables - could you show me what I should run to try that? If not that, perhaps something else about the EC2 infrastructure is using sequence number randomization? Are there other things I can look for? Thanks again, Lincoln On Wed, Dec 2, 2009 at 5:42 PM, Willy Tarreau wrote: > Hi, > > On Wed, Dec 02, 2009 at 04:47:01PM -0500, Lincoln wrote: > > Hi, I'm running HAProxy as my load balancer and sometimes (but not all > the > > time) clients experience an 11s delay. The delay is always about 11s > when > > it happens. > > > > I used Wireshark to try and see what was happening (screenshot from the > > capture on the haproxy box attached). > > > > As you can see SYNs are retried over and over but not ACKed until for > some > > reason WS, TSV, and TSER are not passed in the request. When this > happens > > it always happens the same number of times and always takes the same > > duration before acknowledgement happens. > > > > Here is my config file (it's worth mentioning that 7 of the 10 aux > servers > > are not in rotation - don't think that has anything to do with anything). > > > > Any ideas on what is going on here? I'm a relative novice at tuning tcp > (my > > tuning script is at the very bottom of this email). Also, this happens > > regardless of whether there is any load on our site. > > well, what you show is rather surprizing. It has nothing to do with your > haproxy config since haproxy only comes into play when the system receives > the ACK from the client. However, I find the case interesting enough to > investigate it. > > The fact that after some time the SYN is retransmitted without any option > is caused by the client which finally attempts to remove some of its > options > (window scaling and timestamps) in the hope that it will finally establish > (and it was right to try because it worked). > > The question is why doesn't the system acknowledge thoe SYN packets ? > > I see nothing wrong in your network settings BTW. > > Hmmm, I have an idea. Would you happen to run your haproxy machine > behind a firewall which does sequence number randomization (typically > a cisco PIX, FWSM or an OpenBSD firewall) ? If so, what generally > happens is that when a source port is reused early by a client and > the session is still present in your system's table in TIME_WAIT > state, then the second session's sequence number is random and can > be less than the last sequence number of the previous session from > the same port. The packet is then considered as invalid by the local > TCP stack (which conforms to RFC793), but at least your stack should > send an ACK back to the client (instead of a SYN/ACK), to indicate > what sequence number it expects. The client can then send an RST and > retry with a new SYN which will get accepted. > > Since we don't see this ACK, it means that either something is > preventing the SYN from reaching the stack or something is preventing > the SYN/ACK from going out. > > Are you running iptables on this machine ? If so, maybe your rules > are too strict and either the faulty SYN or the SYN/ACK can't pass > through. Check if you have a rule matching state INVALID and check > the counter. From memories, I used to accept INVALID states on SYNs > and a few other combinations to permit such situations to work, but > it was in old times, so things might have changed since. > > I see that your local system has tcp_timetamps disabled. Normally, > it's the workaround for random sequence numbers. You just enable > timestamps and the local stack applies the PAWS algorithm to > correctly identify that the new packet is not from the old session. > Netfilter also supports it, but I think it will only work for it > if your local system emits timestamps. You can do that by echoing > 1 into /proc/sys/net/ipv4/tcp_timestamps. It may very well fix the > issue. > > Please try that and keep us informed. And if your provider runs a > firewall as described above, please tell them that their option is > broken. At least on cisco it can be disabled on a per-rule basis > now :-) > > Regards, > Willy > >
Re: weird tcp syn/ack problem
I should also mention that the lb is an m1small EC2 instance. On Wed, Dec 2, 2009 at 4:47 PM, Lincoln wrote: > Hi, I'm running HAProxy as my load balancer and sometimes (but not all the > time) clients experience an 11s delay. The delay is always about 11s when > it happens. > > I used Wireshark to try and see what was happening (screenshot from the > capture on the haproxy box attached). > > As you can see SYNs are retried over and over but not ACKed until for some > reason WS, TSV, and TSER are not passed in the request. When this happens > it always happens the same number of times and always takes the same > duration before acknowledgement happens. > > Here is my config file (it's worth mentioning that 7 of the 10 aux servers > are not in rotation - don't think that has anything to do with anything). > > Any ideas on what is going on here? I'm a relative novice at tuning tcp > (my tuning script is at the very bottom of this email). Also, this happens > regardless of whether there is any load on our site. > > Thanks, > Lincoln > > Config file-- > > global > log 127.0.0.1 local0 > log 127.0.0.1 local1 notice > > maxconn 5 > ulimit-n 15 > chroot /var/lib/haproxy > user haproxy > group haproxy > daemon > > nbproc 1 # Number of processing cores. Dual Dual-core Opteron is 4 cores > for example. > > defaults > log global > mode http > option httplog > option dontlognull > retries 3 > option redispatch > option forwardfor > option httpclose > timeout check 2s > timeout client 5s > timeout server 5s > timeout connect 2s > timeout http-request 4s > timeout queue 5s > > frontend www *:80 > mode http > acl events.hotpotato.com hdr_end(host) -i events.hotpotato.com > acl go_to_engine path_beg /engine/ > use_backend events_server if events.hotpotato.com > use_backend events_server if go_to_engine > default_backend nginxunicorn > > backend events_server > mode http > fullconn 12000 > option httpchk HEAD /hcheck > reqirep ^([^\ ]*)\ /engine/(.*)\1\ /\2 > server leftevents left.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server rightevents right.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server arbiterevents arbiter.int.hotpotato.com:5050 minconn 50 maxconn > 1000 check > server aux01events aux01.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux02events aux02.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux03events aux03.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux04events aux04.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux05events aux05.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux06events aux06.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux07events aux07.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux08events aux08.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux09events aux09.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > server aux10events aux10.int.hotpotato.com:5050 minconn 50 maxconn 1000 > check > > backend nginxunicorn > mode http > fullconn 6000 > option httpchk HEAD /check > server left left.int.hotpotato.com:81 minconn 25 maxconn 500 check > server right right.int.hotpotato.com:81 minconn 25 maxconn 500 check > server arbiter arbiter.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux01 aux01.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux02 aux02.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux03 aux03.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux04 aux04.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux05 aux05.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux06 aux06.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux07 aux07.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux08 aux08.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux09 aux09.int.hotpotato.com:81 minconn 25 maxconn 500 check > server aux10 aux10.int.hotpotato.com:81 minconn 25 maxconn 500 check > > > > TCP Tuning script - this runs every time the box comes up: > > echo 3000 > /proc/sys/net/core/netdev_max_backlog > echo 1 > /proc/sys/net/core/somaxconn > echo 1024 65023 > /proc/sys/net/ipv4/ip_local_port_range > echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout > echo 15 > /proc/sys/net/ipv4/tcp_keepalive_intvl > echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes > echo 40 > /proc/sys/net/ipv4/tcp_max_tw_buckets > echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle > echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse > echo 10240 > /proc/sys/net/ipv4/tcp_max_syn_backlog > echo 6 > /proc/sys/net/ipv4/tcp_max_orphans > echo 2 > /proc/sys/net/ipv4/tcp_synack_retries >
subscribe
subscribe
