Hi, we recently switched our squid server from a OBSD server on VMware a
Centos server on XEN but there appears to be an issue somewhere between
the centos server and our OBSD Routers (DMZ) or our external OBSD firewalls.
If I log into the Centos server and run either wget or curl to an
exnternal http server I get a kind of random 1 in 3 chance or it working
or taking upto 30 seconds to complete. I've run tcpdump on the Centos
box and on the router and have imported the results into wireshare and
they both show lots of TCP Dup ACK's as shown below.
We don't have any issues with any of our other servers that are also on
the same lan as this squid server so I think it's either a Centos,
Centos/Xen, or a OBSD issue. does anyone have any ideas what might be
going on here ?
This dump was captured on our OBSD router.
No. Time Source Destination Protocol Length Info
3917 2.797310 10.0.0.X 20.0.0.X TCP 74
35247 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1
TSval=2936085 TSecr=0 WS=64
3922 2.799411 10.0.0.X 20.0.0.X TCP 66
35247 > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2936087 TSecr=0
3923 2.799543 10.0.0.X 20.0.0.X HTTP 175
GET / HTTP/1.0
3926 2.801331 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 3923#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936089 TSecr=0
3927 2.801333 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 3923#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936089 TSecr=0
3930 2.802423 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 3923#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936090 TSecr=0
3931 2.802425 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 3923#4] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936090 TSecr=0
4140 3.002585 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
4142 3.003391 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 4140#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936291 TSecr=0
4663 3.410632 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
4665 3.411451 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 4663#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2936699 TSecr=0
5538 4.226611 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
5541 4.227445 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 5538#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2937515 TSecr=0
9846 5.843961 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 5538#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2939132 TSecr=0
9851 5.844811 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 5538#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2939133 TSecr=0
9861 5.858633 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
9863 5.859432 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 9861#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2939147 TSecr=0
14821 9.122718 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
14823 9.123526 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 14821#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2942411 TSecr=0
17858 11.859699 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 14821#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2945148 TSecr=0
17863 11.860531 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 14821#3] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2945148 TSecr=0
25393 15.650790 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
25395 15.651626 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 25393#1] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2948939 TSecr=0
45327 23.890899 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 25393#2] 35247 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2957178 TSecr=0
48330 25.906963 10.0.0.X 20.0.0.X TCP 74
35248 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1
TSval=2959194 TSecr=0 WS=64
48337 25.908983 10.0.0.X 20.0.0.X TCP 66
35248 > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2959197 TSecr=0
48338 25.909077 10.0.0.X 20.0.0.X HTTP 175
GET / HTTP/1.0
48342 25.911184 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 48338#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959199 TSecr=0
48343 25.911186 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 48338#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959199 TSecr=0
48346 25.912272 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 48338#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959200 TSecr=0
48347 25.912274 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 48338#4] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959200 TSecr=0
48788 26.112919 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
48794 26.113718 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 48788#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959401 TSecr=0
49385 26.520920 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
49387 26.521745 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 49385#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2959809 TSecr=0
50594 27.336952 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
50596 27.337765 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 50594#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2960625 TSecr=0
52574 28.921899 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 50594#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2962210 TSecr=0
52576 28.922743 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 50594#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2962210 TSecr=0
52639 28.968964 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
52641 28.969752 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 52639#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2962257 TSecr=0
55547 32.233026 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
55549 32.233851 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 55547#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2965521 TSecr=0
59833 34.937494 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 55547#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2968225 TSecr=0
59835 34.938503 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 55547#3] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2968226 TSecr=0
66466 38.761131 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
66468 38.761969 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 66466#1] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2972049 TSecr=0
82253 46.859463 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 66466#2] 35248 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2980147 TSecr=0
97032 51.906615 10.0.0.X 20.0.0.X TCP 74
35249 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1
TSval=2985194 TSecr=0 WS=64
97034 51.908763 10.0.0.X 20.0.0.X TCP 66
35249 > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=2985196 TSecr=0
97035 51.908849 10.0.0.X 20.0.0.X HTTP 175
GET / HTTP/1.0
97038 51.910955 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97035#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985198 TSecr=0
97039 51.910957 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97035#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985198 TSecr=0
97042 51.912054 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97035#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985199 TSecr=0
97043 51.912056 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97035#4] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985199 TSecr=0
97301 52.112305 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
97304 52.113105 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97301#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985400 TSecr=0
97833 52.520290 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
97835 52.521102 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 97833#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2985808 TSecr=0
99547 53.336306 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
99549 53.337113 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 99547#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2986624 TSecr=0
103240 54.952961 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 99547#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2988240 TSecr=0
103242 54.953807 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 99547#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2988241 TSecr=0
103256 54.968334 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
103260 54.969125 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 103256#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2988256 TSecr=0
109179 58.232391 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
109181 58.233424 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 109179#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2991521 TSecr=0
114534 60.968474 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 109179#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2994256 TSecr=0
114536 60.969325 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 109179#3] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2994256 TSecr=0
122863 64.760491 10.0.0.X 20.0.0.X HTTP 175
[TCP Retransmission] GET / HTTP/1.0
122865 64.761316 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 122863#1] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=2998048 TSecr=0
140956 72.999821 10.0.0.X 20.0.0.X TCP 66
[TCP Dup ACK 122863#2] 35249 > http [ACK] Seq=110 Ack=1 Win=14656 Len=0
TSval=3006287 TSecr=0
149188 77.906440 10.0.0.X 20.0.0.X TCP 74
35250 > http [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1
TSval=3011193 TSecr=0 WS=64
149190 77.908726 10.0.0.X 20.0.0.X TCP 66
35250 > http [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=3011196 TSecr=0
149191 77.908820 10.0.0.X 20.0.0.X HTTP 175
GET / HTTP/1.0
149194 77.910921 10.0.0.X 20.0.0.X TCP 66
35250 > http [ACK] Seq=110 Ack=1449 Win=17536 Len=0 TSval=3011198
TSecr=1465368908
149195 77.910923 10.0.0.X 20.0.0.X TCP 66
35250 > http [ACK] Seq=110 Ack=1794 Win=20416 Len=0 TSval=3011198
TSecr=1465368908
149196 77.912997 10.0.0.X 20.0.0.X TCP 66
35250 > http [FIN, ACK] Seq=110 Ack=1794 Win=20416 Len=0 TSval=3011200
TSecr=1465368908
149199 77.914014 10.0.0.X 20.0.0.X TCP 66
35250 > http [ACK] Seq=111 Ack=1795 Win=20416 Len=0 TSval=3011201
TSecr=1465368908
Cheers
Keith
