On 26.11.11 21:50, Joseph Mack NA3T wrote: > On Sat, 26 Nov 2011, Alois Treindl wrote: > >> The httpd real server introduces 200 millisecond delays >> between packets when sending. > ^^^^^^^ > I assume you mean "replying"
yes. replying to the client's http request, trasnmitting data to the client > > Whenever I get this it's something like > > o the routing is wrong (are you getting a shower of icmp > packets?) how would I see them with tcpdump? > o the httpd is phoning home (asking for a service on the > internet eg DNS) that has to time out turned this off, I think. httpd is logging without using DNS, nothing is running on real servers except the rare requests from my LVS test access. As I said, I can remove the LVS router and can connect a cable directly and then I get high performance. I try to look into TCP_NODELAY because the 200 msec point in that direction. Your help is really appreciated. I have tried to compile ipvsadm 1.26 but it has too many problems compiling on my RHEL6. I have to stick with the CentOs 6 ipvsadm 1.25.9 I begin to despair because I have a fixed setup date for the new cluster to keep early next week, and I cannot test it when the LVS balancer is not working. --------- Here is a fraction of tcpdump, as seen on the client, started off on the client by wget http://LVS/large_file_name as80.stro.ch is the client, lv2 the LVS 19:51:25.610063 IP as80.astro.ch.39594 > lv2.astro.com.http: Flags [S], seq 3584270897, win 5840, options [mss 1460,sackOK,TS val 432865933 ecr 0,nop,wscale 7], length 0 19:52:45.207361 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [S], seq 545625682, win 5840, options [mss 1460,sackOK,TS val 432945531 ecr 0,nop,wscale 7], length 0 19:52:45.207746 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [S.], seq 951261745, ack 545625683, win 5792, options [mss 1460,sackOK,TS val 12093210 ecr 432945531,nop,wscale 7], length 0 19:52:45.207763 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.], ack 1, win 46, options [nop,nop,TS val 432945531 ecr 12093210], length 0 19:52:45.207882 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [P.], seq 1:122, ack 1, win 46, options [nop,nop,TS val 432945532 ecr 12093210], length 121 19:52:45.208202 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], ack 122, win 46, options [nop,nop,TS val 12093211 ecr 432945532], length 0 19:52:45.209374 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 432945532], length 1448 19:52:45.209384 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 432945532], length 1448 19:52:45.209389 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 12093212 ecr 432945532], length 1448 DELAY 19:52:45.410088 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 432945532], length 1448 19:52:45.410103 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.], ack 1449, win 69, options [nop,nop,TS val 432945734 ecr 12093413], length 0 19:52:45.410482 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 432945734], length 1448 19:52:45.410492 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 12093413 ecr 432945734], length 1448 DELAY 19:52:45.611086 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.], seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093614 ecr 432945734], length 1448 ---------------- here is the traffic as seen on the LVS, internal interface to the client w1 (it is not the same file transfer, this is why the timestamps are different) 18:53:44.819241 IP as80.astro.ch.36962 > w1.http: Flags [S], seq 862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr 0,nop,wscale 7], length 0 18:53:44.819412 IP w1.http > as80.astro.ch.36962: Flags [S.], seq 1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val 8552820 ecr 429405138,nop,wscale 7], length 0 18:53:44.819505 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 0 18:53:44.819588 IP as80.astro.ch.36962 > w1.http: Flags [P.], seq 1:122, ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 121 18:53:44.819746 IP w1.http > as80.astro.ch.36962: Flags [.], ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0 18:53:44.820270 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:4345, ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 4344 DELAY 18:53:45.021411 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:1449, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405138], length 1448 18:53:45.021577 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1449, win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0 18:53:45.021757 IP w1.http > as80.astro.ch.36962: Flags [.], seq 4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405341], length 2896 DELAY 18:53:45.222410 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 429405341], length 1448 18:53:45.222593 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 2897, win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0 18:53:45.222752 IP w1.http > as80.astro.ch.36962: Flags [.], seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 429405542], length 2896 DELAY 18:53:45.423386 IP w1.http > as80.astro.ch.36962: Flags [.], seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 429405542], length 1448 18:53:45.423567 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 4345, win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0 18:53:45.423728 IP w1.http > as80.astro.ch.36962: Flags [.], seq 10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 429405742], length 2896 DELAY 18:53:45.624412 IP w1.http > as80.astro.ch.36962: Flags [.], seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 429405742], length 1448 18:53:45.624572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 5793, win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0 18:53:45.624753 IP w1.http > as80.astro.ch.36962: Flags [P.], seq 13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 429405943], length 2896 DELAY 18:53:45.825385 IP w1.http > as80.astro.ch.36962: Flags [.], seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 429405943], length 1448 18:53:45.825547 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 7241, win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0 18:53:45.825728 IP w1.http > as80.astro.ch.36962: Flags [.], seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 429406144], length 2896 DELAY 18:53:46.026413 IP w1.http > as80.astro.ch.36962: Flags [.], seq 7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr 429406144], length 1448 18:53:46.026572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 8689, win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0 ------------ same transfer, observed on LVS external interface, time stamps agree with sample 2 18:53:44.819225 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [S], seq 862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr 0,nop,wscale 7], length 0 18:53:44.819424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [S.], seq 1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val 8552820 ecr 429405138,nop,wscale 7], length 0 18:53:44.819499 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 0 18:53:44.819584 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [P.], seq 1:122, ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 121 18:53:44.819758 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0 18:53:44.820284 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 1:4345, ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 4344 DELAY 18:53:45.021424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 1:1449, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405138], length 1448 18:53:45.021567 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 1449, win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0 18:53:45.021771 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405341], length 2896 DELAY 18:53:45.222424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 429405341], length 1448 18:53:45.222583 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 2897, win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0 18:53:45.222766 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr 429405542], length 2896 DELAY 18:53:45.423399 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 429405542], length 1448 18:53:45.423557 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 4345, win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0 18:53:45.423742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr 429405742], length 2896 DELAY 18:53:45.624425 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 429405742], length 1448 18:53:45.624562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 5793, win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0 18:53:45.624915 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [P.], seq 13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr 429405943], length 2896 DELAY 18:53:45.825398 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 429405943], length 1448 18:53:45.825537 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 7241, win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0 18:53:45.825742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr 429406144], length 2896 18:53:46.026426 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.], seq 7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr 429406144], length 1448 18:53:46.026562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.], ack 8689, win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0 ---------- I also tried to look at the tcp socks involved in the transfer, with ss But I do not really understand what I see. seen on client: ss -t -i -o -e dst lv2.astro.com State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.53.104.2:51478 192.53.104.100:http uid:101 ino:22082073 sk:ffff88082b018100 ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:50 cwnd:4 send 46.3Mbps rcv_rtt:174.75 rcv_space:60816 seen on w1 real server: ss -t -i -o -e dst 192.53.104.2 ESTAB 0 36200 10.1.1.1:http 192.53.104.2:51506 timer:(on,132ms,0) uid:104 ino:37970 sk:ffff88081164f6c0 ts sack cubic wscale:7,7 rto:201 rtt:1.875/0.75 ato:40 cwnd:10 ssthresh:2 send 61.8Mbps rcv_space:5792 -------------- when I do a direct http transfer not through the LVS, (which is fast, everything is conencted with 1Gb links), then I see these tcp socks on client: ss -t -i -o -e dst 192.53.104.2 State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.53.104.2:36300 192.53.104.99:http uid:101 ino:22146399 sk:ffff880451ef4180 ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:40 cwnd:4 send 46.3Mbps rcv_rtt:1.875 rcv_space:6199088 on server: ESTAB 0 2920616 ::ffff:192.53.104.99:http ::ffff:192.53.104.2:36305 timer:(on,206ms,0) uid:104 ino:47402 sk:ffff8801261071c0 ts sack cubic wscale:7,7 rto:206 rtt:6.5/0.75 ato:40 cwnd:507 ssthresh:504 send 903.6Mbps rcv_space:5792 They seem to have the same values. _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
