Hi All, There is a subtle but important distinction between iperf 2's in progress stats. One is called InF for in-flight and computed by taking the stats from the stack. The second is called InP fo in-progress and taken by computing the queue depth per Little's Law. The latter is e2e. The following two runs illustrate the differences.
First a run where there is a very large send side buffer and TCP_NOTSENT_LOWAT is not used. In this case the send side socket buffer should build up, which it does, and the e2e delay is seen per InP but not InF. That's because InF is from the stack's view of the network while InP is from the application view. Notice the InP is near the 100Mbytes socket buffer size while the InF is about 1.5Mbytes The second run sets TCP_NOTSENT_LOWAT and no the InF and InP are aligned at about 1.5Mbytes Run 1: No TCP_NOTSENT_LOWAT rjmcmahon@fedora:~/Code/inflight/iperf2-code$ iperf -c 192.168.1.35 --trip-times -e -i 1 --sync-transfer-id -w 50m --tcp-write-prefetch 0 ------------------------------------------------------------ Client connecting to 192.168.1.35, TCP port 5001 with pid 240696 (1/0 flows/load) Write buffer size: 131072 Byte TCP congestion control using cubic TOS set to 0x0 (dscp=0,ecn=0) (Nagle on) TCP window size: 95.4 MByte (WARNING: requested 47.7 MByte) ------------------------------------------------------------ [ 1] local 192.168.1.103%enp4s0 port 44754 connected with 192.168.1.35 port 5001 (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/176) (ct=0.24 ms) on 2024-03-25 16:50:54.655 (PDT) [ ID] Interval Transfer Bandwidth Write/Err Rtry InF(pkts)/Cwnd/RTT(var) NetPwr [ 1] 0.00-1.00 sec 192 MBytes 1.61 Gbits/sec 1533/0 61 1538K(1088)/1555K/13234(110) us 15183 [ 1] 1.00-2.00 sec 96.4 MBytes 808 Mbits/sec 771/0 0 1667K(1179)/1674K/14273(81) us 7080 [ 1] 2.00-3.00 sec 128 MBytes 1.08 Gbits/sec 1026/0 0 1790K(1266)/1801K/15451(109) us 8704 [ 1] 3.00-4.00 sec 96.2 MBytes 807 Mbits/sec 770/0 0 1859K(1315)/1875K/16008(84) us 6305 [ 1] 4.00-5.00 sec 128 MBytes 1.08 Gbits/sec 1026/0 2 1340K(948)/1365K/11658(133) us 11535 [ 1] 5.00-6.00 sec 96.2 MBytes 807 Mbits/sec 770/0 0 1409K(997)/1448K/12301(98) us 8205 [ 1] 6.00-7.00 sec 128 MBytes 1.08 Gbits/sec 1026/0 0 1474K(1043)/1530K/13059(123) us 10298 [ 1] 7.00-8.00 sec 96.2 MBytes 807 Mbits/sec 770/0 0 1534K(1085)/1575K/13494(111) us 7479 [ 1] 8.00-9.00 sec 128 MBytes 1.08 Gbits/sec 1027/0 0 1602K(1133)/1614K/13807(132) us 9749 [ 1] 9.00-10.00 sec 96.2 MBytes 807 Mbits/sec 770/0 0 1593K(1127)/1636K/13980(130) us 7219 [ 1] 10.00-10.57 sec 128 KBytes 1.83 Mbits/sec 1/0 0 0K(0)/1665K/14358(111) us 15.90 [ 1] 0.00-10.57 sec 1.16 GBytes 941 Mbits/sec 9490/0 63 0K(0)/1665K/14358(111) us 8193 root@rpi5-35:~# iperf -s -i 1 -e ------------------------------------------------------------ Server listening on TCP port 5001 with pid 35250 Read buffer size: 128 KByte (Dist bin width=16.0 KByte) TCP congestion control default cubic TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.1.35%eth0 port 5001 connected with 192.168.1.103 port 44754 (trip-times) (sock=4) (peer 2.2.0-rc) (icwnd/mss/irtt=14/1448/182) on 2024-03-25 16:50:54.655 (PDT) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist [ 1] 0.00-1.00 sec 112 MBytes 941 Mbits/sec 451.474/1.878/839.327/236.465 ms (897/131093) 171 MByte 260 22019=21972:11:11:8:0:2:2:13 [ 1] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 709.032/560.182/839.576/80.886 ms (898/131052) 69.9 MByte 166 22841=22836:3:2:0:0:0:0:0 [ 1] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 690.533/559.994/839.511/79.692 ms (897/131201) 90.0 MByte 170 18997=18993:4:0:0:0:0:0:0 [ 1] 3.00-4.00 sec 112 MBytes 942 Mbits/sec 708.975/559.939/839.014/80.667 ms (898/131063) 69.8 MByte 166 19325=19325:0:0:0:0:0:0:0 [ 1] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 689.814/559.899/838.858/79.718 ms (898/131053) 90.0 MByte 171 19552=19552:0:0:0:0:0:0:0 [ 1] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 709.082/559.756/838.974/80.714 ms (898/131053) 69.8 MByte 166 19050=19030:4:0:0:1:0:1:14 [ 1] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 690.027/559.817/839.295/79.776 ms (898/131050) 90.0 MByte 171 19573=19570:2:1:0:0:0:0:0 [ 1] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 709.150/560.197/839.131/80.593 ms (898/131056) 69.8 MByte 166 20616=20608:4:3:1:0:0:0:0 [ 1] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 690.031/560.092/839.122/79.848 ms (898/131050) 90.0 MByte 171 18866=18864:2:0:0:0:0:0:0 [ 1] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 709.217/560.065/838.999/80.484 ms (898/131056) 69.8 MByte 166 18885=18882:2:1:0:0:0:0:0 [ 1] 10.00-10.57 sec 64.0 MBytes 941 Mbits/sec 699.737/560.046/838.851/80.583 ms (512/131051) 69.8 MByte 168 210924=11199:1:0:0:0:0:0:0 [ 1] 0.00-10.57 sec 1.16 GBytes 941 Mbits/sec 677.050/1.878/839.576/128.445 ms (9490/131072) 75.7 MByte 174 210924=210831:33:18:9:1:2:3:27 Run 2: With TCP_NOTSENT_LOWAT rjmcmahon@fedora:~/Code/inflight/iperf2-code$ iperf -c 192.168.1.35 --trip-times -e -i 1 --sync-transfer-id -w 50m --tcp-write-prefetch 128K ------------------------------------------------------------ Client connecting to 192.168.1.35, TCP port 5001 with pid 240730 (1/0 flows/load) Write buffer size: 131072 Byte TCP congestion control using cubic TOS set to 0x0 (dscp=0,ecn=0) (Nagle on) TCP window size: 95.4 MByte (WARNING: requested 47.7 MByte) Event based writes (pending queue watermark at 131072 bytes) ------------------------------------------------------------ [ 1] local 192.168.1.103%enp4s0 port 53192 connected with 192.168.1.35 port 5001 (prefetch=131072) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/169) (ct=0.23 ms) on 2024-03-25 16:54:02.796 (PDT) [ ID] Interval Transfer Bandwidth Write/Err Rtry InF(pkts)/Cwnd/RTT(var) NetPwr [ 1] 0.00-1.00 sec 114 MBytes 955 Mbits/sec 911/0 72 1534K(1085)/1534K/13047(100) us 9152 [ 1] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1675K(1185)/1675K/14295(114) us 8234 [ 1] 2.00-3.00 sec 112 MBytes 943 Mbits/sec 899/0 0 1783K(1261)/1788K/15220(188) us 7742 [ 1] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 899/0 0 1875K(1326)/1875K/16128(191) us 7306 [ 1] 4.00-5.00 sec 112 MBytes 937 Mbits/sec 894/0 1 1356K(959)/1356K/11166(273) us 10494 [ 1] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1450K(1026)/1450K/12334(159) us 9543 [ 1] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1524K(1078)/1524K/12974(165) us 9072 [ 1] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 899/0 0 1576K(1115)/1576K/13436(112) us 8770 [ 1] 8.00-9.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1610K(1139)/1610K/13789(157) us 8536 [ 1] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 898/0 0 1638K(1159)/1638K/14001(104) us 8407 [ 1] 0.00-10.03 sec 1.10 GBytes 940 Mbits/sec 8993/0 73 0K(0)/1638K/14017(120) us 8386 root@rpi5-35:~# iperf -s -i 1 -e ------------------------------------------------------------ Server listening on TCP port 5001 with pid 35260 Read buffer size: 128 KByte (Dist bin width=16.0 KByte) TCP congestion control default cubic TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 1] local 192.168.1.35%eth0 port 5001 connected with 192.168.1.103 port 53192 (trip-times) (sock=4) (peer 2.2.0-rc) (icwnd/mss/irtt=14/1448/171) on 2024-03-25 16:54:02.796 (PDT) [ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist [ 1] 0.00-1.00 sec 112 MBytes 941 Mbits/sec 13.100/1.372/34.740/4.362 ms (897/131165) 1.49 MByte 8981 18597=18552:13:5:6:1:3:5:12 [ 1] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 15.244/14.370/16.076/0.389 ms (898/131054) 1.71 MByte 7720 19588=19588:0:0:0:0:0:0:0 [ 1] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 16.325/15.569/17.066/0.318 ms (898/131061) 1.83 MByte 7209 19568=19568:0:0:0:0:0:0:0 [ 1] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 17.187/16.504/17.841/0.272 ms (898/131053) 1.93 MByte 6847 19317=19317:0:0:0:0:0:0:0 [ 1] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 17.979/14.707/34.753/1.359 ms (898/131050) 2.01 MByte 6546 19044=19022:3:3:1:0:0:0:15 [ 1] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 13.481/12.410/14.241/0.298 ms (898/131054) 1.51 MByte 8730 18915=18910:2:1:1:1:0:0:0 [ 1] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 14.220/13.581/14.831/0.245 ms (897/131199) 1.60 MByte 8276 19495=19491:2:2:0:0:0:0:0 [ 1] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 14.762/14.231/15.296/0.211 ms (898/131049) 1.66 MByte 7972 19164=19161:2:1:0:0:0:0:0 [ 1] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 15.144/14.693/15.589/0.180 ms (898/131052) 1.70 MByte 7771 19270=19267:2:0:1:0:0:0:0 [ 1] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 15.398/15.012/15.817/0.172 ms (898/131055) 1.73 MByte 7643 18872=18866:5:0:1:0:0:0:0 [ 1] 0.00-10.02 sec 1.10 GBytes 941 Mbits/sec 15.285/1.372/34.753/2.068 ms (8993/131072) 1.68 MByte 7699 192149=192061:29:12:10:2:3:5:27 Bob -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Iperf-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/iperf-users
