On 2020/10/29 12:50, Vishwanath Pai wrote: > On 10/28/20 10:37 PM, Yunsheng Lin wrote: >> On 2020/10/29 4:04, Vishwanath Pai wrote: >>> On 10/28/20 1:47 PM, Cong Wang wrote: >>>> On Wed, Oct 28, 2020 at 8:37 AM Pai, Vishwanath <v...@akamai.com> wrote: >>>>> Hi, >>>>> >>>>> We noticed some problems when testing the latest 5.4 LTS kernel and >>>>> traced it >>>>> back to this commit using git bisect. When running our tests the machine >>>>> stops >>>>> responding to all traffic and the only way to recover is a reboot. I do >>>>> not see >>>>> a stack trace on the console. >>>> >>>> Do you mean the machine is still running fine just the network is down? >>>> >>>> If so, can you dump your tc config with stats when the problem is >>>> happening? >>>> (You can use `tc -s -d qd show ...`.) >>>> >>>>> >>>>> This can be reproduced using the packetdrill test below, it should be run >>>>> a >>>>> few times or in a loop. You should hit this issue within a few tries but >>>>> sometimes might take up to 15-20 tries. >>>> ... >>>>> I can reproduce the issue easily on v5.4.68, and after reverting this >>>>> commit it >>>>> does not happen anymore. >>>> >>>> This is odd. The patch in this thread touches netdev reset path, if >>>> packetdrill >>>> is the only thing you use to trigger the bug (that is netdev is always >>>> active), >>>> I can not connect them. >>>> >>>> Thanks. >>> >>> Hi Cong, >>> >>>> Do you mean the machine is still running fine just the network is down? >>> >>> I was able to access the machine via serial console, it looks like it is >>> up and running, just that networking is down. >>> >>>> If so, can you dump your tc config with stats when the problem is >>>> happening? >>>> (You can use `tc -s -d qd show ...`.) >>> >>> If I try running tc when the machine is in this state the command never >>> returns. It doesn't print anything but doesn't exit either. >>> >>>> This is odd. The patch in this thread touches netdev reset path, if >>>> packetdrill >>>> is the only thing you use to trigger the bug (that is netdev is always >>>> active), >>>> I can not connect them. >>> >>> I think packetdrill creates a tun0 interface when it starts the >>> test and tears it down at the end, so it might be hitting this code path >>> during teardown. >> >> Hi, Is there any preparation setup before running the above packetdrill test >> case, I run the above test case in 5.9-rc4 with this patch applied without >> any >> preparation setup, did not reproduce it. >> >> By the way, I am newbie to packetdrill:), it would be good to provide the >> detail setup to reproduce it,thanks. >> >>> >>> P.S: My mail server is having connectivity issues with vger.kernel.org >>> so messages aren't getting delivered to netdev. It'll hopefully get >>> resolved soon. >>> >>> Thanks, >>> Vishwanath >>> >>> >>> . >>> > > I can't reproduce it on v5.9-rc4 either, it is probably an issue only on > 5.4 then (and maybe older LTS versions). Can you give it a try on > 5.4.68? > > For running packetdrill, download the latest version from their github > repo, then run it in a loop without any special arguments. This is what > I do to reproduce it: > > while true; do ./packetdrill <test-file>; done > > I don't think any other setup is necessary.
Hi, run the above test for above an hour using 5.4.68, still did not reproduce it, as below: root@(none)$ cd /home/root/ root@(none)$ ls creat_vlan.sh packetdrill test.pd root@(none)$ cat test.pd 0 `echo 4 > /proc/sys/net/ipv4/tcp_min_tso_segs` 0.400 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.400 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 // set maxseg to 1000 to work with both ipv4 and ipv6 0.500 setsockopt(3, SOL_TCP, TCP_MAXSEG, [1000], 4) = 0 0.500 bind(3, ..., ...) = 0 0.500 listen(3, 1) = 0 // Establish connection 0.600 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 5> 0.600 > S. 0:0(0) ack 1 <...> 0.800 < . 1:1(0) ack 1 win 320 0.800 accept(3, ..., ...) = 4 // Send 4 data segments. +0 write(4, ..., 4000) = 4000 +0 > P. 1:4001(4000) ack 1 // Receive a SACK +.1 < . 1:1(0) ack 1 win 320 <sack 1001:2001,nop,nop> +.3 %{ print "TCP CA state: ",tcpi_ca_state }% root@(none)$ cat creat_vlan.sh #!/bin/sh for((i=0; i<10000; i++)) do ./packetdrill test.pd done root@(none)$ ./creat_vlan.sh TCP CA state: 3 ^C root@(none)$ ifconfig eth0 Link encap:Ethernet HWaddr 5c:e8:83:0d:f7:ed inet addr:192.168.1.93 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3570 errors:0 dropped:0 overruns:0 frame:0 TX packets:3190 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1076349 (1.0 MiB) TX bytes:414874 (405.1 KiB) eth2 Link encap:Ethernet HWaddr 5c:e8:83:0d:f7:ec inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:81848576 errors:0 dropped:0 overruns:0 frame:78 TX packets:72497816 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2044282289568 (1.8 TiB) TX bytes:2457441698852 (2.2 TiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:1 errors:0 dropped:0 overruns:0 frame:0 TX packets:1 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:68 (68.0 B) TX bytes:68 (68.0 B) root@(none)$ ./creat_vlan.sh TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 TCP CA state: 3 ^C root@(none)$ cat /proc/cmdline BOOT_IMAGE=/linyunsheng/Image.5.0 rdinit=/init console=ttyAMA0,115200 earlycon=pl011,mmio32,0x94080000 iommu.strict=1 root@(none)$ cat /proc/version Linux version 5.4.68 (linyunsheng@ubuntu) (gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12)) #1 SMP PREEMPT Thu Oct 29 16:59:37 CST 2020 root@(none)$ > > -Vishwanath > > . >