On 2020/10/30 1:20, Vishwanath Pai wrote: > On 10/29/20 6:24 AM, Yunsheng Lin wrote: >> On 2020/10/29 12:50, Vishwanath Pai wrote: >>> On 10/28/20 10:37 PM, Yunsheng Lin wrote: >>>> On 2020/10/29 4:04, Vishwanath Pai wrote: >>>>> On 10/28/20 1:47 PM, Cong Wang wrote: >>>>>> On Wed, Oct 28, 2020 at 8:37 AM Pai, Vishwanath <v...@akamai.com> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> We noticed some problems when testing the latest 5.4 LTS kernel and >>>>>>> traced it >>>>>>> back to this commit using git bisect. When running our tests the >>>>>>> machine stops >>>>>>> responding to all traffic and the only way to recover is a reboot. I do >>>>>>> not see >>>>>>> a stack trace on the console. >>>>>> >>>>>> Do you mean the machine is still running fine just the network is down? >>>>>> >>>>>> If so, can you dump your tc config with stats when the problem is >>>>>> happening? >>>>>> (You can use `tc -s -d qd show ...`.) >>>>>> >>>>>>> >>>>>>> This can be reproduced using the packetdrill test below, it should be >>>>>>> run a >>>>>>> few times or in a loop. You should hit this issue within a few tries but >>>>>>> sometimes might take up to 15-20 tries. >>>>>> ... >>>>>>> I can reproduce the issue easily on v5.4.68, and after reverting this >>>>>>> commit it >>>>>>> does not happen anymore. >>>>>> >>>>>> This is odd. The patch in this thread touches netdev reset path, if >>>>>> packetdrill >>>>>> is the only thing you use to trigger the bug (that is netdev is always >>>>>> active), >>>>>> I can not connect them. >>>>>> >>>>>> Thanks. >>>>> >>>>> Hi Cong, >>>>> >>>>>> Do you mean the machine is still running fine just the network is down? >>>>> >>>>> I was able to access the machine via serial console, it looks like it is >>>>> up and running, just that networking is down. >>>>> >>>>>> If so, can you dump your tc config with stats when the problem is >>>>>> happening? >>>>>> (You can use `tc -s -d qd show ...`.) >>>>> >>>>> If I try running tc when the machine is in this state the command never >>>>> returns. It doesn't print anything but doesn't exit either. >>>>> >>>>>> This is odd. The patch in this thread touches netdev reset path, if >>>>>> packetdrill >>>>>> is the only thing you use to trigger the bug (that is netdev is always >>>>>> active), >>>>>> I can not connect them. >>>>> >>>>> I think packetdrill creates a tun0 interface when it starts the >>>>> test and tears it down at the end, so it might be hitting this code path >>>>> during teardown. >>>> >>>> Hi, Is there any preparation setup before running the above packetdrill >>>> test >>>> case, I run the above test case in 5.9-rc4 with this patch applied without >>>> any >>>> preparation setup, did not reproduce it. >>>> >>>> By the way, I am newbie to packetdrill:), it would be good to provide the >>>> detail setup to reproduce it,thanks. >>>> >>>>> >>>>> P.S: My mail server is having connectivity issues with vger.kernel.org >>>>> so messages aren't getting delivered to netdev. It'll hopefully get >>>>> resolved soon. >>>>> >>>>> Thanks, >>>>> Vishwanath >>>>> >>>>> >>>>> . >>>>> >>> >>> I can't reproduce it on v5.9-rc4 either, it is probably an issue only on >>> 5.4 then (and maybe older LTS versions). Can you give it a try on >>> 5.4.68? >>> >>> For running packetdrill, download the latest version from their github >>> repo, then run it in a loop without any special arguments. This is what >>> I do to reproduce it: >>> >>> while true; do ./packetdrill <test-file>; done >>> >>> I don't think any other setup is necessary. >> >> Hi, run the above test for above an hour using 5.4.68, still did not >> reproduce it, as below: >> >> >> root@(none)$ cd /home/root/ >> root@(none)$ ls >> creat_vlan.sh packetdrill test.pd >> root@(none)$ cat test.pd >> 0 `echo 4 > /proc/sys/net/ipv4/tcp_min_tso_segs` >> >> 0.400 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 >> 0.400 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 >> >> // set maxseg to 1000 to work with both ipv4 and ipv6 >> 0.500 setsockopt(3, SOL_TCP, TCP_MAXSEG, [1000], 4) = 0 >> 0.500 bind(3, ..., ...) = 0 >> 0.500 listen(3, 1) = 0 >> >> // Establish connection >> 0.600 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 5> >> 0.600 > S. 0:0(0) ack 1 <...> >> >> 0.800 < . 1:1(0) ack 1 win 320 >> 0.800 accept(3, ..., ...) = 4 >> >> // Send 4 data segments. >> +0 write(4, ..., 4000) = 4000 >> +0 > P. 1:4001(4000) ack 1 >> >> // Receive a SACK >> +.1 < . 1:1(0) ack 1 win 320 <sack 1001:2001,nop,nop> >> >> +.3 %{ print "TCP CA state: ",tcpi_ca_state }% >> root@(none)$ cat creat_vlan.sh >> #!/bin/sh >> >> for((i=0; i<10000; i++)) >> do >> ./packetdrill test.pd >> done >> root@(none)$ ./creat_vlan.sh >> TCP CA state: 3 >> ^C >> root@(none)$ ifconfig >> eth0 Link encap:Ethernet HWaddr 5c:e8:83:0d:f7:ed >> inet addr:192.168.1.93 Bcast:192.168.1.255 Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:3570 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:3190 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:1076349 (1.0 MiB) TX bytes:414874 (405.1 KiB) >> >> eth2 Link encap:Ethernet HWaddr 5c:e8:83:0d:f7:ec >> inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:81848576 errors:0 dropped:0 overruns:0 frame:78 >> TX packets:72497816 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:2044282289568 (1.8 TiB) TX bytes:2457441698852 (2.2 TiB) >> >> lo Link encap:Local Loopback >> inet addr:127.0.0.1 Mask:255.0.0.0 >> UP LOOPBACK RUNNING MTU:65536 Metric:1 >> RX packets:1 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:1 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:68 (68.0 B) TX bytes:68 (68.0 B) >> >> root@(none)$ ./creat_vlan.sh >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> TCP CA state: 3 >> ^C >> root@(none)$ cat /proc/cmdline >> BOOT_IMAGE=/linyunsheng/Image.5.0 rdinit=/init console=ttyAMA0,115200 >> earlycon=pl011,mmio32,0x94080000 iommu.strict=1 >> root@(none)$ cat /proc/version >> Linux version 5.4.68 (linyunsheng@ubuntu) (gcc version 5.4.0 20160609 >> (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12)) #1 SMP PREEMPT Thu Oct 29 16:59:37 >> CST 2020 >> root@(none)$ >> >> >> >>> >>> -Vishwanath >>> >>> . >>> > I couldn't get it to reproduce on a ubuntu VM, maybe something is > different with the way we setup our machines. We do have some scripts in > /etc/network/{if-up.d,if-post-down.d} etc, or probably something else. > I'll let you know when I can reliably reproduce it on the VM.
Hi, Vishwanath Please see if the patch in the below link fix your problem, thanks. https://www.spinics.net/lists/netdev/msg695908.html > > . >