Hi, all. virtio-net has a sudden high latency when receiving. i think it is irq related, but i have no idea to fix it. centos 3.10.0-693.21.1.el7.x86_64 ; qemu-kvm-ev-2.6.0-28.el7_3.9.1.x86_64 ; physical machine has 40 cpus, the two vms are on the same node, cpu load is very low.
1) pic1: inside vm1(192.168.91.4) , ping 192.168.91.5; please notice icmp seq 20 [1577252456.600426] 64 bytes from 192.168.91.5: icmp_seq=12 ttl=64 time=0.692 ms [1577252457.600514] 64 bytes from 192.168.91.5: icmp_seq=13 ttl=64 time=0.462 ms [1577252458.600590] 64 bytes from 192.168.91.5: icmp_seq=14 ttl=64 time=0.396 ms [1577252459.600922] 64 bytes from 192.168.91.5: icmp_seq=15 ttl=64 time=0.888 ms [1577252460.600445] 64 bytes from 192.168.91.5: icmp_seq=16 ttl=64 time=0.355 ms [1577252461.601328] 64 bytes from 192.168.91.5: icmp_seq=17 ttl=64 time=0.311 ms [1577252462.600449] 64 bytes from 192.168.91.5: icmp_seq=18 ttl=64 time=0.424 ms [1577252463.601725] 64 bytes from 192.168.91.5: icmp_seq=19 ttl=64 time=0.322 ms [1577252464.606752] 64 bytes from 192.168.91.5: icmp_seq=20 ttl=64 time=5.65 ms [1577252465.603569] 64 bytes from 192.168.91.5: icmp_seq=21 ttl=64 time=0.438 ms [1577252466.603679] 64 bytes from 192.168.91.5: icmp_seq=22 ttl=64 time=0.494 ms [1577252467.603545] 64 bytes from 192.168.91.5: icmp_seq=23 ttl=64 time=0.431 ms tcpdump inside vm1 ; 13:41:04.601016 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64 13:41:04.606647 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64 2) capture packet on its backend tap device on hypervisor ; tcpdump -nn -i tap5803c575-4a icmp and host 192.168.91.5 and host 192.168.91.4 13:41:04.601166 IP 192.168.91.4 > 192.168.91.5: ICMP echo request, id 18375, seq 20, length 64 13:41:04.601362 IP 192.168.91.5 > 192.168.91.4: ICMP echo reply, id 18375, seq 20, length 64 it shows tap takes no much time between send and receive. 3) 606647 - 601362 = 5.3ms , this is consumed by virtio-net. similarly, when 192.168.91.5 receive packets from its backend, it also has sudden high latency. do you all have any ideas? thanks