On Thu, Nov 22, 2018 at 1:51 AM Samudrala, Sridhar <sridhar.samudr...@intel.com> wrote: > > On 11/21/2018 12:04 PM, Sameeh Jubran wrote: > > On Wed, Nov 21, 2018 at 8:41 PM Michael S. Tsirkin <m...@redhat.com> wrote: > >> Great to see you making progress on this! > >> Some comments below: > >> > >> On Wed, Nov 21, 2018 at 05:39:38PM +0200, Sameeh Jubran wrote: > >>> I have created a setup which has two hosts (host A and host B) with X710 > >>> 10G > >>> cards connected back to back. On one host (I'll refer to this host as > >>> host A) I > >>> have configured a bridge with the PF interface as well as vitio-net's > >>> interface > >>> (standby) both attached to it. > >> ... > >> > >>> The command line I used: > >>> > >>> /root/qemu/x86_64-softmmu/qemu-system-x86_64 \ > >>> -netdev > >>> tap,id=hostnet0,script=world_bridge_standalone.sh,downscript=no,ifname= > >>> cc17 \ > >>> -device e1000,netdev=hostnet0,mac=56:cc:c1:01:cc:21,id=cc17 \ > >> What's e1000 doing here? > >> Can this be reason you can not talk to host? > > I don't think so, the e1000 is for enabling WAN connection on the > > guest for downloading packages and ssh connection. It is connected to > > a separate bridge which is connected to the external interface of the > > host. > >>> -netdev > >>> tap,vhost=on,id=hostnet1,script=test_bridge_standalone.sh,downscript= > >>> no,ifname=cc1_72,queues=4 \ > >>> -device virtio-net,host_mtu=1500,netdev=hostnet1,mac=8a:f7:20:29:3b:cb,id= > >>> cc1_72,vectors=10,mq=on,primary=cc1_71 \ > >>> -device vfio-pci,host=65:02.1,id=cc1_71,standby=cc1_72 \ > >>> -enable-kvm \ > >>> -name netkvm \ > >>> -m 3000M \ > >>> -drive file=/dev/shm/fedora_29.qcow2,if=ide,id=drivex \ > >>> -smp 4 \ > >>> -vga qxl \ > >>> -spice port=6110,disable-ticketing \ > >>> -device > >>> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x7 \ > >>> -chardev spicevmc,name=vdagent,id=vdagent \ > >>> -device virtserialport,nr=1,bus=virtio-serial0.0,chardev=vdagent,name= > >>> com.redhat.spice.0 \ > >>> -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 \ > >>> -device virtio-serial \ > >>> -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \ > >>> -monitor stdio > >> > >> ... > >> > >>> Since I couldn't ping from VM to host B, I did an iperf test between the > >>> VM and > >>> host A with the feature enabled and during the test I have unplugged the > >>> sriov > >>> device, the device was unplugged successfully and no drops where observed > >>> as > >>> you can see in the results below: > >>> > >>> [root@dhcp156-44 ~]# ifconfig > >> Well I suspect this won't tell you anything, this shows packet drops at > >> the hardware level. When e.g. link is down linux won't send any packets > >> out. The simplest test is to monitor latency and throughput and see that > >> while it is lower for the duration of migration, there are no huge > >> spikes around the switch. > > Oh, okay will do that. > > > > I have noticed some nasty lag when I tried to ssh to the VM using the > > failover interface while I didn't experience that with the e1000. > > Sridhar Any idea what might be the cause? > > > When using failover interface, i guess you have the VF interface plugged in > and UP. > So you should be using the primary interface. > > Do you see the VFs MAC configured correctly at the host PF? > You can do > ip link show dev <pf> > and it should show the MACs of all the VFs associated with that PF. You are correct, the vf mac was zet to all zeroes, it can be set by using "ip link set ens2f0 vf 1 mac 8a:f7:20:29:3b:cb" for example [root@virtlab517 netkvm_dev]# ip link show dev ens2f0 2: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master test_br0 state UP mode DEFAULT group default qlen 1000 link/ether f8:f2:1e:33:43:30 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 1 MAC 8a:f7:20:29:3b:cb, spoof checking on, link-state auto, trust off vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
I have tried the same test above now but with Iperf test and ping from the VM to Host B and during the test I eject the primary device using device_del command. This is the same mechanism which my implementation does during migration, however, the traffic is lost now as you can see below. Did you test the failover interface with such scenarios ? [root@dhcp156-44 ~]# ifconfig ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.19.156.44 netmask 255.255.248.0 broadcast 10.19.159.255 inet6 2620:52:0:1398:9699:325b:25f9:e7bb prefixlen 64 scopeid 0x0<global> inet6 fe80::d306:561f:9f43:ff77 prefixlen 64 scopeid 0x20<link> ether 56:cc:c1:01:cc:21 txqueuelen 1000 (Ethernet) RX packets 55201 bytes 3496532 (3.3 MiB) RX errors 72 dropped 0 overruns 0 frame 72 TX packets 738 bytes 70323 (68.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.17 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::bc87:86b8:bc86:be4e prefixlen 64 scopeid 0x20<link> ether 8a:f7:20:29:3b:cb txqueuelen 1000 (Ethernet) RX packets 41260 bytes 3067573 (2.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1156043 bytes 1749160131 (1.6 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 8a:f7:20:29:3b:cb txqueuelen 1000 (Ethernet) RX packets 40183 bytes 2848249 (2.7 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1156043 bytes 1749160131 (1.6 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ens4nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 8a:f7:20:29:3b:cb txqueuelen 1000 (Ethernet) RX packets 1077 bytes 219324 (214.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 3 bytes 264 (264.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3 bytes 264 (264.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 _____________________________________________________________________________________________________________________________________ [root@dhcp156-44 ~]# iperf -c 192.168.1.118 -t 100 -i 1 ------------------------------------------------------------ Client connecting to 192.168.1.118, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.17 port 42210 connected with 192.168.1.118 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 1.0- 2.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 2.0- 3.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 3.0- 4.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 4.0- 5.0 sec 1.09 GBytes 9.40 Gbits/sec [ 3] 5.0- 6.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 6.0- 7.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 7.0- 8.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 8.0- 9.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 9.0-10.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 10.0-11.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 11.0-12.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 12.0-13.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 13.0-14.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 14.0-15.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 15.0-16.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 16.0-17.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 17.0-18.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 18.0-19.0 sec 1.09 GBytes 9.40 Gbits/sec [ 3] 19.0-20.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 20.0-21.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 21.0-22.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 22.0-23.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 23.0-24.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 24.0-25.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 25.0-26.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 26.0-27.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 27.0-28.0 sec 1.10 GBytes 9.42 Gbits/sec [ 3] 28.0-29.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 29.0-30.0 sec 1.10 GBytes 9.41 Gbits/sec [ 3] 30.0-31.0 sec 318 MBytes 2.66 Gbits/sec [ 3] 31.0-32.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 32.0-33.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 33.0-34.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 34.0-35.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 35.0-36.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 36.0-37.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 37.0-38.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 38.0-39.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 39.0-40.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 40.0-41.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 41.0-42.0 sec 0.00 Bytes 0.00 bits/sec ^C[ 3] 42.0-43.0 sec 0.00 Bytes 0.00 bits/sec [ 3] 0.0-43.4 sec 33.2 GBytes 6.57 Gbits/sec _____________________________________________________________________________________________________________________________________ [root@dhcp156-44 ~]# ping 192.168.1.118 PING 192.168.1.118 (192.168.1.118) 56(84) bytes of data. 64 bytes from 192.168.1.118: icmp_seq=1 ttl=64 time=0.264 ms 64 bytes from 192.168.1.118: icmp_seq=2 ttl=64 time=0.167 ms 64 bytes from 192.168.1.118: icmp_seq=3 ttl=64 time=0.168 ms 64 bytes from 192.168.1.118: icmp_seq=4 ttl=64 time=0.174 ms 64 bytes from 192.168.1.118: icmp_seq=5 ttl=64 time=0.168 ms 64 bytes from 192.168.1.118: icmp_seq=6 ttl=64 time=0.282 ms 64 bytes from 192.168.1.118: icmp_seq=7 ttl=64 time=0.179 ms 64 bytes from 192.168.1.118: icmp_seq=8 ttl=64 time=0.141 ms 64 bytes from 192.168.1.118: icmp_seq=9 ttl=64 time=0.165 ms 64 bytes from 192.168.1.118: icmp_seq=10 ttl=64 time=0.168 ms 64 bytes from 192.168.1.118: icmp_seq=11 ttl=64 time=0.153 ms 64 bytes from 192.168.1.118: icmp_seq=12 ttl=64 time=0.296 ms 64 bytes from 192.168.1.118: icmp_seq=13 ttl=64 time=0.258 ms 64 bytes from 192.168.1.118: icmp_seq=14 ttl=64 time=0.236 ms 64 bytes from 192.168.1.118: icmp_seq=15 ttl=64 time=0.190 ms 64 bytes from 192.168.1.118: icmp_seq=16 ttl=64 time=0.245 ms 64 bytes from 192.168.1.118: icmp_seq=17 ttl=64 time=0.187 ms 64 bytes from 192.168.1.118: icmp_seq=18 ttl=64 time=0.222 ms 64 bytes from 192.168.1.118: icmp_seq=19 ttl=64 time=0.231 ms 64 bytes from 192.168.1.118: icmp_seq=20 ttl=64 time=0.279 ms 64 bytes from 192.168.1.118: icmp_seq=21 ttl=64 time=0.271 ms 64 bytes from 192.168.1.118: icmp_seq=22 ttl=64 time=0.319 ms 64 bytes from 192.168.1.118: icmp_seq=23 ttl=64 time=0.350 ms 64 bytes from 192.168.1.118: icmp_seq=24 ttl=64 time=0.311 ms 64 bytes from 192.168.1.118: icmp_seq=25 ttl=64 time=0.249 ms 64 bytes from 192.168.1.118: icmp_seq=26 ttl=64 time=0.258 ms 64 bytes from 192.168.1.118: icmp_seq=27 ttl=64 time=0.220 ms 64 bytes from 192.168.1.118: icmp_seq=28 ttl=64 time=0.299 ms 64 bytes from 192.168.1.118: icmp_seq=29 ttl=64 time=0.281 ms 64 bytes from 192.168.1.118: icmp_seq=30 ttl=64 time=0.271 ms 64 bytes from 192.168.1.118: icmp_seq=31 ttl=64 time=0.241 ms 64 bytes from 192.168.1.118: icmp_seq=32 ttl=64 time=0.245 ms 64 bytes from 192.168.1.118: icmp_seq=33 ttl=64 time=0.245 ms 64 bytes from 192.168.1.118: icmp_seq=34 ttl=64 time=0.287 ms 64 bytes from 192.168.1.118: icmp_seq=35 ttl=64 time=0.322 ms 64 bytes from 192.168.1.118: icmp_seq=36 ttl=64 time=0.247 ms 64 bytes from 192.168.1.118: icmp_seq=37 ttl=64 time=0.316 ms 64 bytes from 192.168.1.118: icmp_seq=38 ttl=64 time=0.255 ms 64 bytes from 192.168.1.118: icmp_seq=39 ttl=64 time=0.308 ms 64 bytes from 192.168.1.118: icmp_seq=40 ttl=64 time=0.250 ms 64 bytes from 192.168.1.118: icmp_seq=41 ttl=64 time=0.260 ms ^C --- 192.168.1.118 ping statistics --- 63 packets transmitted, 41 received, 34.9206% packet loss, time 568ms rtt min/avg/max/mdev = 0.141/0.243/0.350/0.054 ms > > > -- Respectfully, Sameeh Jubran Linkedin Software Engineer @ Daynix. --------------------------------------------------------------------- To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org