Yeah I assume something is going wrong with libvirt logic, direct qemu command is probably working. >From the logs it might be related to udev interface which does not detect properly the vf when libvirt tries to discover the interface . Can you repeat your scenario, but with libvirt ?
No specific kernel logs from kernel. here is libvirt debug info: 2000-01-18 00:10:51.573+0000: 2055: info : virEventPollRunOnce:641 : EVENT_POLL_RUN: nhandles=8 timeout=4979 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollRunOnce:651 : Poll got 1 event(s) 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchTimeouts:433 : Dispatch 2 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:479 : Dispatch 8 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=0 w=1 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=1 w=2 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=2 w=3 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=3 w=4 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=4 w=5 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=5 w=6 2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493 : i=6 w=7 2000-01-18 00:10:51.574+0000: 2055: info : virEventPollDispatchHandles:507 : EVENT_POLL_DISPATCH_HANDLE: watch=7 events=1 2000-01-18 00:10:51.574+0000: 2055: debug : udevEventHandleCallback:1543 : udev action: 'add' 2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:124 : udev reports device '35' does not have property 'DRIVER' 2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:139 : Found property key 'SUBSYSTEM' value 'vfio' for device with sysname '35' 2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceType:1245 : Could not determine device type for device with sysfs name '35' 2000-01-18 00:10:51.574+0000: 2055: debug : udevAddOneDevice:1413 : Discarding device -1 0x5641bbfd2570 /sys/devices/virtual/vfio/35 and later i see: 2000-01-18 00:10:51.575+0000: 2055: info : virEventPollDispatchHandles:507 : EVENT_POLL_DISPATCH_HANDLE: watch=7 events=1 2000-01-18 00:10:51.575+0000: 2055: debug : udevEventHandleCallback:1543 : udev action: 'remove' 2000-01-18 00:10:51.575+0000: 2055: debug : udevRemoveOneDevice:1314 : Failed to find device to remove that has udev name '/sys/devices/virtual/vfio/35' 2000-01-18 00:10:51.575+0000: 2055: debug : virEventPollDispatchHandles:493 : i=7 w=31 which causes the vm bootup to fail: 2000-01-18 00:10:51.576+0000: 2057: debug : virNetServerProgramSendError:153 : prog=536903814 ver=1 proc=10 type=1 serial=5 msg=0x5641bbfd3b90 rerr=0x7f49bc0b7db0 2000-01-18 00:10:51.576+0000: 2057: debug : virNetMessageEncodePayload:376 : Encode length as 120 2000-01-18 00:10:51.576+0000: 2057: debug : virNetServerClientSendMessageLocked:1397 : msg=0x5641bbfd3b90 proc=10 len=120 offset=0 2000-01-18 00:10:51.576+0000: 2057: info : virNetServerClientSendMessageLocked:1405 : RPC_SERVER_CLIENT_MSG_TX_QUEUE: client=0x5641bbfd59b0 len=120 prog=536903814 vers=1 proc=10 type=1 status=1 serial=5 2000-01-18 00:10:51.576+0000: 2057: debug : virNetServerClientCalculateHandleMode:157 : tls=(nil) hs=-1, rx=0x5641bbfddc30 tx=0x5641bbfd3b90 2000-01-18 00:10:51.576+0000: 2057: debug : virNetServerClientCalculateHandleMode:192 : mode=3 the xml file: WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit vm64efi or other application using the libvirt API. --> <!-- domain type='qemu' --> <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0"; type="kvm"> <name>VoidVNF-SRIOV</name> <uuid>05ed74d9-e31f-80c2-fb69-6cb052abdb6b</uuid> <memory unit="M">3072</memory> <currentMemory unit="M">3072</currentMemory> <qemu:commandline> <qemu:arg value="-object"/> <qemu:arg value="memory-backend-file,id=mem,size=3072M,mem-path=/mnt/hugepages,share=on"/> <qemu:arg value="-numa"/> <qemu:arg value="node,memdev=mem"/> <qemu:arg value="-mem-prealloc"/> </qemu:commandline> <vcpu placement="static">2</vcpu> <os> <type>hvm</type> <loader>/mnt/vm_snapshot/OVMF.fd</loader> <boot dev="hd"/> </os> <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host'/> </qemu:commandline> <vcpu placement='static' cpuset='0'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='12'/> <vcpupin vcpu='1' cpuset='13'/> </cputune> <features> <acpi/> <apic/> <pae/> </features> <clock offset="utc"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <!-- on_crash>restart</on_crash --> <devices> <!-- emulator>/usr/bin/kvm-spice</emulator --> <disk type="file" device="disk"> <driver name="qemu" type="qcow2"/> <source file="/mnt/vm_snapshot/MiniVoidVNF.qcow2"/> <!-- target dev='sda' bus='sata'/ --> <target dev="hda" bus="ide"/> <address type="drive" controller="0" bus="0" target="0" unit="0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> </disk> <controller type="usb" index="0"> <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> </controller> <controller type="pci" index="0" model="pci-root"/> <interface type="hostdev" managed="yes"> <driver name='vfio'/> <mac address="52:54:12:34:56:78"/> <source> <address type="pci" domain="0x0000" bus="0x05" slot="0x10" function="0x0"/> </source> <parameters interfaceid="0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/> <vlan> <tag id="30"/> </vlan> </interface> <interface type="hostdev" managed="yes"> <driver name='vfio'/> <mac address="52:54:12:34:56:79"/> <source> <address type="pci" domain="0x0000" bus="0x05" slot="0x10" function="0x2"/> </source> <parameters interfaceid="0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/> <vlan> <tag id="40"/> </vlan> </interface> <!-- interface type='bridge'> <mac address='52:54:00:85:0e:d0'/> <source bridge='lxcbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface --> <!-- interface type='bridge'> <mac address='52:54:00:00:a7:e0'/> <source bridge='lxcbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface --> <!--interface type="vhostuser"> <mac address="fa:16:3e:07:88:13"/> <source type="unix" path="/usr/local/var/run/openvswitch/vhost-0ca8d512-ae" mode="client"/> <target dev="vhost-0ca8d512-ae"/> <model type="virtio"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> <parameters interfaceid="0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/> </interface> <interface type="vhostuser"> <mac address="fa:16:3e:07:88:14"/> <source type="unix" path="/usr/local/var/run/openvswitch/vhost-0ca8d512-cc" mode="client"/> <target dev="vhost-0ca8d512-cc"/> <model type="virtio"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/> <parameters interfaceid="0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/> </interface--> <serial type="pty"> <target port="0"/> </serial> <console type="pty"> <target type="serial" port="0"/> </console> <input type="tablet" bus="usb"/> <input type="mouse" bus="ps2"/> <input type="keyboard" bus="ps2"/> <graphics type="vnc" port="-1" autoport="yes" keymap="en-us"/> <video> <model type="cirrus" vram="9216" heads="1"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </video> <memballoon model="virtio"> <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/> </memballoon> </devices> </domain> On Wed, Jul 12, 2017 at 2:43 PM, Rosen, Rami <[email protected]> wrote: > Hi Ran, > > > > >echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs > > >I'm getting 63 pfs in lspci, > > > > You mean VFs and not pfs, I assume. > > > > I tried something similar but with qemu cli, not virsh, using PCU > passtrhough > > (-device pci-assign…) and i40e, but using igb_uio. I tried it with one VF > and it worked OK. > > > > Do you see any errors in the kernel log (dmesg)? > > > > > > Regards, > > Rami Rosen > > > > *From:* ran lifshitz [mailto:[email protected]] > *Sent:* Wednesday, July 12, 2017 14:34 > *To:* Rosen, Rami <[email protected]> > *Cc:* [email protected] > *Subject:* Re: [dpdk-users] Using PF bound to IGB_UIO, together with > sr-iov VFs passed to guests > > > > Hi Rami, > > > > I'm creating the vfs, while the pf is bound to igb_uio: > > echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs > > > > I'm getting 63 pfs in lspci, but when i try to load the vm: > > > > virsh create mini1.xml > > error: Failed to create domain from mini1.xml > > error: An error occurred, but the cause is unknown > > > > > > in the dpdk manual, it's refering to qemu, so I probably assume that the > libvirt try to initialize the VF mac and vlan, but this is not supported by > the igb_uio ? > > it works well when ixgbe.ko is bound. > > > > On Wed, Jul 12, 2017 at 1:11 PM, Rosen, Rami <[email protected]> wrote: > > Hi, Ran, > > What is the internal error you get when bringing up the > guest ? > > Regards, > Rami Rosen > > -----Original Message----- > From: users [mailto:[email protected]] On Behalf Of ran lifshitz > Sent: Tuesday, July 11, 2017 12:24 > To: [email protected] > Subject: [dpdk-users] Using PF bound to IGB_UIO, together with sr-iov VFs > passed to guests > > Hi, > > I'm having trouble trying to build a scenario were the host has the PF > bound to igb_uio, and working with dpdk application, together with sr-iov > VFs passthrough to guest machines. > > Libvirt is not able to bring up the guest, and exit with internal error. > I assume that the vlan/mac initialization which is working well, when the > PF is assigned to ixgbe.k does not work when bound to igb_uio. > > Is this an igb_uio issue or libvirt ? > is there a workaround for this ? > > Regards > Ran > > > > > > -- > > > > [image: Google+] > <https://plus.google.com/111964218283461713253/posts/p/pub>[image: > LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]*website > <http://www.ran-lifshitz.com/> **|** vCard > <http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> **|* > * email* > > [image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/> > > > mobile 972-54-4308444 > > > > > > > -- [image: Google+] <https://plus.google.com/111964218283461713253/posts/p/pub>[image: LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]website <http://www.ran-lifshitz.com/> | vCard <http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> | email [image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/> mobile 972-54-4308444
