On Thu, 2018-07-26 at 20:05 +0300, Marcel Apfelbaum wrote: > > On 07/26/2018 07:42 PM, Knut Omang wrote: > > On Thu, 2018-07-26 at 19:41 +0300, Michael S. Tsirkin wrote: > > > On Thu, Jul 26, 2018 at 06:38:32PM +0200, Knut Omang wrote: > > > > On Thu, 2018-07-26 at 18:51 +0300, Marcel Apfelbaum wrote: > > > > > Hi > > > > > > > > > > On 07/26/2018 05:52 PM, Stefan Hajnoczi wrote: > > > > > > On Thu, Jul 12, 2018 at 07:33:14AM +0000, Elijah Shakkour wrote: > > > > > > > Hey, > > > > > > > > > > > > > > Our team is adding a NIC functional emulation to QEMU. > > > > > > > One of the features we are adding to this NIC is SRIOV. > > > > > > > > > > > > > > Here is the error message I get when checking SRIOV support > > > > > > > of our > > > > > > > emulated NIC on Win2016 server (the hyper-v VM). > > > > > > > " > > > > > > > SR-IOV cannot be used on this system as the PCI Express hardware > > > > > > > does > > > > > > > not > > > > > > > support Access Control Services (ACS) at any root port. > > > > > > > > > > QEMU's emulated PCI Express Root Ports do not support ACS yet, however > > > > > I > > > > > am not sure ACS is a prerequisite > > > > > for SR-IOV. We would need ARI support for allowing more than 8VFs, but > > > > > QEMU doesn't support that either (yet). > > > > > > > > > > Knut Omag has some working patches, he successfully implemented SR-IOV > > > > > with QEMU, see: > > > > > https://github.com/knuto/qemu/tree/sriov_patches_v7 > > > > > > > > > > The code was not merged since we need at least a device with SR-IOV > > > > > support to justify the addition. > > > > > > > > FYI, I recently rebased these to latest master but just didn't get to > > > > push > > > > them > > > > out. I did just now - they are available here: > > > > > > > > https://github.com/knuto/qemu/tree/sriov_patches_v8 > > > > > > > > As far as I know ARI support with my patch set works just fine - I have > > > > tested > > > > it with lots of VFs. > > > > > > > > One of the patches in the series (pci: Make use of the devfn property > > > > when > > > > registering new devices) is necessary to make ARI work as it should with > > > > SR/IOV. > > > > > > > > For the hardware model I developed the SR/IOV patches for, I also added > > > > enough > > > > ACS support in the root port (PCIe capability helper patch + usage in > > > > ioh3420) to make VFIO "happy". I haven't submitted them because they > > > > are "questionable" since they likely do not reflect the actual features > > > > of > > > > the > > > > ioh3420. > > > > > > In that the actual ioh3420 doesn't support ACS? > > > > yes.. I don't have one so I don't know but that was my assumption.. > > Hi Knut, > > We have now a generic PCIe Root Port we can add whatever we want to it. > See please hw/pci-bridge/gen_pcie_root_port.c.
Ok, I see! - I wasn't aware of that. I was thinking more in terms of just emulating a slightly newer Intel root port but never got around to that. > So your patches add both ARI and ACS support, nice! > Maybe it worth merging at least these features. I'll have a look at the generic root port, it is probably a natural extension for v9 of the sriov patch set :-) Thanks, Knut > > Thanks, > Marcel > > > > > Knut > > > > > > I can make those available if interesting. > > > > > > > > Thanks, > > > > Knut > > > > > > > > > > > Contact your system vendor for further information. > > > > > > > " > > > > > > > > > > > > I'm not sure what the status of emulated SR-IOV is so I have CCed > > > > > > Michael Tsirkin and Marcel Apfelbaum, the PCI maintainers in QEMU. > > > > > > > > > > Thanks, > > > > > Marcel > > > > > > > > > > > > Could you please advise about what could be the issue here? > > > > > > > > > > > > > > BTW: I use same configuration (VM XML file attached) when running > > > > > > > linux VM > > > > > > > (RH7.2) image (instead of Win Hyper-V) over the same host and > > > > > > > SRIOV is > > > > > > > working for me there. > > > > > > > > > > > > > > Here the XML file I use to define the VM (our emulated NIC is > > > > > > > added at > > > > > > > the > > > > > > > end of XML): > > > > > > > " > > > > > > > <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/q > > > > > > > emu/ > > > > > > > 1.0' > > > > > > > > ; > > > > > > > > > > > > > > <name>nst105</name> > > > > > > > <uuid>0249a525-2ee2-432b-a1f5-a6db83b089a3</uuid> > > > > > > > <memory unit='KiB'>8388608</memory> > > > > > > > <currentMemory unit='KiB'>8388608</currentMemory> > > > > > > > <vcpu placement='static'>8</vcpu> > > > > > > > <resource> > > > > > > > <partition>/machine</partition> > > > > > > > </resource> > > > > > > > <os> > > > > > > > <type arch='x86_64' machine='pc-q35-2.11'>hvm</type> > > > > > > > </os> > > > > > > > <features> > > > > > > > <acpi/> > > > > > > > <apic/> > > > > > > > <hyperv> > > > > > > > <relaxed state='on'/> > > > > > > > <vapic state='on'/> > > > > > > > <spinlocks state='on' retries='8191'/> > > > > > > > </hyperv> > > > > > > > </features> > > > > > > > <cpu mode='custom' match='exact' check='none'> > > > > > > > <model fallback='forbid'>SandyBridge</model> > > > > > > > <feature policy='require' name='hypervisor'/> > > > > > > > <feature policy='require' name='vmx'/> > > > > > > > </cpu> > > > > > > > <clock offset='localtime'> > > > > > > > <timer name='rtc' tickpolicy='catchup'/> > > > > > > > <timer name='pit' tickpolicy='delay'/> > > > > > > > <timer name='hpet' present='no'/> > > > > > > > <timer name='hypervclock' present='yes'/> > > > > > > > </clock> > > > > > > > <on_poweroff>destroy</on_poweroff> > > > > > > > <on_reboot>restart</on_reboot> > > > > > > > <on_crash>destroy</on_crash> > > > > > > > <pm> > > > > > > > <suspend-to-mem enabled='no'/> > > > > > > > <suspend-to-disk enabled='no'/> > > > > > > > </pm> > > > > > > > <devices> > > > > > > > <emulator>/opt/qemu/bin/qemu-system-x86_64</emulator> > > > > > > > <disk type='file' device='disk'> > > > > > > > <driver name='qemu' type='qcow2'/> > > > > > > > <source file='/images/nst105.qcow2'/> > > > > > > > <target dev='sda' bus='sata'/> > > > > > > > <boot order='2'/> > > > > > > > <address type='drive' controller='0' bus='0' target='0' > > > > > > > unit='0'/> > > > > > > > </disk> > > > > > > > <controller type='usb' index='0' model='ich9-ehci1'> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' > > > > > > > function='0x7'/> > > > > > > > </controller> > > > > > > > <controller type='usb' index='0' model='ich9-uhci1'> > > > > > > > <master startport='0'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' > > > > > > > function='0x0' multifunction='on'/> > > > > > > > </controller> > > > > > > > <controller type='usb' index='0' model='ich9-uhci2'> > > > > > > > <master startport='2'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' > > > > > > > function='0x1'/> > > > > > > > </controller> > > > > > > > <controller type='usb' index='0' model='ich9-uhci3'> > > > > > > > <master startport='4'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' > > > > > > > function='0x2'/> > > > > > > > </controller> > > > > > > > <controller type='sata' index='0'> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' > > > > > > > function='0x2'/> > > > > > > > </controller> > > > > > > > <controller type='pci' index='0' model='pcie-root'/> > > > > > > > <controller type='pci' index='1' model='pcie-root-port'> > > > > > > > <model name='pcie-root-port'/> > > > > > > > <target chassis='1' port='0x10'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > > > > > > > function='0x0' multifunction='on'/> > > > > > > > </controller> > > > > > > > <controller type='pci' index='2' model='pcie-root-port'> > > > > > > > <model name='pcie-root-port'/> > > > > > > > <target chassis='2' port='0x11'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > > > > > > > function='0x1'/> > > > > > > > </controller> > > > > > > > <controller type='pci' index='3' model='pcie-root-port'> > > > > > > > <model name='pcie-root-port'/> > > > > > > > <target chassis='3' port='0x12'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > > > > > > > function='0x2'/> > > > > > > > </controller> > > > > > > > <controller type='pci' index='4' model='dmi-to-pci-bridge'> > > > > > > > <model name='i82801b11-bridge'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' > > > > > > > function='0x0'/> > > > > > > > </controller> > > > > > > > <controller type='pci' index='5' model='pci-bridge'> > > > > > > > <model name='pci-bridge'/> > > > > > > > <target chassisNr='5'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > > > > > > > function='0x0'/> > > > > > > > </controller> > > > > > > > <interface type='bridge'> > > > > > > > <mac address='00:50:56:1b:dd:6f'/> > > > > > > > <source bridge='br0'/> > > > > > > > <model type='virtio'/> > > > > > > > <boot order='1'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' > > > > > > > function='0x0'/> > > > > > > > </interface> > > > > > > > <serial type='pty'> > > > > > > > <target port='0'/> > > > > > > > </serial> > > > > > > > <console type='pty'> > > > > > > > <target type='serial' port='0'/> > > > > > > > </console> > > > > > > > <input type='tablet' bus='usb'> > > > > > > > <address type='usb' bus='0' port='1'/> > > > > > > > </input> > > > > > > > <input type='mouse' bus='ps2'/> > > > > > > > <input type='keyboard' bus='ps2'/> > > > > > > > <graphics type='vnc' port='-1' autoport='yes' > > > > > > > listen='127.0.0.1' > > > > > > > keymap='en-us'> > > > > > > > <listen type='address' address='127.0.0.1'/> > > > > > > > </graphics> > > > > > > > <video> > > > > > > > <model type='vga' vram='16384' heads='1' primary='yes'/> > > > > > > > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > > > > > > > function='0x0'/> > > > > > > > </video> > > > > > > > <memballoon model='virtio'> > > > > > > > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' > > > > > > > function='0x0'/> > > > > > > > </memballoon> > > > > > > > </devices> > > > > > > > <seclabel type='none' model='none'/> > > > > > > > <seclabel type='dynamic' model='dac' relabel='yes'/> > > > > > > > <qemu:commandline> > > > > > > > <qemu:arg value='-device'/> > > > > > > > <qemu:arg value='pcie-root-port,pref64- > > > > > > > reserve=500M,slot=0,id=pcie_port.1'/> > > > > > > > <qemu:arg value='-netdev'/> > > > > > > > <qemu:arg > > > > > > > value='tap,id=tap5,ifname=tap5,script=no,downscript=no'/> > > > > > > > <qemu:arg value='-device'/> > > > > > > > <qemu:arg value='my_emu_nic,netdev=tap5,bus=pcie_port.1'/> > > > > > > > </qemu:commandline> > > > > > > > </domain> > > > > > > > " > > > > > > > ______________________ > > > > > > > General info: > > > > > > > Host OS: RH7.0 (Kernel: 4.14.13) > > > > > > > QEMU version: 2.11 > > > > > > > libvirt version: 3.2.0 > > > > > > > Running the following on the host shows that both nested and IOMMU > > > > > > > are > > > > > > > enabled: > > > > > > > ~]#: cat /sys/module/kvm_intel/parameters/nested > > > > > > > Y > > > > > > > ~]# dmesg | grep -e DMAR -e IOMMU > > > > > > > [ 0.000000] DMAR: IOMMU enabled > > > > > > > > > > > > > > Thanks, > > > > > > > Elijah > > > > > > > > >