On Mon, Jan 25, 2021 at 01:23:04PM -0800, Edwin Peer wrote: > On Mon, Jan 25, 2021 at 12:41 PM Jason Gunthorpe <j...@nvidia.com> wrote: > > > > That's an implementation decision. Nothing mandates that the state has > > > to physically exist in the same structure, only that reads and writes > > > are appropriately responded to. > > > > Yes, PCI does mandate this, you can't store the data on the other side > > of the PCI link, and if you can't cross the PCI link that only leaves > > on die/package memory resources. > > Going off device was not what I was suggesting at all. I meant the > data doesn't necessarily need to be stored in the same physical > layout.
It doesn't change anything, every writable bit must still be stored on-die SRAM. You can compute the minimum by summing all writable and read-reporting bits in the standard SRIOV config space. Every bit used for SRIOV is a bit that couldn't be used to improve device performance. > > > Right, but presumably it still needs to be at least a page. And, > > > nothing says your device's VF BAR protocol can't be equally simple. > > > > Having VFs that are not self-contained would require significant > > changing of current infrastructure, if we are going to change things > > then let's fix everything instead of some half measure. > > I don't understand what you mean by self-contained. Self-contained means you can pass the VF to a VM with vfio and run a driver on it. A VF that only has a write-only doorbell page probably cannot be self contained. > In practice, there will be some kind of configuration channel too, > but this doesn't necessarily need a lot of room either I don't know of any device that can run without configuration, even in a VF case. So this all costs SRAM too. > > The actual complexity inside the kernel is small and the user > > experience to manage them through devlink is dramatically better than > > SRIOV. I think it is a win even if there isn't any HW savings. > > I'm not sure I agree with respect to user experience. Users are > familiar with SR-IOV. Sort of, SRIOV is a very bad fit for these sophisticated devices, and no, users are not familiar with the weird intricate details of SR-IOV in the context of very sophisticated reconfigurable HW like we are seeing now. Look at the other series about MSI-X reconfiguration for some colour on where SRIOV runs into limits due to its specific design. > Now you impose a complementary model for accomplishing the same goal > (without solving all the problems, as per the previous discussion, > so we'll need to reinvent it again later). I'm not sure what you are referring to. > It's not easier for vendors either. Now we need to get users onto new > drivers to exploit it, with all the distribution lags that entails > (where existing drivers would work for SR-IOV). Compatability with existing drivers in a VM is a vendor choice. Drivers can do a lot in a scalable way in hypervisor SW to present whateve programming interface makes sense to the VM. Intel is showing this approach in their IDXD SIOV ADI driver. > Some vendors will support it, some won't, further adding to user > confusion. Such is the nature of all things, some vendors supported SRIOV and other didn't too. Jason
