On Mon, Jan 25, 2021 at 12:22:13PM -0800, Edwin Peer wrote: > On Mon, Jan 25, 2021 at 11:59 AM Jason Gunthorpe <j...@nvidia.com> wrote: > > > Every writable data mandated by the PCI spec requires very expensive > > on-die SRAM to store it. > > That's an implementation decision. Nothing mandates that the state has > to physically exist in the same structure, only that reads and writes > are appropriately responded to.
Yes, PCI does mandate this, you can't store the data on the other side of the PCI link, and if you can't cross the PCI link that only leaves on die/package memory resources. > > We've seen Intel drivers that show their SIOV ADIs don't even have a > > register file and the only PCI presence is just a write-only doorbell > > page in the BAR. > > Right, but presumably it still needs to be at least a page. And, > nothing says your device's VF BAR protocol can't be equally simple. Having VFs that are not self-contained would require significant changing of current infrastructure, if we are going to change things then let's fix everything instead of some half measure. SRIOV really doesn't bring much benefits, it has lots of odd little restrictions and strange lifecycle rules for what modern devices want to do. > > It is hard to argue a write-only register in a BAR page vs all the > > SRIOV trappings when it comes to HW cost. > > Say it's double the cost? I don't know that it is, but does that > warrant the additional complexity of SFs? We should try to quantify. The actual complexity inside the kernel is small and the user experience to manage them through devlink is dramatically better than SRIOV. I think it is a win even if there isn't any HW savings. Jason
