On Thu, Oct 5, 2023 at 3:31 AM Shunsuke Mie <m...@igel.co.jp> wrote: > > Hi Jiri, Mattias and all. > > 2023年10月4日(水) 16:36 Mattias Nissler <mniss...@rivosinc.com>: >>> >>> hi shunsuke, all, >>> what about vfio-user + qemu? > > Thank you for the suggestion. > >> FWIW, I have had some good success using VFIO-user to bridge software >> components to hardware designs. For the most part, I have been hooking up >> software endpoint models to hardware design components speaking the PCIe >> transaction layer protocol. The central piece you need is a way to translate >> between the VFIO-user protocol and PCIe transaction layer messages, >> basically converting ECAM accesses, memory accesses (DMA+MMIO), and >> interrupts between the two worlds. I have some code which implements the >> basics of that. It's certainly far from complete (TLP is a massive >> protocol), but it works well enough for me. I believe we should be able to >> open-source this if there's interest, let me know. > > It is what I want to do, but I'm not familiar with the vfio and vfio-user, > and I have a question. QEMU has a PCI TLP communication implementation for > Multi-process QEMU[1]. It is similar to your success.
I'm no qemu expert, but my understanding is that the plan is for the existing multi-process QEMU implementation to eventually be superseded/replaced by the VFIO-user based one (qemu folks, please correct me if I'm wrong). From a functional perspective they are more or less equivalent AFAICT. > The multi-process qemu also communicates TLP over UDS. Could you let me know > your opinion about it? Note that neither multi-process qemu nor VFIO-user actually pass around TLPs, but rather have their own command language to encode ECAM, MMIO, DMA, interrupts etc. However, translation from/to TLP is possible and works well enough in my experience. > >> One thing to note is that there are currently some limits to bridging >> VFIO-user / TLP that I haven't figured out and/or will need further work: >> Advanced PCIe concepts like PASID, ATS/PRI, SR-IOV etc. may lack equivalents >> on the VFIO-user side that would have to be filled in. The folk behind >> libvfio-user[2] have been very approachable and open to improvements in my >> experience though. >> >> If I understand correctly, the specific goal here is testing PCIe endpoint >> designs against a Linux host. What you'd need for that is a PCI host >> controller for the Linux side to talk to and then hooking up endpoints on >> the transaction layer. QEMU can simulate host controllers that work with >> existing Linux drivers just fine. Then you can put a vfio-user-pci stub >> device (I don't think this has landed in qemu yet, but you can find the code >> at [1]) on the simulated PCI bus which will expose any software interactions >> with the endpoint as VFIO-user protocol messages over unix domain socket. >> The piece you need to bring is a VFIO-user server that handles these >> messages. Its task is basically translating between VFIO-user and TLP and >> then injecting TLP into your hardware design. > > Yes, If the pci host controller you said can be implemented, I can achieve my > goal. I meant to say that the existing PCIe host controller implementations in qemu can be used as is. > > To begin with, I'll investigate the vfio and libvfio-user. Thanks!. > > [1] https://www.qemu.org/docs/master/system/multi-process.html > > Best, > Shunsuke >> >> >> [1] https://github.com/oracle/qemu/tree/vfio-user-p3.1 - I believe that's >> the latest version, Jagannathan Raman will know best >> [2] https://github.com/nutanix/libvfio-user >> >
