On Fri, Oct 6, 2023 at 1:51 PM Shunsuke Mie <m...@igel.co.jp> wrote: > > > On 2023/10/05 16:02, Mattias Nissler wrote: > > On Thu, Oct 5, 2023 at 3:31 AM Shunsuke Mie <m...@igel.co.jp> wrote: > >> Hi Jiri, Mattias and all. > >> > >> 2023年10月4日(水) 16:36 Mattias Nissler <mniss...@rivosinc.com>: > >>>> hi shunsuke, all, > >>>> what about vfio-user + qemu? > >> Thank you for the suggestion. > >> > >>> FWIW, I have had some good success using VFIO-user to bridge software > >>> components to hardware designs. For the most part, I have been hooking up > >>> software endpoint models to hardware design components speaking the PCIe > >>> transaction layer protocol. The central piece you need is a way to > >>> translate between the VFIO-user protocol and PCIe transaction layer > >>> messages, basically converting ECAM accesses, memory accesses (DMA+MMIO), > >>> and interrupts between the two worlds. I have some code which implements > >>> the basics of that. It's certainly far from complete (TLP is a massive > >>> protocol), but it works well enough for me. I believe we should be able > >>> to open-source this if there's interest, let me know. > >> It is what I want to do, but I'm not familiar with the vfio and vfio-user, > >> and I have a question. QEMU has a PCI TLP communication implementation for > >> Multi-process QEMU[1]. It is similar to your success. > > I'm no qemu expert, but my understanding is that the plan is for the > > existing multi-process QEMU implementation to eventually be > > superseded/replaced by the VFIO-user based one (qemu folks, please > > correct me if I'm wrong). From a functional perspective they are more > > or less equivalent AFAICT. > > > The project is promising. > > I found a session about the vfio adapts to Multi-process QEMU[1] in KVM > Forun 2021, butI couldn't found some posted patches. > If anyone knows status of this project, could you please let me know?
Again, I'm just an interested bystander, so take my words with a grain of salt. That said, my understanding is that there is an intention to get the vfio-user client code into qemu in the foreseeable future. The most recent version of the code that I'm aware of is here: https://github.com/oracle/qemu/tree/vfio-user-p3.1 > > [1] https://www.youtube.com/watch?v=NBT8rImx3VE > >> The multi-process qemu also communicates TLP over UDS. Could you let me > >> know your opinion about it? > > Note that neither multi-process qemu nor VFIO-user actually pass > > around TLPs, but rather have their own command language to encode > > ECAM, MMIO, DMA, interrupts etc. However, translation from/to TLP is > > possible and works well enough in my experience. > I agree. > >>> One thing to note is that there are currently some limits to bridging > >>> VFIO-user / TLP that I haven't figured out and/or will need further work: > >>> Advanced PCIe concepts like PASID, ATS/PRI, SR-IOV etc. may lack > >>> equivalents on the VFIO-user side that would have to be filled in. The > >>> folk behind libvfio-user[2] have been very approachable and open to > >>> improvements in my experience though. > >>> > >>> If I understand correctly, the specific goal here is testing PCIe > >>> endpoint designs against a Linux host. What you'd need for that is a PCI > >>> host controller for the Linux side to talk to and then hooking up > >>> endpoints on the transaction layer. QEMU can simulate host controllers > >>> that work with existing Linux drivers just fine. Then you can put a > >>> vfio-user-pci stub device (I don't think this has landed in qemu yet, but > >>> you can find the code at [1]) on the simulated PCI bus which will expose > >>> any software interactions with the endpoint as VFIO-user protocol > >>> messages over unix domain socket. The piece you need to bring is a > >>> VFIO-user server that handles these messages. Its task is basically > >>> translating between VFIO-user and TLP and then injecting TLP into your > >>> hardware design. > >> Yes, If the pci host controller you said can be implemented, I can achieve > >> my goal. > > I meant to say that the existing PCIe host controller implementations > > in qemu can be used as is. > Sorry, I misunderstood. > >> To begin with, I'll investigate the vfio and libvfio-user. Thanks!. > >> > >> [1] https://www.qemu.org/docs/master/system/multi-process.html > >> > >> Best, > >> Shunsuke > >>> > >>> [1] https://github.com/oracle/qemu/tree/vfio-user-p3.1 - I believe that's > >>> the latest version, Jagannathan Raman will know best > >>> [2] https://github.com/nutanix/libvfio-user > >>>
