On Wed, 8 Feb 2023 16:28:44 -0600 zhiting zhu <zhiti...@cs.utexas.edu> wrote:
> Hi, > > I saw a PoC: > https://lore.kernel.org/qemu-devel/20220525121422.00003...@huawei.com/T/ to > implement memory pooling and fabric manager on qemu. Is there any further > development on this? Can qemu emulate a memory pooling on a simple case > that two virtual machines connected to a CXL switch where some memory > devices are attached to? > > Best, > Zhiting Hi Zhiting, +CC linux-cxl as it's not as much of a firehose as qemu-devel +CC Slava who has been driving discussion around fabric management. > No progress on that particular approach though some discussion on what the FM architecture itself might look like. https://lore.kernel.org/linux-cxl/7f001eaf-c512-436a-a9dd-e08730c91...@bytedance.com/ There was a sticky problem with doing MCTP over I2C which is that there are very few I2C controllers that support the combination of master and subordinate needed to do MCTP. The one that was used for that (aspeed) doesn't have ACPI bindings (and they are non trivial to add due to clocks etc and likely to be controversial on kernel side given I just want it for emulation!). So far we don't have DT bindings for CXL (either the CFMWS - CXL fixed memory windows or pxb-cxl - the host bridge) I'll be sending out one of the precursors for that as an RFC soon. So we are in the fun position that we can either emulate the comms path to the devices, or we can emulate the host actually using the devices. I was planning to get back to that eventually but we have other options now CXL 3.0 has been published. CXL 3.0 provides two paths forwards that let us test the equivalent functionality with fewer moving parts. 1) CXL SWCCI which is an extra PCI function next to the switch upstream port that provides a mailbox that takes FM-API commands. PoC Kernel code at: https://lore.kernel.org/linux-cxl/20221025104243.20836-1-jonathan.came...@huawei.com/ Latest branch in gitlab.com/jic23/qemu should have switch CCI emulation support. (branches are dated) Note we have a lot of stuff outstanding, either out for review or backed up behind things that are. 2) Multi Headed Devices. These allow FM-API commands over the normal CXL mailbox. I did a very basic PoC to see how this would fit in with the kernel side of things but recently there has been far too much we need to enable in the shorter term. Note though that there is a long way to go before we can do what you want. The steps I'd expect to see along the way: 1) Emulate an Multi Headed Device. Initially connect two heads to different host bridges on a single QEMU machine. That lets us test most of the code flows without needing to handle tests that involve multiple machines. Later, we could add a means to connect between two instances of QEMU. 2) Add DCD support (we'll need the kernel side of that as well) 3) Wire it all up. 4) Do the same for a Switch with MLDs behind it so we can poke the fun corners. Note that in common with memory emulation in general for CXL on QEMU the need to do live address decoding will make performance terrible. There are probably ways to improve that, but whilst we are at the stage of trying to get as much functional as possible for testing purposes, I'm not sure anyone will pursue those options. May not make sense in the longer term either. I'm more than happy to offer suggestions / feedback on approaches to this and will get back to it myself once some more pressing requirements are dealt with. Jonathan