Hi, this is a reworked version of the Dom0 GICv3-ITS emulation series. This is still not fully where I want it and has some loose bits and pieces still, but since there are significant changes in the architecture I wanted to have an opinion before going ahead and replacing every single number with a named constant ;-) If that smells like a "send out before the end of the year", you are spot on.
This series introduces ARM GICv3 ITS emulation, for now restricted to Dom0 only. The ITS is an interrupt controller widget providing a sophisticated way to deal with MSIs in a scalable manner. For hardware which relies on the ITS to provide interrupts for its peripherals this code is needed to get a machine booted into Dom0 at all. ITS emulation for DomUs is only really useful with PCI passthrough, which is not yet available for ARM. It is expected that this feature will be co-developed with the ITS DomU code. However this code drop here considered DomU emulation already, to keep later architectural changes to a minimum. Some generic design principles: * The current GIC code statically allocates structures for each supported IRQ (both for the host and the guest), which due to the potentially millions of LPI interrupts is not feasible to copy for the ITS. So we refrain from introducing the ITS as a first class Xen interrupt controller, also we don't hold struct irq_desc's or struct pending_irq's for each possible LPI. Fortunately LPIs are only interesting to guests, so we get away with storing only the virtual IRQ number and the guest VCPU for each allocated host LPI, which can be stashed into one uint64_t. This data is stored in a two-level table, which is both memory efficient and quick to access. We hook into the existing IRQ handling and VGIC code to avoid accessing the normal structures, providing alternative methods for getting the needed information (priority, is enabled?) for LPIs. For interrupts which are queued to or are actually in a guest we allocate struct pending_irq's on demand. As it is expected that only a very small number of interrupts is ever on a VCPU at the same time, this seems like the best approach. For now allocated structs are re-used and held in a linked list. * On the guest side we (later will) have to deal with malicious guests trying to hog Xen with mapping requests for a lot of LPIs, for instance. As the ITS actually uses system memory for storing status information, we use this memory (which the guest has to provide) to naturally limit a guest. For those tables which are page sized (devices, collections (CPUs), LPI properties) we map those pages into Xen, so we can easily access them from the virtual GIC code. Unfortunately the actual interrupt mapping tables are not necessarily page aligned, also can be much smaller than a page, so mapping all of them permanently is fiddly. As ITS commands in need to iterate those tables are pretty rare after all, we for now map them on demand upon emulating a virtual ITS command. * An obvious approach to handling some guest ITS commands would be to propagate them to the host, for instance to map devices and LPIs and to enable or disable LPIs. However this (later with DomU support) will create an attack vector, as a malicious guest could try to fill the host command queue with propagated commands. So in contrast to the previous RFC post this version now completely avoids this situation. For mapping devices and LPIs we rely on this being done via a hypercall prior to the actual guest run. For enabling and disabling LPIs we keep this bit on the virtual side and let LPIs always be enabled on the host side, dealing with the consequences this approach creates. This series is still a draft, with some known and many unknown issues. I made ITS support a Kconfig option, also it is only supported on arm64. This leads to some hideous constructs like an #ifdef'ed header file with empty function stubs, but I guess we can clean this up later in the upstreaming process. There are numerous changes compared to the last post, mainly affecting the now missing ITS command progagation. I also added locking to the "usual suspects" data structures. I picked some low hanging fruits from the review comments. Things I haven't addresses well is the whole memory management, in terms of marking pages r/o for a guest or allocating Xen memory from the proper bucket. This will be addresses with the next post. For now this code happens to boot Dom0 on an ARM fast model with ITS support. I still haven't had the chance to get hold of a Xen supported hardware platform with an ITS yet, so running on real hardware is a bit terra incognita. The code can also be found on the its/rfc-v2 branch here: git://linux-arm.org/xen-ap.git http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/its/rfc-v2 Cheers, Andre Andre Przywara (26): ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT ARM: GICv3: allocate LPI pending and property table ARM: GICv3 ITS: allocate device and collection table ARM: GICv3 ITS: map ITS command buffer ARM: GICv3 ITS: introduce ITS command handling ARM: GICv3 ITS: introduce device mapping ARM: GICv3 ITS: introduce host LPI array ARM: GICv3 ITS: map device and LPIs to the ITS on physdev_op hypercall ARM: GICv3: introduce separate pending_irq structs for LPIs ARM: GICv3: forward pending LPIs to guests ARM: GICv3: enable ITS and LPIs on the host ARM: vGICv3: handle virtual LPI pending and property tables ARM: vGICv3: Handle disabled LPIs ARM: vGICv3: introduce basic ITS emulation bits ARM: vITS: handle CLEAR command ARM: vITS: handle INT command ARM: vITS: handle MAPC command ARM: vITS: handle MAPD command ARM: vITS: handle MAPTI command ARM: vITS: handle MOVI command ARM: vITS: handle DISCARD command ARM: vITS: handle INV command ARM: vITS: handle INVALL command ARM: vITS: create and initialize virtual ITSes for Dom0 ARM: vITS: create ITS subnodes for Dom0 DT ARM: vGIC: advertising LPI support xen/arch/arm/Kconfig | 20 + xen/arch/arm/Makefile | 2 + xen/arch/arm/efi/efi-boot.h | 1 - xen/arch/arm/gic-its.c | 934 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/gic-v3.c | 92 +++- xen/arch/arm/gic.c | 9 +- xen/arch/arm/physdev.c | 24 + xen/arch/arm/vgic-its.c | 842 ++++++++++++++++++++++++++++++++++ xen/arch/arm/vgic-v3.c | 274 +++++++++-- xen/arch/arm/vgic.c | 71 ++- xen/include/asm-arm/bitops.h | 2 + xen/include/asm-arm/cache.h | 4 + xen/include/asm-arm/domain.h | 13 +- xen/include/asm-arm/gic-its.h | 217 +++++++++ xen/include/asm-arm/gic_v3_defs.h | 67 ++- xen/include/asm-arm/irq.h | 8 + xen/include/asm-arm/vgic.h | 15 + 17 files changed, 2558 insertions(+), 37 deletions(-) create mode 100644 xen/arch/arm/gic-its.c create mode 100644 xen/arch/arm/vgic-its.c create mode 100644 xen/include/asm-arm/gic-its.h -- 2.9.0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
