Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-09-02 at 12:39 +0100, Julien Grall wrote: > Hi Ian, > > On 02/09/15 12:30, Ian Campbell wrote: > > > Although, when running on Xen, FreeBSD is creating a Xen console by > > > default with the higher priority. It will be grabbed by FreeBSD as > > > the > > > default one. > > > > ... Linux should be doing the same. The problem is that the existing > > code > > to call add_preferred_console doesn't happen early enough on ARM > > > > Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux > > earlier ages ago, but I think it failed to actually get in. > > The patch is present in Linux 4.2: Not the patch I was thinking of, but since it achieves the end goal another way: Yeehaw! > commit f1118c08ce383b7262f4e6440927bdf4 > Author: Ard Biesheuvel > Date: Wed May 6 14:14:22 2015 + > > xen/arm: allow console=hvc0 to be omitted for guests > > From: Ard Biesheuvel > > This patch registers hvc0 as the preferred console if no console > has been specified explicitly on the kernel command line. > > The purpose is to allow platform agnostic kernels and boot images > (such as distro installers) to boot in a Xen/ARM domU without the > need to modify the command line by hand. > > Signed-off-by: Ard Biesheuvel > Signed-off-by: Stefano Stabellini > Reviewed-by: Julien Grall > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi Ian, On 02/09/15 12:30, Ian Campbell wrote: >> Although, when running on Xen, FreeBSD is creating a Xen console by >> default with the higher priority. It will be grabbed by FreeBSD as the >> default one. > > ... Linux should be doing the same. The problem is that the existing code > to call add_preferred_console doesn't happen early enough on ARM > > Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux > earlier ages ago, but I think it failed to actually get in. The patch is present in Linux 4.2: commit f1118c08ce383b7262f4e6440927bdf4 Author: Ard Biesheuvel Date: Wed May 6 14:14:22 2015 + xen/arm: allow console=hvc0 to be omitted for guests From: Ard Biesheuvel This patch registers hvc0 as the preferred console if no console has been specified explicitly on the kernel command line. The purpose is to allow platform agnostic kernels and boot images (such as distro installers) to boot in a Xen/ARM domU without the need to modify the command line by hand. Signed-off-by: Ard Biesheuvel Signed-off-by: Stefano Stabellini Reviewed-by: Julien Grall -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 13:20 +0100, Julien Grall wrote: > On 12/08/15 11:36, Andrew Turner wrote: > > Would it be possible to add a stdout property and node for the hvc0 > > device? It would help FreeBSD as we use this to find the kernel > > console. We check for the stdout-path, linux,stdout-path, stdout, > > stdin-path, and stdin properties, in that order, with the first > > property selected as the console. If none are found we fall back to > > searching for a serial0 device. You can see how we find the device at > > [1]. > > the stdout-path property is used by Xen in order to get the UART. The > property will be removed from the device tree passed to DOM0. > > The Xen console is not UART a driver so having a property with "hvc" > wouldn't help here. If it were helpful we could of course create some sort of dummy node or do something else to make it work. But it really shouldn't be necessary because... > Although, when running on Xen, FreeBSD is creating a Xen console by > default with the higher priority. It will be grabbed by FreeBSD as the > default one. ... Linux should be doing the same. The problem is that the existing code to call add_preferred_console doesn't happen early enough on ARM Someone (you? Ard? Stefano?) has a patch to move Xen detection in Linux earlier ages ago, but I think it failed to actually get in. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 13:11 +0100, Julien Grall wrote: > On 12/08/15 12:23, Ian Campbell wrote: > > Strictly it is considered a separate thing, much like loader.efi, despite > > where it lives e.g. it is self contained and not allowed to call into the > > kernel proper except via the formal interface provided for the hand-off. > > > > That might seem like semantic quibbling, but I just want to clarify that > > the Linux and BSD approaches here are basically the same. > > > > Given that these device tree bindings are really just Linux's equivalent of > > the "a format the kernel understands" which BSD uses as described above. I > > don't know what format BSD uses, Linux just happened to have a DTB library > > handy... > > IIRC, on FreeBSD, the loader and the kernel is talking through a custom > format called metadata. There is no modification of device tree by the > loader. Right, and as I say above Linux's equivalent of "a custom format called metadata" is to declare a device tree binding. > Although, I would prefer to see a common interface between Xen and DOM0 > rather than implementing a custom one for each OS we will support. Agreed. > I have to think how everything will work together. AFAIK, on x86, the > loader is loading Xen and the FreeBSD kernel in the memory. The metadata > necessary for the kernel is passed as a multiboot entry. > > I will speak with Royger to see what we can do here. > > In the meantime, I think we should drop "linux," when we standardize > them to show that they are generic and not linux specific. "we" here would be broader than just Xen of course. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/17 18:36, Roger Pau Monné wrote: > El 11/08/15 a les 16.19, Ian Campbell ha escrit: >> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >>> This document is going to explain the design details of Xen booting with >>> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are >>> welcome. >> >> Some small subsets of this seem like they might overlap with what will be >> required for PVH on x86 (a new x86 guest mode not dissimilar to the sole >> ARM guest mode). If so then it would be preferable IMHO if PVH x86 could >> use the same interfaces. >> >> I've trimmed the quotes to just those bits and CCd some of the PVH people >> (Boris and Roger[0]) in case they have any thoughts. >> >> Actually, having done the trimming there is only one such bit: >> >> [...] >>> 4. Map MMIO regions >>> --- >>> Register a bus_notifier for platform and amba bus in Linux. Add a new >>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >>> device is newly added, then call hypercall XENMEM_add_to_physmap to map >>> the mmio regions. >> >> Ian. >> >> [0] Roger is away for a week or so, but I'm expect feedback to be of the >> "we could use one extra field" type rather than "this needs to be done some >> totally different way for x86/PVH" (in which case we wouldn't want to share >> the interface anyway I suppose) so need to block on awaiting that feedback. > > This looks right to me. AFAICT this new memory space > (XENMAPSPACE_dev_mmio) will only be available to the hardware domain on > x86. I expect that for DomUs the toolstack will already map the > appropriate MMIO regions when creating the domain if there are > pass-through devices assigned, not sure if that's also the plan on ARM. > IMHO this document should also list the usage of the hypercall parameters: > > - space: XENMAPSPACE_dev_mmio. > - idxs: native physical addresses. > - gpfns: guest physical addresses where the mapping should appear. > > This is quite obvious but I think it's worth spelling it out. > Will add. Thanks :) -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
El 11/08/15 a les 16.19, Ian Campbell ha escrit: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >> This document is going to explain the design details of Xen booting with >> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are >> welcome. > > Some small subsets of this seem like they might overlap with what will be > required for PVH on x86 (a new x86 guest mode not dissimilar to the sole > ARM guest mode). If so then it would be preferable IMHO if PVH x86 could > use the same interfaces. > > I've trimmed the quotes to just those bits and CCd some of the PVH people > (Boris and Roger[0]) in case they have any thoughts. > > Actually, having done the trimming there is only one such bit: > > [...] >> 4. Map MMIO regions >> --- >> Register a bus_notifier for platform and amba bus in Linux. Add a new >> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >> device is newly added, then call hypercall XENMEM_add_to_physmap to map >> the mmio regions. > > Ian. > > [0] Roger is away for a week or so, but I'm expect feedback to be of the > "we could use one extra field" type rather than "this needs to be done some > totally different way for x86/PVH" (in which case we wouldn't want to share > the interface anyway I suppose) so need to block on awaiting that feedback. This looks right to me. AFAICT this new memory space (XENMAPSPACE_dev_mmio) will only be available to the hardware domain on x86. I expect that for DomUs the toolstack will already map the appropriate MMIO regions when creating the domain if there are pass-through devices assigned, not sure if that's also the plan on ARM. IMHO this document should also list the usage of the hypercall parameters: - space: XENMAPSPACE_dev_mmio. - idxs: native physical addresses. - gpfns: guest physical addresses where the mapping should appear. This is quite obvious but I think it's worth spelling it out. Roger. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/12 17:11, Julien Grall wrote: On 12/08/2015 08:22, Shannon Zhao wrote: Hi, Hi Shannon, It's not part of the design discussion and we are avoiding to mix discussion. Can you please create another thread (or at least renaming the subject)? I'm working on re-spinning this patchset while encountering a werid problem about xzalloc_bytes. Since I need to copy some ACPI tables, I need to allocate some memory for it. So there are a few places calling xzalloc_bytes. And it fails at the fifth one. The log is shown as following: Do you copy data in the newly allocated memory between 2 xzalloc_bytes? No, I just use xzalloc_bytes to allocate some place and copy ACPI to the allocated place, modify the content, then call raw_copy_to_guest_flush_dcache to copy the modified tables to guest memory. The only thing I have in mind based on the log below is your are overriding the metadata of the memory allocator. (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0008fa315000 (XEN) Allocating 1:1 mappings totalling 256MB for dom0: (XEN) BANK[0] 0x009000-0x00a000 (256MB) (XEN) Grant table range: 0x0087e0-0x0087e5b000 (XEN) Loading zImage from 0008fa315000 to 9008-909e0ec8 (XEN) Hypervisor Trap. HSR=0x9644 EC=0x25 IL=1 Syndrome=0x44 (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) [ Xen-4.6-unstable arm64 debug=y Not tainted ] (XEN) CPU:0 (XEN) PC: 00238b78 xmem_pool_alloc+0x348/0x4b0 (XEN) LR: 00238960 (XEN) SP: 002bf4e0 (XEN) CPSR: 2249 MODE:64-bit EL2h (Hypervisor, handler) (XEN) Xen call trace: (XEN)[<00238b78>] xmem_pool_alloc+0x348/0x4b0 (PC) (XEN)[<00238960>] xmem_pool_alloc+0x130/0x4b0 (LR) (XEN)[<00239100>] _xmalloc+0xf4/0x290 (XEN)[<002392b0>] _xzalloc+0x14/0x38 (XEN)[<00245430>] construct_dom0+0xbc0/0x14cc (XEN)[<0028c4c4>] start_xen+0xbf4/0xcb0 (XEN)[<0020060c>] paging+0x84/0xbc (XEN) (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) (XEN) (XEN) (XEN) Reboot in five seconds... Regards, -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, 14 Aug 2015, Shannon Zhao wrote: > On 2015/8/13 18:29, Christoffer Dall wrote: > > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: > >> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: > >>> > > > > > > > > For example it is only natural for the kernel to try to use the > > > > > > GIC > > > > > > hyp > > > > > > functionalities if they are described, while actually they are > > > > > > not > > > > > > emulated by Xen at all. > > > > > > > > See Ian's earlier reply: It can also be considered natural for it > > > > to > > > > be aware that when run in EL2 to not use EL1 functionality. > >> > > >> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. > >> > > >>> > > It is not just about the GIC Hyp functionalities. > >> > > >> > What else is there which is not subject to this logic? Timers are too, it > >> > even applies to IOMMU's which have both stage1 and stage2 bits. > >> > > >> > BTW, I think kernels _already_ need to deal with a lot of this because in > >> > reality nobody modifies the DTB when they use a firmware which launches > >> > the > >> > kernel in EL1. IOW I think the kernel is already aware of which resources > >> > can be used by which privilege level. > >> > > > Yes, for resources specific to EL2 I believe that is indeed the case > > (the GIC driver doesn't look at the hypervisor control register address, > > and KVM does not even get that far if you're not booted in EL2, and the > > timer only uses the virtual timer if not booted in EL2 - we never > > attempt to use the hyp timer until Marc's VHE patches land, but they > > also depend on being booted in hyp mode). > > > > However, what about for other resources? Having code somewhere that > > says "hide this random piece of hardware if you're Xen dom0" sounds > > awful to me. I know it's only the serial port right now, but still. > > If we remove the entry of SPCR table from XSDT table, would it work for > Dom0 to ignore the uart? And it doesn't need the STAO table? I don't think that is enough because the very same uart is likely to be described in the dsdt too. I am pretty sure that we need the STAO. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/13 18:29, Christoffer Dall wrote: > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: >> > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: >>> > > > > > > > For example it is only natural for the kernel to try to use the > > > > > GIC > > > > > hyp > > > > > functionalities if they are described, while actually they are not > > > > > emulated by Xen at all. > > > > > > See Ian's earlier reply: It can also be considered natural for it to > > > be aware that when run in EL2 to not use EL1 functionality. >> > >> > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. >> > >>> > > It is not just about the GIC Hyp functionalities. >> > >> > What else is there which is not subject to this logic? Timers are too, it >> > even applies to IOMMU's which have both stage1 and stage2 bits. >> > >> > BTW, I think kernels _already_ need to deal with a lot of this because in >> > reality nobody modifies the DTB when they use a firmware which launches the >> > kernel in EL1. IOW I think the kernel is already aware of which resources >> > can be used by which privilege level. >> > > Yes, for resources specific to EL2 I believe that is indeed the case > (the GIC driver doesn't look at the hypervisor control register address, > and KVM does not even get that far if you're not booted in EL2, and the > timer only uses the virtual timer if not booted in EL2 - we never > attempt to use the hyp timer until Marc's VHE patches land, but they > also depend on being booted in hyp mode). > > However, what about for other resources? Having code somewhere that > says "hide this random piece of hardware if you're Xen dom0" sounds > awful to me. I know it's only the serial port right now, but still. If we remove the entry of SPCR table from XSDT table, would it work for Dom0 to ignore the uart? And it doesn't need the STAO table? -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 2015-08-13 at 13:08 +0100, Julien Grall wrote: > > The MADT has to be modified in order to let DOM0 knows about his CPU > topology. If ever need the original MADT for power management, then we > should find a Xen specific way to do it (even if it's passing the > orginal MADT as a backup). Right, I was about to say something along these lines, the default set of tables should be the ones which are expected for the guest's (including dom0's) actual configuration and own use. Any additional magic stuff (like host-cpufreq in dom0) already needs to do Xen specific magic to bring itself up in a world which doesn't match that and should request whatever real host information it needs itself, not expect to find it in the "proper" places. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 13/08/15 12:55, Jan Beulich wrote: On 13.08.15 at 13:00, wrote: >> On 13/08/15 11:54, Jan Beulich wrote: >> On 13.08.15 at 12:48, wrote: On 2015/8/13 18:29, Christoffer Dall wrote: > However, what about for other resources? Having code somewhere that > says "hide this random piece of hardware if you're Xen dom0" sounds > awful to me. I know it's only the serial port right now, but still. > It needs to modify MADT table according to dom0_max_vcpus and modify EFI MMAP table according to dom0_mem. And modify FADT table to set hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. >>> >>> None of this should be required: >>> >>> Unaltered MADT may (or should I say will) be needed to drive P- or >>> C-states. >> >> We have to alter MADT for different reasons: >> - restricting the number of vCPUs >> - update the CPU bring up method >> - changing the CPUID >> >> Maybe we should pass a backup but we don't support cpufreq driver right >> now and it would need quite a look of work to do it (see [1]). > > I.e. you intend to not even leave open the road there. Please beware that currently porting a guest to Xen on ARM is only a couple of hundreds lines which is mainly detecting Xen and initializing the Xen code. Other than that everything is the same as booting on baremetal. I don't take into account the drivers as they usually make usage of the kernel framework and don't touch boot code. This is a strong advantage for Xen because developer would not have to hack the internal OS in order to get running on Xen. We are trying to aim the same for ACPI. Any change compare to baremetal boot will result to more work for the OS developer. The MADT has to be modified in order to let DOM0 knows about his CPU topology. If ever need the original MADT for power management, then we should find a Xen specific way to do it (even if it's passing the orginal MADT as a backup). Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 13:00, wrote: > On 13/08/15 11:54, Jan Beulich wrote: > On 13.08.15 at 12:48, wrote: >>> On 2015/8/13 18:29, Christoffer Dall wrote: However, what about for other resources? Having code somewhere that says "hide this random piece of hardware if you're Xen dom0" sounds awful to me. I know it's only the serial port right now, but still. >>> It needs to modify MADT table according to dom0_max_vcpus and modify EFI >>> MMAP table according to dom0_mem. And modify FADT table to set >>> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. >> >> None of this should be required: >> >> Unaltered MADT may (or should I say will) be needed to drive P- or >> C-states. > > We have to alter MADT for different reasons: > - restricting the number of vCPUs > - update the CPU bring up method > - changing the CPUID > > Maybe we should pass a backup but we don't support cpufreq driver right > now and it would need quite a look of work to do it (see [1]). I.e. you intend to not even leave open the road there. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 13 Aug 2015, Julien Grall wrote: > On 13/08/15 11:54, Jan Beulich wrote: > On 13.08.15 at 12:48, wrote: > >> On 2015/8/13 18:29, Christoffer Dall wrote: > >>> However, what about for other resources? Having code somewhere that > >>> says "hide this random piece of hardware if you're Xen dom0" sounds > >>> awful to me. I know it's only the serial port right now, but still. > >>> > >> It needs to modify MADT table according to dom0_max_vcpus and modify EFI > >> MMAP table according to dom0_mem. And modify FADT table to set > >> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. > > > > None of this should be required: > > > > Unaltered MADT may (or should I say will) be needed to drive P- or > > C-states. > > We have to alter MADT for different reasons: > - restricting the number of vCPUs > - update the CPU bring up method > - changing the CPUID > > Maybe we should pass a backup but we don't support cpufreq driver right > now and it would need quite a look of work to do it (see [1]). > > > UEFI memmap shouldn't be exposed to Dom0 at all. > > We have to craft a UEFI memmap in order to notify Dom0 where are the RAM > regions. Right. Keep in mind that there are no legacy interfaces to get these info on ARM. All we have is ACPI and EFI. > > Detecting running on Xen should (hopefully) be possible by other > > means for Dom0. > > How there is no cpuid like on ARM. So no possibility to know whether you > are running on Xen or another hypervisor... > > Modifying the FADT is the only viable solution we have today in order to > tell that it's running on Xen. We also have the XENV table. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 13/08/15 11:54, Jan Beulich wrote: On 13.08.15 at 12:48, wrote: >> On 2015/8/13 18:29, Christoffer Dall wrote: >>> However, what about for other resources? Having code somewhere that >>> says "hide this random piece of hardware if you're Xen dom0" sounds >>> awful to me. I know it's only the serial port right now, but still. >>> >> It needs to modify MADT table according to dom0_max_vcpus and modify EFI >> MMAP table according to dom0_mem. And modify FADT table to set >> hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. > > None of this should be required: > > Unaltered MADT may (or should I say will) be needed to drive P- or > C-states. We have to alter MADT for different reasons: - restricting the number of vCPUs - update the CPU bring up method - changing the CPUID Maybe we should pass a backup but we don't support cpufreq driver right now and it would need quite a look of work to do it (see [1]). > UEFI memmap shouldn't be exposed to Dom0 at all. We have to craft a UEFI memmap in order to notify Dom0 where are the RAM regions. > Detecting running on Xen should (hopefully) be possible by other > means for Dom0. How there is no cpuid like on ARM. So no possibility to know whether you are running on Xen or another hypervisor... Modifying the FADT is the only viable solution we have today in order to tell that it's running on Xen. Regards, [1]http://lists.xen.org/archives/html/xen-devel/2014-10/msg02883.html -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 12:48, wrote: > On 2015/8/13 18:29, Christoffer Dall wrote: >> However, what about for other resources? Having code somewhere that >> says "hide this random piece of hardware if you're Xen dom0" sounds >> awful to me. I know it's only the serial port right now, but still. >> > It needs to modify MADT table according to dom0_max_vcpus and modify EFI > MMAP table according to dom0_mem. And modify FADT table to set > hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. None of this should be required: Unaltered MADT may (or should I say will) be needed to drive P- or C-states. UEFI memmap shouldn't be exposed to Dom0 at all. Detecting running on Xen should (hopefully) be possible by other means for Dom0. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/13 18:29, Christoffer Dall wrote: > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: >> On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: >>> > For example it is only natural for the kernel to try to use the GIC > hyp > functionalities if they are described, while actually they are not > emulated by Xen at all. See Ian's earlier reply: It can also be considered natural for it to be aware that when run in EL2 to not use EL1 functionality. >> >> NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. >> >>> It is not just about the GIC Hyp functionalities. >> >> What else is there which is not subject to this logic? Timers are too, it >> even applies to IOMMU's which have both stage1 and stage2 bits. >> >> BTW, I think kernels _already_ need to deal with a lot of this because in >> reality nobody modifies the DTB when they use a firmware which launches the >> kernel in EL1. IOW I think the kernel is already aware of which resources >> can be used by which privilege level. >> > Yes, for resources specific to EL2 I believe that is indeed the case > (the GIC driver doesn't look at the hypervisor control register address, > and KVM does not even get that far if you're not booted in EL2, and the > timer only uses the virtual timer if not booted in EL2 - we never > attempt to use the hyp timer until Marc's VHE patches land, but they > also depend on being booted in hyp mode). > > However, what about for other resources? Having code somewhere that > says "hide this random piece of hardware if you're Xen dom0" sounds > awful to me. I know it's only the serial port right now, but still. > It needs to modify MADT table according to dom0_max_vcpus and modify EFI MMAP table according to dom0_mem. And modify FADT table to set hypervisor_id as "XenVMM" to tell Dom0 that it runs on Xen hypervisor. -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 13/08/15 10:20, Jan Beulich wrote: >> BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I >> think), to hide the IOMMU from the guest? That's another table we would >> want to frob on ARM I think (or it's equivalent, which I think is IORT). > > Eliminating that hack is supposed to be on the VT-d maintainers' > TODO list(s) - Dom0 has no business looking at that table (and its > AMD counterpart already isn't being fiddled with in the same way). ARM SMMU is supporting 2 stage in order to protect the device also by the domain. At some point we will expose it to DOM0 and therefore may need to modify IORT. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 2015-08-13 at 12:29 +0200, Christoffer Dall wrote: > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: > > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: > > > > > > > > For example it is only natural for the kernel to try to use the > > > > > GIC > > > > > hyp > > > > > functionalities if they are described, while actually they are > > > > > not > > > > > emulated by Xen at all. > > > > > > > > See Ian's earlier reply: It can also be considered natural for it > > > > to > > > > be aware that when run in EL2 to not use EL1 functionality. > > > > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. > > > > > It is not just about the GIC Hyp functionalities. > > > > What else is there which is not subject to this logic? Timers are too, > > it > > even applies to IOMMU's which have both stage1 and stage2 bits. > > > > BTW, I think kernels _already_ need to deal with a lot of this because > > in > > reality nobody modifies the DTB when they use a firmware which launches > > the > > kernel in EL1. IOW I think the kernel is already aware of which > > resources > > can be used by which privilege level. > > > Yes, for resources specific to EL2 I believe that is indeed the case > (the GIC driver doesn't look at the hypervisor control register address, > and KVM does not even get that far if you're not booted in EL2, and the > timer only uses the virtual timer if not booted in EL2 - we never > attempt to use the hyp timer until Marc's VHE patches land, but they > also depend on being booted in hyp mode). Right, and I think that's almost always going to be the case by virtue of the architecture. You can't use these resources from EL1 (however you got there) so you need checks. > However, what about for other resources? Having code somewhere that > says "hide this random piece of hardware if you're Xen dom0" sounds > awful to me. I know it's only the serial port right now, but still. Right, that's the only bit I'm aware of that I'm not sure about, which is why I was curious how x86 managed it (I've seen but not digested Jan's answer). At least this has the virtue of being an extra table, and tagging on a new one of those has more limited impact on the tables I think. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 13 Aug 2015, Christoffer Dall wrote: > On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: > > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: > > > > > > > > For example it is only natural for the kernel to try to use the GIC > > > > > hyp > > > > > functionalities if they are described, while actually they are not > > > > > emulated by Xen at all. > > > > > > > > See Ian's earlier reply: It can also be considered natural for it to > > > > be aware that when run in EL2 to not use EL1 functionality. > > > > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. > > > > > It is not just about the GIC Hyp functionalities. > > > > What else is there which is not subject to this logic? Timers are too, it > > even applies to IOMMU's which have both stage1 and stage2 bits. > > > > BTW, I think kernels _already_ need to deal with a lot of this because in > > reality nobody modifies the DTB when they use a firmware which launches the > > kernel in EL1. IOW I think the kernel is already aware of which resources > > can be used by which privilege level. > > > Yes, for resources specific to EL2 I believe that is indeed the case > (the GIC driver doesn't look at the hypervisor control register address, > and KVM does not even get that far if you're not booted in EL2, and the > timer only uses the virtual timer if not booted in EL2 - we never > attempt to use the hyp timer until Marc's VHE patches land, but they > also depend on being booted in hyp mode). > > However, what about for other resources? Having code somewhere that > says "hide this random piece of hardware if you're Xen dom0" sounds > awful to me. I know it's only the serial port right now, but still. I completely agree. Non-EL2 specific resources should defenitely be removed. I can see that EL2 stuff could be left there, but I think that removing it now would avoid us trouble in the future. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, Aug 13, 2015 at 11:22:19AM +0100, Ian Campbell wrote: > On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: > > > > > > For example it is only natural for the kernel to try to use the GIC > > > > hyp > > > > functionalities if they are described, while actually they are not > > > > emulated by Xen at all. > > > > > > See Ian's earlier reply: It can also be considered natural for it to > > > be aware that when run in EL2 to not use EL1 functionality. > > NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. > > > It is not just about the GIC Hyp functionalities. > > What else is there which is not subject to this logic? Timers are too, it > even applies to IOMMU's which have both stage1 and stage2 bits. > > BTW, I think kernels _already_ need to deal with a lot of this because in > reality nobody modifies the DTB when they use a firmware which launches the > kernel in EL1. IOW I think the kernel is already aware of which resources > can be used by which privilege level. > Yes, for resources specific to EL2 I believe that is indeed the case (the GIC driver doesn't look at the hypervisor control register address, and KVM does not even get that far if you're not booted in EL2, and the timer only uses the virtual timer if not booted in EL2 - we never attempt to use the hyp timer until Marc's VHE patches land, but they also depend on being booted in hyp mode). However, what about for other resources? Having code somewhere that says "hide this random piece of hardware if you're Xen dom0" sounds awful to me. I know it's only the serial port right now, but still. -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 2015-08-13 at 11:13 +0100, Stefano Stabellini wrote: > > > > For example it is only natural for the kernel to try to use the GIC > > > hyp > > > functionalities if they are described, while actually they are not > > > emulated by Xen at all. > > > > See Ian's earlier reply: It can also be considered natural for it to > > be aware that when run in EL2 to not use EL1 functionality. NB EL2 == Hyp and EL1 == Kernel, so it's the other way round, FWIW. > It is not just about the GIC Hyp functionalities. What else is there which is not subject to this logic? Timers are too, it even applies to IOMMU's which have both stage1 and stage2 bits. BTW, I think kernels _already_ need to deal with a lot of this because in reality nobody modifies the DTB when they use a firmware which launches the kernel in EL1. IOW I think the kernel is already aware of which resources can be used by which privilege level. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 13 Aug 2015, Jan Beulich wrote: > >>> On 13.08.15 at 11:43, wrote: > > On Thu, 13 Aug 2015, Jan Beulich wrote: > >> >>> On 12.08.15 at 18:18, wrote: > >> > On 12/08/15 16:58, Jan Beulich wrote: > >> > On 12.08.15 at 17:51, wrote: > >> >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: > >> >>> On 07.08.15 at 04:11, wrote: > >> > All these tables will be copied to Dom0 memory except that the reused > >> > tables(DSDT, SPCR, etc) will be mapped to Dom0. > >> > >> Which again seems odd - such tables should be considered MMIO > >> (despite living in RAM), and hence not be part of Dom0's memory > >> assignment. > >> > >> >>> not sure if this applies to the context of your comment, but we had > >> >>> issues before when trying to deal with this data as device memory > >> >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device > >> >>> memory, and so Dom0 would crash at memory abort exceptions during > >> >>> boot, so this really does have to be mapped as RAM to Dom0. If you > >> >>> meant some internal Xen bookkeeping thing, then just ignore me. > >> >> > >> >> Hmm, how would native Linux avoid such unaligned accesses then? > >> > > >> > ACPI table are living on RAM on ARM. So there is no issue with Linux > >> > baremetal. > >> > >> I'm sure our interpretation of the meaning of RAM here differs: > >> While from a physical perspective the tables live in RAM too on > >> x86, from a memory map perspective they don't. And since iirc > >> ACPI implies UEFI, and since UEFI has special memory types > >> for such tables (to prevent the OS from using the space as > >> normal memory), I can't believe this to be different under ARM. > >> > >> > The "problem" is from Xen where we are mapping memory region with Device > >> > attribute. This is because until now we never had a reason to directly > >> > map other thing than MMIO to the domain. > >> > > >> > This could be fixed by adding new helper in Xen to directly map RAM > >> > region. > >> > >> Which would seem to be the correct route. > >> > >> > Nonetheless, we still have to copy some table in Xen in order to modify > >> > them and/or new one. I have in mind the FADT table to set the hypervisor > >> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to > >> > avoid the kernel thinks there is hyp mode available. > >> > >> "Have to"? Or rather "would like to"? As said in another reply to > >> Shannon, I didn't see any rationale spelled out for this fundamental > >> difference to x86. > > > > Just because it was done this way before, it doesn't mean that it is the > > right way of doing it. > > Correct. Which is why I'm not saying outright "no" but asking for > justification. > > > I think is bad design to expose the presence of certain functionalities > > in ACPI tables and then expect that the Dom0 kernel won't use them. > > ACPI tables should describe only and all the hardware which is exposed > > to Dom0. Anything else is a mistake. > > That depends on the perspective you take: Especially for a PV > Dom0 it is quite reasonable for the kernel to be aware of certain > implications from running under a (or the) hypervisor it knows of. > I agree that the perspective may shift when it comes to HVM-like > Dom0 (albeit in the spectrum I'd rather see ARM near PVH, which > again is supposed to be aware). PVH has always been the wrong analogy for ARM guests. Guest kernels boot following native boot paths. They don't require emulation because the ARM architecture is much more flexible and legacy free compared to x86, not because we instructed the kernel to ignore certain pieces of hardware. So they are like PV on HVM without any need for QEMU because there is nothing to emulate. Maybe the right analogy is HVMLite. > > For example it is only natural for the kernel to try to use the GIC hyp > > functionalities if they are described, while actually they are not > > emulated by Xen at all. > > See Ian's earlier reply: It can also be considered natural for it to > be aware that when run in EL2 to not use EL1 functionality. It is not just about the GIC Hyp functionalities. > > I would rather teach Xen to fix the tables now, than later try teach > > every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info > > which are wrong but still passed to them anyway. Moreover the list of > > things to ignore can change over time. It is just a bad ABI to maintain. > > I don't think there's a clear advantage to teaching Xen of new > tables to hide over teaching Dom0 of new tables to ignore - it > much depends on release cycles and such which one would be > more beneficial. There are a couple of very obvious advantages: - there is one Xen, there are many Dom0 kernels - if Xen changes ACPI tables, Dom0 won't be surprised by the fact that actually they don't match the hardware. Keep in mind that we don't have 10 years of PV history on ARM. People still think of this as
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 11:43, wrote: > On Thu, 13 Aug 2015, Jan Beulich wrote: >> >>> On 12.08.15 at 18:18, wrote: >> > On 12/08/15 16:58, Jan Beulich wrote: >> > On 12.08.15 at 17:51, wrote: >> >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: >> >>> On 07.08.15 at 04:11, wrote: >> > All these tables will be copied to Dom0 memory except that the reused >> > tables(DSDT, SPCR, etc) will be mapped to Dom0. >> >> Which again seems odd - such tables should be considered MMIO >> (despite living in RAM), and hence not be part of Dom0's memory >> assignment. >> >> >>> not sure if this applies to the context of your comment, but we had >> >>> issues before when trying to deal with this data as device memory >> >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device >> >>> memory, and so Dom0 would crash at memory abort exceptions during >> >>> boot, so this really does have to be mapped as RAM to Dom0. If you >> >>> meant some internal Xen bookkeeping thing, then just ignore me. >> >> >> >> Hmm, how would native Linux avoid such unaligned accesses then? >> > >> > ACPI table are living on RAM on ARM. So there is no issue with Linux >> > baremetal. >> >> I'm sure our interpretation of the meaning of RAM here differs: >> While from a physical perspective the tables live in RAM too on >> x86, from a memory map perspective they don't. And since iirc >> ACPI implies UEFI, and since UEFI has special memory types >> for such tables (to prevent the OS from using the space as >> normal memory), I can't believe this to be different under ARM. >> >> > The "problem" is from Xen where we are mapping memory region with Device >> > attribute. This is because until now we never had a reason to directly >> > map other thing than MMIO to the domain. >> > >> > This could be fixed by adding new helper in Xen to directly map RAM region. >> >> Which would seem to be the correct route. >> >> > Nonetheless, we still have to copy some table in Xen in order to modify >> > them and/or new one. I have in mind the FADT table to set the hypervisor >> > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to >> > avoid the kernel thinks there is hyp mode available. >> >> "Have to"? Or rather "would like to"? As said in another reply to >> Shannon, I didn't see any rationale spelled out for this fundamental >> difference to x86. > > Just because it was done this way before, it doesn't mean that it is the > right way of doing it. Correct. Which is why I'm not saying outright "no" but asking for justification. > I think is bad design to expose the presence of certain functionalities > in ACPI tables and then expect that the Dom0 kernel won't use them. > ACPI tables should describe only and all the hardware which is exposed > to Dom0. Anything else is a mistake. That depends on the perspective you take: Especially for a PV Dom0 it is quite reasonable for the kernel to be aware of certain implications from running under a (or the) hypervisor it knows of. I agree that the perspective may shift when it comes to HVM-like Dom0 (albeit in the spectrum I'd rather see ARM near PVH, which again is supposed to be aware). > For example it is only natural for the kernel to try to use the GIC hyp > functionalities if they are described, while actually they are not > emulated by Xen at all. See Ian's earlier reply: It can also be considered natural for it to be aware that when run in EL2 to not use EL1 functionality. > I would rather teach Xen to fix the tables now, than later try teach > every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info > which are wrong but still passed to them anyway. Moreover the list of > things to ignore can change over time. It is just a bad ABI to maintain. I don't think there's a clear advantage to teaching Xen of new tables to hide over teaching Dom0 of new tables to ignore - it much depends on release cycles and such which one would be more beneficial. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 13 Aug 2015, Jan Beulich wrote: > >>> On 12.08.15 at 18:18, wrote: > > On 12/08/15 16:58, Jan Beulich wrote: > > On 12.08.15 at 17:51, wrote: > >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: > >>> On 07.08.15 at 04:11, wrote: > > All these tables will be copied to Dom0 memory except that the reused > > tables(DSDT, SPCR, etc) will be mapped to Dom0. > > Which again seems odd - such tables should be considered MMIO > (despite living in RAM), and hence not be part of Dom0's memory > assignment. > > >>> not sure if this applies to the context of your comment, but we had > >>> issues before when trying to deal with this data as device memory > >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device > >>> memory, and so Dom0 would crash at memory abort exceptions during > >>> boot, so this really does have to be mapped as RAM to Dom0. If you > >>> meant some internal Xen bookkeeping thing, then just ignore me. > >> > >> Hmm, how would native Linux avoid such unaligned accesses then? > > > > ACPI table are living on RAM on ARM. So there is no issue with Linux > > baremetal. > > I'm sure our interpretation of the meaning of RAM here differs: > While from a physical perspective the tables live in RAM too on > x86, from a memory map perspective they don't. And since iirc > ACPI implies UEFI, and since UEFI has special memory types > for such tables (to prevent the OS from using the space as > normal memory), I can't believe this to be different under ARM. > > > The "problem" is from Xen where we are mapping memory region with Device > > attribute. This is because until now we never had a reason to directly > > map other thing than MMIO to the domain. > > > > This could be fixed by adding new helper in Xen to directly map RAM region. > > Which would seem to be the correct route. > > > Nonetheless, we still have to copy some table in Xen in order to modify > > them and/or new one. I have in mind the FADT table to set the hypervisor > > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to > > avoid the kernel thinks there is hyp mode available. > > "Have to"? Or rather "would like to"? As said in another reply to > Shannon, I didn't see any rationale spelled out for this fundamental > difference to x86. Just because it was done this way before, it doesn't mean that it is the right way of doing it. I think is bad design to expose the presence of certain functionalities in ACPI tables and then expect that the Dom0 kernel won't use them. ACPI tables should describe only and all the hardware which is exposed to Dom0. Anything else is a mistake. For example it is only natural for the kernel to try to use the GIC hyp functionalities if they are described, while actually they are not emulated by Xen at all. I would rather teach Xen to fix the tables now, than later try teach every possible Dom0 kernel (Linux, FreeBSD, etc) to ignore a set of info which are wrong but still passed to them anyway. Moreover the list of things to ignore can change over time. It is just a bad ABI to maintain. In terms of code we could share in Linux between Xen x86 and Xen ARM regarding dealing with ACPI info we want to ignore, unfortunately there isn't much of it, because we are mostly talking about new ARM specific tables here (GIC, arch_timer, etc). ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 11:05, wrote: > The other reason for modification is to hide the Xen console device (i.e. > one UART) from the dom0 kernel, since it is unusable. How does that work on > x86? Do you just not bother and you expect the admin to arrange the > configuration to work or is there some other trick? When it's I/O port based we simply disallow access to the ports (i.e. Dom0 reads return all ones, Dom0 writes get dropped). When it's MMIO based (which iirc became an option only in the not so distant past) we can't always do that, since there could be other devices on the same MMIO page. In any event we then pci_hide_devices() or pci_ro_device() the device (see ns16550_init_postirq()) to limit the damage Dom0 can do. > BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I > think), to hide the IOMMU from the guest? That's another table we would > want to frob on ARM I think (or it's equivalent, which I think is IORT). Eliminating that hack is supposed to be on the VT-d maintainers' TODO list(s) - Dom0 has no business looking at that table (and its AMD counterpart already isn't being fiddled with in the same way). Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, 2015-08-13 at 00:41 -0600, Jan Beulich wrote: > > > > Nonetheless, we still have to copy some table in Xen in order to modify > > them and/or new one. I have in mind the FADT table to set the hypervisor > > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to > > avoid the kernel thinks there is hyp mode available. > > "Have to"? Or rather "would like to"? As said in another reply to > Shannon, I didn't see any rationale spelled out for this fundamental > difference to x86. I think the fundamental difference is that h/w virt features are not (always) "discoverable" through h/w interfaces on ARM whereas they are visible in e.g. cpuid on x86 and not described in ACPI (x86 is only just gaining hardware support for virtualised interrupts so perhaps this will change? I think it doesn't have any hypervisor dedicated timer sources). So on ARM the firmware tables contain things like the additional virtualisation register regions in the interrupt controller description and the hypervisor timer interrupt in the timer block description. So we would like to hide these from dom0. Perhaps instead we should teach dom0 to notice that it was launched in Kernel mode rather than Hyp mode (which is detectable) and therefore ignore these unusable resources. Now you mention it that does sound sensible and I imagine is even already (close to) true for a Linux kernel to handle e.g. older firmware with newer device tree. The other reason for modification is to hide the Xen console device (i.e. one UART) from the dom0 kernel, since it is unusable. How does that work on x86? Do you just not bother and you expect the admin to arrange the configuration to work or is there some other trick? BTW, IIRC x86 does modify at least one ACPI table which is the DMAR (I think), to hide the IOMMU from the guest? That's another table we would want to frob on ARM I think (or it's equivalent, which I think is IORT). Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 10:01, wrote: > On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich wrote: > On 12.08.15 at 18:18, wrote: >>> Nonetheless, we still have to copy some table in Xen in order to modify >>> them and/or new one. I have in mind the FADT table to set the hypervisor >>> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to >>> avoid the kernel thinks there is hyp mode available. >> >> "Have to"? Or rather "would like to"? As said in another reply to >> Shannon, I didn't see any rationale spelled out for this fundamental >> difference to x86. >> > So the suggestion is to just edit the tables in place rather than > copying them and relying on Xen having already read whatever > 'original' information it needs before modifying them? And just to clarify / amend my reply just sent: I also don't see why you would need to expose the System Table to Dom0. Did anyone of you look at how we draw the boundary between Xen and Dom0 on x86? If not, I'd suggest doing so. If so, I'd like to ask for proper reasoning for any item of data exposed to Dom0 beyond what we currently expose. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 13.08.15 at 10:01, wrote: > On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich wrote: > On 12.08.15 at 18:18, wrote: >>> Nonetheless, we still have to copy some table in Xen in order to modify >>> them and/or new one. I have in mind the FADT table to set the hypervisor >>> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to >>> avoid the kernel thinks there is hyp mode available. >> >> "Have to"? Or rather "would like to"? As said in another reply to >> Shannon, I didn't see any rationale spelled out for this fundamental >> difference to x86. >> > So the suggestion is to just edit the tables in place rather than > copying them and relying on Xen having already read whatever > 'original' information it needs before modifying them? No, the suggestion is to leave the tables alone and teach Dom0 to ignore parts it has no interest in. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Thu, Aug 13, 2015 at 8:41 AM, Jan Beulich wrote: On 12.08.15 at 18:18, wrote: >> On 12/08/15 16:58, Jan Beulich wrote: >> On 12.08.15 at 17:51, wrote: On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: On 07.08.15 at 04:11, wrote: >> All these tables will be copied to Dom0 memory except that the reused >> tables(DSDT, SPCR, etc) will be mapped to Dom0. > > Which again seems odd - such tables should be considered MMIO > (despite living in RAM), and hence not be part of Dom0's memory > assignment. > not sure if this applies to the context of your comment, but we had issues before when trying to deal with this data as device memory (MMIO), because ARM doesn't allow unaligned accesses etc. on device memory, and so Dom0 would crash at memory abort exceptions during boot, so this really does have to be mapped as RAM to Dom0. If you meant some internal Xen bookkeeping thing, then just ignore me. >>> >>> Hmm, how would native Linux avoid such unaligned accesses then? >> >> ACPI table are living on RAM on ARM. So there is no issue with Linux >> baremetal. > > I'm sure our interpretation of the meaning of RAM here differs: > While from a physical perspective the tables live in RAM too on > x86, from a memory map perspective they don't. And since iirc > ACPI implies UEFI, and since UEFI has special memory types > for such tables (to prevent the OS from using the space as > normal memory), I can't believe this to be different under ARM. > >> The "problem" is from Xen where we are mapping memory region with Device >> attribute. This is because until now we never had a reason to directly >> map other thing than MMIO to the domain. >> >> This could be fixed by adding new helper in Xen to directly map RAM region. > > Which would seem to be the correct route. > >> Nonetheless, we still have to copy some table in Xen in order to modify >> them and/or new one. I have in mind the FADT table to set the hypervisor >> field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to >> avoid the kernel thinks there is hyp mode available. > > "Have to"? Or rather "would like to"? As said in another reply to > Shannon, I didn't see any rationale spelled out for this fundamental > difference to x86. > So the suggestion is to just edit the tables in place rather than copying them and relying on Xen having already read whatever 'original' information it needs before modifying them? Is there ever a case where the tables live in ROM or shouldn't be modified for any other reason? -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 12.08.15 at 18:18, wrote: > On 12/08/15 16:58, Jan Beulich wrote: > On 12.08.15 at 17:51, wrote: >>> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: >>> On 07.08.15 at 04:11, wrote: > All these tables will be copied to Dom0 memory except that the reused > tables(DSDT, SPCR, etc) will be mapped to Dom0. Which again seems odd - such tables should be considered MMIO (despite living in RAM), and hence not be part of Dom0's memory assignment. >>> not sure if this applies to the context of your comment, but we had >>> issues before when trying to deal with this data as device memory >>> (MMIO), because ARM doesn't allow unaligned accesses etc. on device >>> memory, and so Dom0 would crash at memory abort exceptions during >>> boot, so this really does have to be mapped as RAM to Dom0. If you >>> meant some internal Xen bookkeeping thing, then just ignore me. >> >> Hmm, how would native Linux avoid such unaligned accesses then? > > ACPI table are living on RAM on ARM. So there is no issue with Linux > baremetal. I'm sure our interpretation of the meaning of RAM here differs: While from a physical perspective the tables live in RAM too on x86, from a memory map perspective they don't. And since iirc ACPI implies UEFI, and since UEFI has special memory types for such tables (to prevent the OS from using the space as normal memory), I can't believe this to be different under ARM. > The "problem" is from Xen where we are mapping memory region with Device > attribute. This is because until now we never had a reason to directly > map other thing than MMIO to the domain. > > This could be fixed by adding new helper in Xen to directly map RAM region. Which would seem to be the correct route. > Nonetheless, we still have to copy some table in Xen in order to modify > them and/or new one. I have in mind the FADT table to set the hypervisor > field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to > avoid the kernel thinks there is hyp mode available. "Have to"? Or rather "would like to"? As said in another reply to Shannon, I didn't see any rationale spelled out for this fundamental difference to x86. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 12/08/15 16:58, Jan Beulich wrote: On 12.08.15 at 17:51, wrote: >> On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: >> On 07.08.15 at 04:11, wrote: All these tables will be copied to Dom0 memory except that the reused tables(DSDT, SPCR, etc) will be mapped to Dom0. >>> >>> Which again seems odd - such tables should be considered MMIO >>> (despite living in RAM), and hence not be part of Dom0's memory >>> assignment. >>> >> not sure if this applies to the context of your comment, but we had >> issues before when trying to deal with this data as device memory >> (MMIO), because ARM doesn't allow unaligned accesses etc. on device >> memory, and so Dom0 would crash at memory abort exceptions during >> boot, so this really does have to be mapped as RAM to Dom0. If you >> meant some internal Xen bookkeeping thing, then just ignore me. > > Hmm, how would native Linux avoid such unaligned accesses then? ACPI table are living on RAM on ARM. So there is no issue with Linux baremetal. The "problem" is from Xen where we are mapping memory region with Device attribute. This is because until now we never had a reason to directly map other thing than MMIO to the domain. This could be fixed by adding new helper in Xen to directly map RAM region. Nonetheless, we still have to copy some table in Xen in order to modify them and/or new one. I have in mind the FADT table to set the hypervisor field and hiding the hypervisor specific data (GIC hyp, timer hyp...) to avoid the kernel thinks there is hyp mode available. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 12 Aug 2015 10:21:55 +0100 Julien Grall wrote: > Hi, > > (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64) > > On 12/08/2015 09:52, Ian Campbell wrote: > > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: > >> Hi Julien, > >> > >> On 2015/8/12 0:19, Julien Grall wrote: > >>> Hi Shannon, > >>> > >>> On 07/08/15 03:11, Shannon Zhao wrote: > 2. Create minimal DT to pass required information to Dom0 > -- > The minimal DT mainly passes Dom0 bootargs, address and size of > initrd > (if available), address and size of uefi system table, address > and size > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > An example of the minimal DT: > / { > #address-cells = <2>; > #size-cells = <1>; > chosen { > bootargs = "kernel=Image console=hvc0 > earlycon=pl011,0x1c09 > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > linux,initrd-start = <0x>; > linux,initrd-end = <0x>; > linux,uefi-system-table = <0x>; > linux,uefi-mmap-start = <0x>; > linux,uefi-mmap-size = <0x>; > linux,uefi-mmap-desc-size = <0x>; > linux,uefi-mmap-desc-ver = <0x>; > }; Would it be possible to add a stdout property and node for the hvc0 device? It would help FreeBSD as we use this to find the kernel console. We check for the stdout-path, linux,stdout-path, stdout, stdin-path, and stdin properties, in that order, with the first property selected as the console. If none are found we fall back to searching for a serial0 device. You can see how we find the device at [1]. > }; > > For details loook at > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi. > txt > >>> > >>> AFAICT, the device tree properties in this documentation are only > >>> used in order to communicate between the UEFI stub and Linux. > >>> > >>> This means that those properties are not standardize and can > >>> change at any time by Linux folks. They don't even live in > >>> Documentation/devicetree/ > >>> > >>> I would also expect to see the same needs for FreeBSD running as > >>> DOM0 with ACPI. > >>> > >> I'm not very clear about how FreeBSD communicates with UEFI. And > >> when booting with DT, how does FreeBSD communicate with UEFI? Not > >> through these properties? FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows how to communicate with it. It then loads the kernel and reads any important data the kernel may need. Finally it puts this data into a format the kernel understands, exits the boot services, and boots the kernel. The kernel never communicates with UEFI, we have loaded any data we need (however this may change in the future). In the case of the memory may loader.efi calls GetMemoryMap from EFI_BOOT_SERVICES. It then passes this data directly to the kernel for the kernel to parse in the early boot code. > > > > These properties are in effect a Linux internal interface defined > > between the "Linux UEFI stub" and the "Linux kernel proper". The > > stub and the kernel are notionally separate entities, although they > > are in the same tree etc there is a well defined transition/entry > > point between the two. Since they are in the same tree even though > > they are in theory "separate" I expect they will tend to co-evolve. > > > > IIRC we discussed with some of the maintainers (at Connect?) making > > this a more formal interface, i.e. exposing the entry point to > > "Linux kernel proper" which understands these properties to other > > than just the "Linux UEFI stub" specifically to external entities > > such as Xen. > > > > Probably part of this work needs to formalise that, such as by > > moving this binding into the proper external bindings dir. > > > > At which point BSD can (hopefully!) choose to support the same > > interface. What are the advantages of these bindings over the existing UEFI calls to get the memory map? Andrew [1] https://svnweb.freebsd.org/base/head/sys/dev/uart/uart_cpu_fdt.c?revision=281438&view=markup#l127 -- ABT Systems Ltd Unit 11, Hove Business Centre, Fonthill Road, Hove, BN3 6HA Registered in England and Wales, No. 9285513 ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 12.08.15 at 17:51, wrote: > On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: > On 07.08.15 at 04:11, wrote: >>> All these tables will be copied to Dom0 memory except that the reused >>> tables(DSDT, SPCR, etc) will be mapped to Dom0. >> >> Which again seems odd - such tables should be considered MMIO >> (despite living in RAM), and hence not be part of Dom0's memory >> assignment. >> > not sure if this applies to the context of your comment, but we had > issues before when trying to deal with this data as device memory > (MMIO), because ARM doesn't allow unaligned accesses etc. on device > memory, and so Dom0 would crash at memory abort exceptions during > boot, so this really does have to be mapped as RAM to Dom0. If you > meant some internal Xen bookkeeping thing, then just ignore me. Hmm, how would native Linux avoid such unaligned accesses then? Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, Aug 12, 2015 at 5:45 PM, Jan Beulich wrote: On 07.08.15 at 04:11, wrote: >> 1. Copy and change some EFI and ACPI tables >> --- > > A key thing I'm missing here is reasoning of why this copying approach > is needed in the first place. Remember that on the x86 side we get > away without any such copying-and-changing. Yet the farther you > diverge from x86's model, the more changes you'll need to common > Xen code in e.g. Linux. > >> All these tables will be copied to Dom0 memory except that the reused >> tables(DSDT, SPCR, etc) will be mapped to Dom0. > > Which again seems odd - such tables should be considered MMIO > (despite living in RAM), and hence not be part of Dom0's memory > assignment. > not sure if this applies to the context of your comment, but we had issues before when trying to deal with this data as device memory (MMIO), because ARM doesn't allow unaligned accesses etc. on device memory, and so Dom0 would crash at memory abort exceptions during boot, so this really does have to be mapped as RAM to Dom0. If you meant some internal Xen bookkeeping thing, then just ignore me. -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 11.08.15 at 16:12, wrote: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >> > [...] >> 3. Dom0 gets grant table and event channel irq information >> --- >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >> it runs on Xen hypervisor. >> >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >> and HVM_PARAM_GNTTAB_SIZE. > > The reason we expose this range is essentially to allow OS authors to take > a short cut by telling them about an IPA range which is unused, so it is > available for remapping the grant table into. On x86 there is a BAR on the > Xen platform PCI device which serves a similar purpose. Ah, this answers a question I just raised, but means the items are mis-named and badly described. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
>>> On 07.08.15 at 04:11, wrote: > 1. Copy and change some EFI and ACPI tables > --- A key thing I'm missing here is reasoning of why this copying approach is needed in the first place. Remember that on the x86 side we get away without any such copying-and-changing. Yet the farther you diverge from x86's model, the more changes you'll need to common Xen code in e.g. Linux. > All these tables will be copied to Dom0 memory except that the reused > tables(DSDT, SPCR, etc) will be mapped to Dom0. Which again seems odd - such tables should be considered MMIO (despite living in RAM), and hence not be part of Dom0's memory assignment. > 3. Dom0 gets grant table and event channel irq information > --- > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that > it runs on Xen hypervisor. > > For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS > and HVM_PARAM_GNTTAB_SIZE. Mind explaining why you need this on ARM when x86 gets away without? Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 12/08/15 11:36, Andrew Turner wrote: > Would it be possible to add a stdout property and node for the hvc0 > device? It would help FreeBSD as we use this to find the kernel > console. We check for the stdout-path, linux,stdout-path, stdout, > stdin-path, and stdin properties, in that order, with the first > property selected as the console. If none are found we fall back to > searching for a serial0 device. You can see how we find the device at > [1]. the stdout-path property is used by Xen in order to get the UART. The property will be removed from the device tree passed to DOM0. The Xen console is not UART a driver so having a property with "hvc" wouldn't help here. Although, when running on Xen, FreeBSD is creating a Xen console by default with the higher priority. It will be grabbed by FreeBSD as the default one. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 12/08/15 12:23, Ian Campbell wrote: > Strictly it is considered a separate thing, much like loader.efi, despite > where it lives e.g. it is self contained and not allowed to call into the > kernel proper except via the formal interface provided for the hand-off. > > That might seem like semantic quibbling, but I just want to clarify that > the Linux and BSD approaches here are basically the same. > > Given that these device tree bindings are really just Linux's equivalent of > the "a format the kernel understands" which BSD uses as described above. I > don't know what format BSD uses, Linux just happened to have a DTB library > handy... IIRC, on FreeBSD, the loader and the kernel is talking through a custom format called metadata. There is no modification of device tree by the loader. Although, I would prefer to see a common interface between Xen and DOM0 rather than implementing a custom one for each OS we will support. I have to think how everything will work together. AFAIK, on x86, the loader is loading Xen and the FreeBSD kernel in the memory. The metadata necessary for the kernel is passed as a multiboot entry. I will speak with Royger to see what we can do here. In the meantime, I think we should drop "linux," when we standardize them to show that they are generic and not linux specific. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 11:48 +0100, Stefano Stabellini wrote: > On Wed, 12 Aug 2015, Andrew Turner wrote: > > On Wed, 12 Aug 2015 10:21:55 +0100 > > Julien Grall wrote: > > > > > Hi, > > > > > > (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64) > > > > > > On 12/08/2015 09:52, Ian Campbell wrote: > > > > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: > > > > > Hi Julien, > > > > > > > > > > On 2015/8/12 0:19, Julien Grall wrote: > > > > > > Hi Shannon, > > > > > > > > > > > > On 07/08/15 03:11, Shannon Zhao wrote: > > > > > > > 2. Create minimal DT to pass required information to Dom0 > > > > > > > -- > > > > > > > The minimal DT mainly passes Dom0 bootargs, address and size > > > > > > > of > > > > > > > initrd > > > > > > > (if available), address and size of uefi system table, > > > > > > > address > > > > > > > and size > > > > > > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc > > > > > > > -ver. > > > > > > > > > > > > > > An example of the minimal DT: > > > > > > > / { > > > > > > > #address-cells = <2>; > > > > > > > #size-cells = <1>; > > > > > > > chosen { > > > > > > > bootargs = "kernel=Image console=hvc0 > > > > > > > earlycon=pl011,0x1c09 > > > > > > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > > > > > > > linux,initrd-start = <0x>; > > > > > > > linux,initrd-end = <0x>; > > > > > > > linux,uefi-system-table = <0x>; > > > > > > > linux,uefi-mmap-start = <0x>; > > > > > > > linux,uefi-mmap-size = <0x>; > > > > > > > linux,uefi-mmap-desc-size = <0x>; > > > > > > > linux,uefi-mmap-desc-ver = <0x>; > > > > > > > }; > > > > Would it be possible to add a stdout property and node for the hvc0 > > device? It would help FreeBSD as we use this to find the kernel > > console. We check for the stdout-path, linux,stdout-path, stdout, > > stdin-path, and stdin properties, in that order, with the first > > property selected as the console. If none are found we fall back to > > searching for a serial0 device. You can see how we find the device at > > [1]. > > > > > > > > > }; > > > > > > > > > > > > > > For details loook at > > > > > > > https://github.com/torvalds/linux/blob/master/Documentation/a > > > > > > > rm/uefi. > > > > > > > txt > > > > > > > > > > > > AFAICT, the device tree properties in this documentation are > > > > > > only > > > > > > used in order to communicate between the UEFI stub and Linux. > > > > > > > > > > > > This means that those properties are not standardize and can > > > > > > change at any time by Linux folks. They don't even live in > > > > > > Documentation/devicetree/ > > > > > > > > > > > > I would also expect to see the same needs for FreeBSD running > > > > > > as > > > > > > DOM0 with ACPI. > > > > > > > > > > > I'm not very clear about how FreeBSD communicates with UEFI. And > > > > > when booting with DT, how does FreeBSD communicate with UEFI? Not > > > > > through these properties? > > > > FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows > > how to communicate with it. It then loads the kernel and reads any > > important data the kernel may need. Finally it puts this data into a > > format the kernel understands, exits the boot services, and boots the > > kernel. The kernel never communicates with UEFI, we have loaded any > > data we need (however this may change in the future). > > > > In the case of the memory may loader.efi calls GetMemoryMap from > > EFI_BOOT_SERVICES. It then passes this data directly to the kernel for > > the kernel to parse in the early boot code. > > > > > > > > > > These properties are in effect a Linux internal interface defined > > > > between the "Linux UEFI stub" and the "Linux kernel proper". The > > > > stub and the kernel are notionally separate entities, although they > > > > are in the same tree etc there is a well defined transition/entry > > > > point between the two. Since they are in the same tree even though > > > > they are in theory "separate" I expect they will tend to co-evolve. > > > > > > > > IIRC we discussed with some of the maintainers (at Connect?) making > > > > this a more formal interface, i.e. exposing the entry point to > > > > "Linux kernel proper" which understands these properties to other > > > > than just the "Linux UEFI stub" specifically to external entities > > > > such as Xen. > > > > > > > > Probably part of this work needs to formalise that, such as by > > > > moving this binding into the proper external bindings dir. > > > > > > > > At which point BSD can (hopefully!) choose to support the same > > > > interface. > > > > What are the advantages of these bindings over the existing UEFI calls > > to get the memory map? > > The UEFI calls couldn't be used because ExitBootServices has already > b
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 12 Aug 2015, Andrew Turner wrote: > On Wed, 12 Aug 2015 10:21:55 +0100 > Julien Grall wrote: > > > Hi, > > > > (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64) > > > > On 12/08/2015 09:52, Ian Campbell wrote: > > > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: > > >> Hi Julien, > > >> > > >> On 2015/8/12 0:19, Julien Grall wrote: > > >>> Hi Shannon, > > >>> > > >>> On 07/08/15 03:11, Shannon Zhao wrote: > > 2. Create minimal DT to pass required information to Dom0 > > -- > > The minimal DT mainly passes Dom0 bootargs, address and size of > > initrd > > (if available), address and size of uefi system table, address > > and size > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > > > An example of the minimal DT: > > / { > > #address-cells = <2>; > > #size-cells = <1>; > > chosen { > > bootargs = "kernel=Image console=hvc0 > > earlycon=pl011,0x1c09 > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > > linux,initrd-start = <0x>; > > linux,initrd-end = <0x>; > > linux,uefi-system-table = <0x>; > > linux,uefi-mmap-start = <0x>; > > linux,uefi-mmap-size = <0x>; > > linux,uefi-mmap-desc-size = <0x>; > > linux,uefi-mmap-desc-ver = <0x>; > > }; > > Would it be possible to add a stdout property and node for the hvc0 > device? It would help FreeBSD as we use this to find the kernel > console. We check for the stdout-path, linux,stdout-path, stdout, > stdin-path, and stdin properties, in that order, with the first > property selected as the console. If none are found we fall back to > searching for a serial0 device. You can see how we find the device at > [1]. > > > }; > > > > For details loook at > > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi. > > txt > > >>> > > >>> AFAICT, the device tree properties in this documentation are only > > >>> used in order to communicate between the UEFI stub and Linux. > > >>> > > >>> This means that those properties are not standardize and can > > >>> change at any time by Linux folks. They don't even live in > > >>> Documentation/devicetree/ > > >>> > > >>> I would also expect to see the same needs for FreeBSD running as > > >>> DOM0 with ACPI. > > >>> > > >> I'm not very clear about how FreeBSD communicates with UEFI. And > > >> when booting with DT, how does FreeBSD communicate with UEFI? Not > > >> through these properties? > > FreeBSD has a tool called loader.efi. It gets loaded by UEFI, and knows > how to communicate with it. It then loads the kernel and reads any > important data the kernel may need. Finally it puts this data into a > format the kernel understands, exits the boot services, and boots the > kernel. The kernel never communicates with UEFI, we have loaded any > data we need (however this may change in the future). > > In the case of the memory may loader.efi calls GetMemoryMap from > EFI_BOOT_SERVICES. It then passes this data directly to the kernel for > the kernel to parse in the early boot code. > > > > > > > These properties are in effect a Linux internal interface defined > > > between the "Linux UEFI stub" and the "Linux kernel proper". The > > > stub and the kernel are notionally separate entities, although they > > > are in the same tree etc there is a well defined transition/entry > > > point between the two. Since they are in the same tree even though > > > they are in theory "separate" I expect they will tend to co-evolve. > > > > > > IIRC we discussed with some of the maintainers (at Connect?) making > > > this a more formal interface, i.e. exposing the entry point to > > > "Linux kernel proper" which understands these properties to other > > > than just the "Linux UEFI stub" specifically to external entities > > > such as Xen. > > > > > > Probably part of this work needs to formalise that, such as by > > > moving this binding into the proper external bindings dir. > > > > > > At which point BSD can (hopefully!) choose to support the same > > > interface. > > What are the advantages of these bindings over the existing UEFI calls > to get the memory map? The UEFI calls couldn't be used because ExitBootServices has already been called. The Linux UEFI stub is a bit like your loader.efi, except that is part of the kernel. These bindings are used to pass data from the Linux UEFI stub to the kernel proper, after ExitBootServices is called. We plan to use the same bindings in Xen to pass data to the Dom0 kernel, again after ExitBootServices is called (by Xen in this case). The question is whether FreeBSD could support these bindings too. ___ Xen-devel mailing lis
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 12 Aug 2015, Ian Campbell wrote: > On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: > > Hi Julien, > > > > On 2015/8/12 0:19, Julien Grall wrote: > > > Hi Shannon, > > > > > > On 07/08/15 03:11, Shannon Zhao wrote: > > > > 2. Create minimal DT to pass required information to Dom0 > > > > -- > > > > The minimal DT mainly passes Dom0 bootargs, address and size of > > > > initrd > > > > (if available), address and size of uefi system table, address and > > > > size > > > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > > > > > > > An example of the minimal DT: > > > > / { > > > > #address-cells = <2>; > > > > #size-cells = <1>; > > > > chosen { > > > > bootargs = "kernel=Image console=hvc0 > > > > earlycon=pl011,0x1c09 > > > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > > > > linux,initrd-start = <0x>; > > > > linux,initrd-end = <0x>; > > > > linux,uefi-system-table = <0x>; > > > > linux,uefi-mmap-start = <0x>; > > > > linux,uefi-mmap-size = <0x>; > > > > linux,uefi-mmap-desc-size = <0x>; > > > > linux,uefi-mmap-desc-ver = <0x>; > > > > }; > > > > }; > > > > > > > > For details loook at > > > > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi. > > > > txt > > > > > > AFAICT, the device tree properties in this documentation are only used > > > in order to communicate between the UEFI stub and Linux. > > > > > > This means that those properties are not standardize and can change at > > > any time by Linux folks. They don't even live in > > > Documentation/devicetree/ > > > > > > I would also expect to see the same needs for FreeBSD running as DOM0 > > > with ACPI. > > > > > I'm not very clear about how FreeBSD communicates with UEFI. And when > > booting with DT, how does FreeBSD communicate with UEFI? Not through > > these properties? > > These properties are in effect a Linux internal interface defined between > the "Linux UEFI stub" and the "Linux kernel proper". The stub and the > kernel are notionally separate entities, although they are in the same tree > etc there is a well defined transition/entry point between the two. Since > they are in the same tree even though they are in theory "separate" I > expect they will tend to co-evolve. > > IIRC we discussed with some of the maintainers (at Connect?) making this a > more formal interface, i.e. exposing the entry point to "Linux kernel > proper" which understands these properties to other than just the "Linux > UEFI stub" specifically to external entities such as Xen. > > Probably part of this work needs to formalise that, such as by moving this > binding into the proper external bindings dir. > > At which point BSD can (hopefully!) choose to support the same interface. CCing Ard, who originally proposed this. Yes, I think it would be good to submit a patch to Linux to formalize the bindings and clear out that the interface has become external. As a reference, this was the original thread: http://marc.info/?l=linux-kernel&m=142362266626403&w=2 ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi, (Cc Andrew Turner who worked on the ACPI port for FreeBSD ARM64) On 12/08/2015 09:52, Ian Campbell wrote: On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: Hi Julien, On 2015/8/12 0:19, Julien Grall wrote: Hi Shannon, On 07/08/15 03:11, Shannon Zhao wrote: 2. Create minimal DT to pass required information to Dom0 -- The minimal DT mainly passes Dom0 bootargs, address and size of initrd (if available), address and size of uefi system table, address and size of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. An example of the minimal DT: / { #address-cells = <2>; #size-cells = <1>; chosen { bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c09 root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; linux,initrd-start = <0x>; linux,initrd-end = <0x>; linux,uefi-system-table = <0x>; linux,uefi-mmap-start = <0x>; linux,uefi-mmap-size = <0x>; linux,uefi-mmap-desc-size = <0x>; linux,uefi-mmap-desc-ver = <0x>; }; }; For details loook at https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi. txt AFAICT, the device tree properties in this documentation are only used in order to communicate between the UEFI stub and Linux. This means that those properties are not standardize and can change at any time by Linux folks. They don't even live in Documentation/devicetree/ I would also expect to see the same needs for FreeBSD running as DOM0 with ACPI. I'm not very clear about how FreeBSD communicates with UEFI. And when booting with DT, how does FreeBSD communicate with UEFI? Not through these properties? These properties are in effect a Linux internal interface defined between the "Linux UEFI stub" and the "Linux kernel proper". The stub and the kernel are notionally separate entities, although they are in the same tree etc there is a well defined transition/entry point between the two. Since they are in the same tree even though they are in theory "separate" I expect they will tend to co-evolve. IIRC we discussed with some of the maintainers (at Connect?) making this a more formal interface, i.e. exposing the entry point to "Linux kernel proper" which understands these properties to other than just the "Linux UEFI stub" specifically to external entities such as Xen. Probably part of this work needs to formalise that, such as by moving this binding into the proper external bindings dir. At which point BSD can (hopefully!) choose to support the same interface. I haven't yet figure out how to exactly boot FreeBSD ARM64 as DOM0 (with both ACPI and DT). But given that the only way to get the memory map on DOM0 with ACPI will be via those properties, we will surely needs them at some point in FreeBSD. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 12/08/2015 08:22, Shannon Zhao wrote: Hi, Hi Shannon, It's not part of the design discussion and we are avoiding to mix discussion. Can you please create another thread (or at least renaming the subject)? I'm working on re-spinning this patchset while encountering a werid problem about xzalloc_bytes. Since I need to copy some ACPI tables, I need to allocate some memory for it. So there are a few places calling xzalloc_bytes. And it fails at the fifth one. The log is shown as following: Do you copy data in the newly allocated memory between 2 xzalloc_bytes? The only thing I have in mind based on the log below is your are overriding the metadata of the memory allocator. (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0008fa315000 (XEN) Allocating 1:1 mappings totalling 256MB for dom0: (XEN) BANK[0] 0x009000-0x00a000 (256MB) (XEN) Grant table range: 0x0087e0-0x0087e5b000 (XEN) Loading zImage from 0008fa315000 to 9008-909e0ec8 (XEN) Hypervisor Trap. HSR=0x9644 EC=0x25 IL=1 Syndrome=0x44 (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) [ Xen-4.6-unstable arm64 debug=y Not tainted ] (XEN) CPU:0 (XEN) PC: 00238b78 xmem_pool_alloc+0x348/0x4b0 (XEN) LR: 00238960 (XEN) SP: 002bf4e0 (XEN) CPSR: 2249 MODE:64-bit EL2h (Hypervisor, handler) (XEN) Xen call trace: (XEN)[<00238b78>] xmem_pool_alloc+0x348/0x4b0 (PC) (XEN)[<00238960>] xmem_pool_alloc+0x130/0x4b0 (LR) (XEN)[<00239100>] _xmalloc+0xf4/0x290 (XEN)[<002392b0>] _xzalloc+0x14/0x38 (XEN)[<00245430>] construct_dom0+0xbc0/0x14cc (XEN)[<0028c4c4>] start_xen+0xbf4/0xcb0 (XEN)[<0020060c>] paging+0x84/0xbc (XEN) (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) (XEN) (XEN) (XEN) Reboot in five seconds... Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 12/08/2015 09:46, Ian Campbell wrote: On Tue, 2015-08-11 at 17:01 +0100, Julien Grall wrote: On 11/08/15 16:19, Ian Campbell wrote: IIRC we talked about it few months ago and you said that using balloon page will split in 4K the 1G/2M mapping done in the stage-2 p2m. Did I? Odd because I'm also of the opinion that alloc_ballooned_pages should operate in chunks of 2M at the hypercall layer and keep any resulting spare 4K pages on a free list to use for future such allocations. That from what I recall from an IRL talk. Anyway, I've looked in my archive to see why we decided to keep the grant table parameters (in Xen ACPI table at this point). We were not sure that the domain as all the key in hand in order to find memory hole. I think it's quite important to not think only about Linux but all other Operating Systems. If we ever require a parameters later, it would mean that the OS won't be able to run as DOM0 on older Xen. Linux is using ballooned page, which means loosing ~128KB It's not "loosing" anything. It is _using_ 128KB for a legitimate data structure needed to support the platform (the fact that it happens to not really be using the underlying RAM is irrelevant here). It's not really inherently any different to "loosing" memory in order to allocate e.g. page tables. If you read my sentence until the end, you would have noticed why I say "loosing". It's because we never give back this memory to Xen due to the 1:1 mapping... The grant table are already allocated in Xen separately which means doubling the size of memory used for DOM0 grant table. Anyway, that's small amount of data and can optimized later using the hotplug. Cheers, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/12 16:47, Ian Campbell wrote: > On Wed, 2015-08-12 at 10:47 +0800, Shannon Zhao wrote: >> >> On 2015/8/11 23:52, Boris Ostrovsky wrote: >>> On 08/11/2015 11:35 AM, Ian Campbell wrote: On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote: > On 08/11/2015 10:21 AM, Ian Campbell wrote: >> On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: >>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: This document is going to explain the design details of Xen booting with ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are welcome. >>> Some small subsets of this seem like they might overlap with >>> what >>> will be >>> required for PVH on x86 (a new x86 guest mode not dissimilar to >>> the >>> sole >>> ARM guest mode). If so then it would be preferable IMHO if PVH >>> x86 >>> could >>> use the same interfaces. >>> >>> I've trimmed the quotes to just those bits and CCd some of the >>> PVH >>> people >>> (Boris and Roger[0]) in case they have any thoughts. >>> >>> Actually, having done the trimming there is only one such bit: >>> >>> [...] 4. Map MMIO regions --- Register a bus_notifier for platform and amba bus in Linux. >> Previously PCI was included in this scheme, which is why I >> thought of >> PVH, >> having missed that PCI wasn't mentioned here now. >> >> Is it handled some other way now or is that just an accidental >> omission? > PCI already has a bus notifier which is probably why it's not > explicitly > called out here. Right, but I was more specifically thinking of the bit which followed regarding the use of the notifier to map the MMIO, which is the more important bit. >>> >>> This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell >>> MMIO is mapped there (for IOMMU, which IIUIC is the main issue here). >> >> Right, at this moment we only add platform and amba bus bus_notifier. >> The PCI device can reuse existing bus_notifier. > > Super. Perhaps add a sentence to that effect so I remember that this > already exists next time? > Ok. Will add. -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 11:04 +0800, Shannon Zhao wrote: > Hi Julien, > > On 2015/8/12 0:19, Julien Grall wrote: > > Hi Shannon, > > > > On 07/08/15 03:11, Shannon Zhao wrote: > > > 2. Create minimal DT to pass required information to Dom0 > > > -- > > > The minimal DT mainly passes Dom0 bootargs, address and size of > > > initrd > > > (if available), address and size of uefi system table, address and > > > size > > > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > > > > > An example of the minimal DT: > > > / { > > > #address-cells = <2>; > > > #size-cells = <1>; > > > chosen { > > > bootargs = "kernel=Image console=hvc0 > > > earlycon=pl011,0x1c09 > > > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > > > linux,initrd-start = <0x>; > > > linux,initrd-end = <0x>; > > > linux,uefi-system-table = <0x>; > > > linux,uefi-mmap-start = <0x>; > > > linux,uefi-mmap-size = <0x>; > > > linux,uefi-mmap-desc-size = <0x>; > > > linux,uefi-mmap-desc-ver = <0x>; > > > }; > > > }; > > > > > > For details loook at > > > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi. > > > txt > > > > AFAICT, the device tree properties in this documentation are only used > > in order to communicate between the UEFI stub and Linux. > > > > This means that those properties are not standardize and can change at > > any time by Linux folks. They don't even live in > > Documentation/devicetree/ > > > > I would also expect to see the same needs for FreeBSD running as DOM0 > > with ACPI. > > > I'm not very clear about how FreeBSD communicates with UEFI. And when > booting with DT, how does FreeBSD communicate with UEFI? Not through > these properties? These properties are in effect a Linux internal interface defined between the "Linux UEFI stub" and the "Linux kernel proper". The stub and the kernel are notionally separate entities, although they are in the same tree etc there is a well defined transition/entry point between the two. Since they are in the same tree even though they are in theory "separate" I expect they will tend to co-evolve. IIRC we discussed with some of the maintainers (at Connect?) making this a more formal interface, i.e. exposing the entry point to "Linux kernel proper" which understands these properties to other than just the "Linux UEFI stub" specifically to external entities such as Xen. Probably part of this work needs to formalise that, such as by moving this binding into the proper external bindings dir. At which point BSD can (hopefully!) choose to support the same interface. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 10:47 +0800, Shannon Zhao wrote: > > On 2015/8/11 23:52, Boris Ostrovsky wrote: > > On 08/11/2015 11:35 AM, Ian Campbell wrote: > > > On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote: > > > > On 08/11/2015 10:21 AM, Ian Campbell wrote: > > > > > On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: > > > > > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > > > > > > > This document is going to explain the design details of Xen > > > > > > > booting > > > > > > > with > > > > > > > ACPI on ARM. Maybe parts of it may not be appropriate. Any > > > > > > > comments > > > > > > > are > > > > > > > welcome. > > > > > > Some small subsets of this seem like they might overlap with > > > > > > what > > > > > > will be > > > > > > required for PVH on x86 (a new x86 guest mode not dissimilar to > > > > > > the > > > > > > sole > > > > > > ARM guest mode). If so then it would be preferable IMHO if PVH > > > > > > x86 > > > > > > could > > > > > > use the same interfaces. > > > > > > > > > > > > I've trimmed the quotes to just those bits and CCd some of the > > > > > > PVH > > > > > > people > > > > > > (Boris and Roger[0]) in case they have any thoughts. > > > > > > > > > > > > Actually, having done the trimming there is only one such bit: > > > > > > > > > > > > [...] > > > > > > > 4. Map MMIO regions > > > > > > > --- > > > > > > > Register a bus_notifier for platform and amba bus in Linux. > > > > > Previously PCI was included in this scheme, which is why I > > > > > thought of > > > > > PVH, > > > > > having missed that PCI wasn't mentioned here now. > > > > > > > > > > Is it handled some other way now or is that just an accidental > > > > > omission? > > > > PCI already has a bus notifier which is probably why it's not > > > > explicitly > > > > called out here. > > > Right, but I was more specifically thinking of the bit which followed > > > regarding the use of the notifier to map the MMIO, which is the more > > > important bit. > > > > This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell > > MMIO is mapped there (for IOMMU, which IIUIC is the main issue here). > > Right, at this moment we only add platform and amba bus bus_notifier. > The PCI device can reuse existing bus_notifier. Super. Perhaps add a sentence to that effect so I remember that this already exists next time? ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Wed, 2015-08-12 at 10:42 +0800, Shannon Zhao wrote: > > On 2015/8/12 0:01, Julien Grall wrote: > > On 11/08/15 16:19, Ian Campbell wrote: > > > > IIRC we talked about it few months ago and you said that using > > > > balloon > > > > page will split in 4K the 1G/2M mapping done in the stage-2 p2m. > > > > > > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages > > > should operate in chunks of 2M at the hypercall layer and keep any > > > resulting spare 4K pages on a free list to use for future such > > > allocations. > > > > That from what I recall from an IRL talk. > > > > Anyway, I've looked in my archive to see why we decided to keep the > > grant table parameters (in Xen ACPI table at this point). We were not > > sure that the domain as all the key in hand in order to find memory > > hole. > > > > I think it's quite important to not think only about Linux but all > > other > > Operating Systems. If we ever require a parameters later, it would mean > > that the OS won't be able to run as DOM0 on older Xen. > > > > Linux is using ballooned page, which means loosing ~128KB (default of > > the grant table on ARM) of memory because we never give back the page > > to > > Xen due the 1:1 mapping. Although I guess this is not a big deal as > > it's > > quite small and Linux, as said by David, will support memory hotplug > > soon. > > > > FreeBSD is using memory hole in the address space so there is no issue > > here. > > > > So I guess we could skip this parameters as 128KB doesn't seem to be a > > big deal. > > > > > IOW it should avoid such shattering where it can. > > > > That would work too. > > > > So use xlated_setup_gnttab_pages for both DT and ACPI? I think we might as well, yes. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Tue, 2015-08-11 at 17:01 +0100, Julien Grall wrote: > On 11/08/15 16:19, Ian Campbell wrote: > > > IIRC we talked about it few months ago and you said that using > > > balloon > > > page will split in 4K the 1G/2M mapping done in the stage-2 p2m. > > > > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages > > should operate in chunks of 2M at the hypercall layer and keep any > > resulting spare 4K pages on a free list to use for future such > > allocations. > > That from what I recall from an IRL talk. > > Anyway, I've looked in my archive to see why we decided to keep the > grant table parameters (in Xen ACPI table at this point). We were not > sure that the domain as all the key in hand in order to find memory hole. > > I think it's quite important to not think only about Linux but all other > Operating Systems. If we ever require a parameters later, it would mean > that the OS won't be able to run as DOM0 on older Xen. > > Linux is using ballooned page, which means loosing ~128KB It's not "loosing" anything. It is _using_ 128KB for a legitimate data structure needed to support the platform (the fact that it happens to not really be using the underlying RAM is irrelevant here). It's not really inherently any different to "loosing" memory in order to allocate e.g. page tables. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi, I'm working on re-spinning this patchset while encountering a werid problem about xzalloc_bytes. Since I need to copy some ACPI tables, I need to allocate some memory for it. So there are a few places calling xzalloc_bytes. And it fails at the fifth one. The log is shown as following: (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0008fa315000 (XEN) Allocating 1:1 mappings totalling 256MB for dom0: (XEN) BANK[0] 0x009000-0x00a000 (256MB) (XEN) Grant table range: 0x0087e0-0x0087e5b000 (XEN) Loading zImage from 0008fa315000 to 9008-909e0ec8 (XEN) Hypervisor Trap. HSR=0x9644 EC=0x25 IL=1 Syndrome=0x44 (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) [ Xen-4.6-unstable arm64 debug=y Not tainted ] (XEN) CPU:0 (XEN) PC: 00238b78 xmem_pool_alloc+0x348/0x4b0 (XEN) LR: 00238960 (XEN) SP: 002bf4e0 (XEN) CPSR: 2249 MODE:64-bit EL2h (Hypervisor, handler) (XEN) Xen call trace: (XEN)[<00238b78>] xmem_pool_alloc+0x348/0x4b0 (PC) (XEN)[<00238960>] xmem_pool_alloc+0x130/0x4b0 (LR) (XEN)[<00239100>] _xmalloc+0xf4/0x290 (XEN)[<002392b0>] _xzalloc+0x14/0x38 (XEN)[<00245430>] construct_dom0+0xbc0/0x14cc (XEN)[<0028c4c4>] start_xen+0xbf4/0xcb0 (XEN)[<0020060c>] paging+0x84/0xbc (XEN) (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) CPU0: Unexpected Trap: Hypervisor (XEN) (XEN) (XEN) (XEN) Reboot in five seconds... -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi Julien, On 2015/8/12 0:19, Julien Grall wrote: > Hi Shannon, > > On 07/08/15 03:11, Shannon Zhao wrote: >> 2. Create minimal DT to pass required information to Dom0 >> -- >> The minimal DT mainly passes Dom0 bootargs, address and size of initrd >> (if available), address and size of uefi system table, address and size >> of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. >> >> An example of the minimal DT: >> / { >> #address-cells = <2>; >> #size-cells = <1>; >> chosen { >> bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c09 >> root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; >> linux,initrd-start = <0x>; >> linux,initrd-end = <0x>; >> linux,uefi-system-table = <0x>; >> linux,uefi-mmap-start = <0x>; >> linux,uefi-mmap-size = <0x>; >> linux,uefi-mmap-desc-size = <0x>; >> linux,uefi-mmap-desc-ver = <0x>; >> }; >> }; >> >> For details loook at >> https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt > > AFAICT, the device tree properties in this documentation are only used > in order to communicate between the UEFI stub and Linux. > > This means that those properties are not standardize and can change at > any time by Linux folks. They don't even live in Documentation/devicetree/ > > I would also expect to see the same needs for FreeBSD running as DOM0 > with ACPI. > I'm not very clear about how FreeBSD communicates with UEFI. And when booting with DT, how does FreeBSD communicate with UEFI? Not through these properties? > So it looks like to me that a generic name would be better for all those > properties. > If we change these name, it needs change some functions in Linux. Will it impact the use of Linux with UEFI not on Xen? Thanks, -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/11 23:52, Boris Ostrovsky wrote: > On 08/11/2015 11:35 AM, Ian Campbell wrote: >> On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote: >>> On 08/11/2015 10:21 AM, Ian Campbell wrote: On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >> This document is going to explain the design details of Xen booting >> with >> ACPI on ARM. Maybe parts of it may not be appropriate. Any comments >> are >> welcome. > Some small subsets of this seem like they might overlap with what > will be > required for PVH on x86 (a new x86 guest mode not dissimilar to the > sole > ARM guest mode). If so then it would be preferable IMHO if PVH x86 > could > use the same interfaces. > > I've trimmed the quotes to just those bits and CCd some of the PVH > people > (Boris and Roger[0]) in case they have any thoughts. > > Actually, having done the trimming there is only one such bit: > > [...] >> 4. Map MMIO regions >> --- >> Register a bus_notifier for platform and amba bus in Linux. Previously PCI was included in this scheme, which is why I thought of PVH, having missed that PCI wasn't mentioned here now. Is it handled some other way now or is that just an accidental omission? >>> PCI already has a bus notifier which is probably why it's not explicitly >>> called out here. >> Right, but I was more specifically thinking of the bit which followed >> regarding the use of the notifier to map the MMIO, which is the more >> important bit. > > This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell > MMIO is mapped there (for IOMMU, which IIUIC is the main issue here). Right, at this moment we only add platform and amba bus bus_notifier. The PCI device can reuse existing bus_notifier. -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/12 0:01, Julien Grall wrote: > On 11/08/15 16:19, Ian Campbell wrote: >>> IIRC we talked about it few months ago and you said that using balloon >>> page will split in 4K the 1G/2M mapping done in the stage-2 p2m. >> >> Did I? Odd because I'm also of the opinion that alloc_ballooned_pages >> should operate in chunks of 2M at the hypercall layer and keep any >> resulting spare 4K pages on a free list to use for future such allocations. > > That from what I recall from an IRL talk. > > Anyway, I've looked in my archive to see why we decided to keep the > grant table parameters (in Xen ACPI table at this point). We were not > sure that the domain as all the key in hand in order to find memory hole. > > I think it's quite important to not think only about Linux but all other > Operating Systems. If we ever require a parameters later, it would mean > that the OS won't be able to run as DOM0 on older Xen. > > Linux is using ballooned page, which means loosing ~128KB (default of > the grant table on ARM) of memory because we never give back the page to > Xen due the 1:1 mapping. Although I guess this is not a big deal as it's > quite small and Linux, as said by David, will support memory hotplug soon. > > FreeBSD is using memory hole in the address space so there is no issue here. > > So I guess we could skip this parameters as 128KB doesn't seem to be a > big deal. > >> IOW it should avoid such shattering where it can. > > That would work too. > So use xlated_setup_gnttab_pages for both DT and ACPI? -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi Shannon, On 07/08/15 03:11, Shannon Zhao wrote: > 2. Create minimal DT to pass required information to Dom0 > -- > The minimal DT mainly passes Dom0 bootargs, address and size of initrd > (if available), address and size of uefi system table, address and size > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > An example of the minimal DT: > / { > #address-cells = <2>; > #size-cells = <1>; > chosen { > bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c09 > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > linux,initrd-start = <0x>; > linux,initrd-end = <0x>; > linux,uefi-system-table = <0x>; > linux,uefi-mmap-start = <0x>; > linux,uefi-mmap-size = <0x>; > linux,uefi-mmap-desc-size = <0x>; > linux,uefi-mmap-desc-ver = <0x>; > }; > }; > > For details loook at > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt AFAICT, the device tree properties in this documentation are only used in order to communicate between the UEFI stub and Linux. This means that those properties are not standardize and can change at any time by Linux folks. They don't even live in Documentation/devicetree/ I would also expect to see the same needs for FreeBSD running as DOM0 with ACPI. So it looks like to me that a generic name would be better for all those properties. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 16:19, Ian Campbell wrote: >> IIRC we talked about it few months ago and you said that using balloon >> page will split in 4K the 1G/2M mapping done in the stage-2 p2m. > > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages > should operate in chunks of 2M at the hypercall layer and keep any > resulting spare 4K pages on a free list to use for future such allocations. That from what I recall from an IRL talk. Anyway, I've looked in my archive to see why we decided to keep the grant table parameters (in Xen ACPI table at this point). We were not sure that the domain as all the key in hand in order to find memory hole. I think it's quite important to not think only about Linux but all other Operating Systems. If we ever require a parameters later, it would mean that the OS won't be able to run as DOM0 on older Xen. Linux is using ballooned page, which means loosing ~128KB (default of the grant table on ARM) of memory because we never give back the page to Xen due the 1:1 mapping. Although I guess this is not a big deal as it's quite small and Linux, as said by David, will support memory hotplug soon. FreeBSD is using memory hole in the address space so there is no issue here. So I guess we could skip this parameters as 128KB doesn't seem to be a big deal. > IOW it should avoid such shattering where it can. That would work too. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 08/11/2015 11:35 AM, Ian Campbell wrote: On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote: On 08/11/2015 10:21 AM, Ian Campbell wrote: On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: This document is going to explain the design details of Xen booting with ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are welcome. Some small subsets of this seem like they might overlap with what will be required for PVH on x86 (a new x86 guest mode not dissimilar to the sole ARM guest mode). If so then it would be preferable IMHO if PVH x86 could use the same interfaces. I've trimmed the quotes to just those bits and CCd some of the PVH people (Boris and Roger[0]) in case they have any thoughts. Actually, having done the trimming there is only one such bit: [...] 4. Map MMIO regions --- Register a bus_notifier for platform and amba bus in Linux. Previously PCI was included in this scheme, which is why I thought of PVH, having missed that PCI wasn't mentioned here now. Is it handled some other way now or is that just an accidental omission? PCI already has a bus notifier which is probably why it's not explicitly called out here. Right, but I was more specifically thinking of the bit which followed regarding the use of the notifier to map the MMIO, which is the more important bit. This notifier calls PHYSDEVOP_pci_device_add and as far as I can tell MMIO is mapped there (for IOMMU, which IIUIC is the main issue here). -boris Add a new XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the device is newly added, then call hypercall XENMEM_add_to_physmap to map the mmio regions. Ian. [0] Roger is away for a week or so, but I'm expect feedback to be of the "we could use one extra field" type rather than "this needs to be done some totally different way for x86/PVH" (in which case we wouldn't want to share the interface anyway I suppose) so need to block on awaiting that feedback. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Tue, 2015-08-11 at 11:29 -0400, Boris Ostrovsky wrote: > On 08/11/2015 10:21 AM, Ian Campbell wrote: > > On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: > > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > > > > This document is going to explain the design details of Xen booting > > > > with > > > > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments > > > > are > > > > welcome. > > > Some small subsets of this seem like they might overlap with what > > > will be > > > required for PVH on x86 (a new x86 guest mode not dissimilar to the > > > sole > > > ARM guest mode). If so then it would be preferable IMHO if PVH x86 > > > could > > > use the same interfaces. > > > > > > I've trimmed the quotes to just those bits and CCd some of the PVH > > > people > > > (Boris and Roger[0]) in case they have any thoughts. > > > > > > Actually, having done the trimming there is only one such bit: > > > > > > [...] > > > > 4. Map MMIO regions > > > > --- > > > > Register a bus_notifier for platform and amba bus in Linux. > > Previously PCI was included in this scheme, which is why I thought of > > PVH, > > having missed that PCI wasn't mentioned here now. > > > > Is it handled some other way now or is that just an accidental > > omission? > > PCI already has a bus notifier which is probably why it's not explicitly > called out here. Right, but I was more specifically thinking of the bit which followed regarding the use of the notifier to map the MMIO, which is the more important bit. > > > > > Add a new > > > > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if > > > > the > > > > device is newly added, then call hypercall XENMEM_add_to_physmap to > > > > map > > > > the mmio regions. > > > Ian. > > > > > > [0] Roger is away for a week or so, but I'm expect feedback to be of > > > the > > > "we could use one extra field" type rather than "this needs to be > > > done > > > some > > > totally different way for x86/PVH" (in which case we wouldn't want to > > > share > > > the interface anyway I suppose) so need to block on awaiting that > > > feedback. > > > ___ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 08/11/2015 10:21 AM, Ian Campbell wrote: On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: This document is going to explain the design details of Xen booting with ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are welcome. Some small subsets of this seem like they might overlap with what will be required for PVH on x86 (a new x86 guest mode not dissimilar to the sole ARM guest mode). If so then it would be preferable IMHO if PVH x86 could use the same interfaces. I've trimmed the quotes to just those bits and CCd some of the PVH people (Boris and Roger[0]) in case they have any thoughts. Actually, having done the trimming there is only one such bit: [...] 4. Map MMIO regions --- Register a bus_notifier for platform and amba bus in Linux. Previously PCI was included in this scheme, which is why I thought of PVH, having missed that PCI wasn't mentioned here now. Is it handled some other way now or is that just an accidental omission? PCI already has a bus notifier which is probably why it's not explicitly called out here. -borsi Add a new XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the device is newly added, then call hypercall XENMEM_add_to_physmap to map the mmio regions. Ian. [0] Roger is away for a week or so, but I'm expect feedback to be of the "we could use one extra field" type rather than "this needs to be done some totally different way for x86/PVH" (in which case we wouldn't want to share the interface anyway I suppose) so need to block on awaiting that feedback. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 16:19, Ian Campbell wrote: > On Tue, 2015-08-11 at 16:11 +0100, Julien Grall wrote: >> On 11/08/15 15:59, Ian Campbell wrote: >>> On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote: On 11/08/15 15:12, Ian Campbell wrote: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >> > [...] >> 3. Dom0 gets grant table and event channel irq information >> --- >> As said above, we assign the hypervisor_id be "XenVMM" to tell >> Dom0 >> that >> it runs on Xen hypervisor. >> >> For grant table, add two new HVM_PARAMs: >> HVM_PARAM_GNTTAB_START_ADDRESS >> and HVM_PARAM_GNTTAB_SIZE. > > The reason we expose this range is essentially to allow OS authors > to > take > a short cut by telling them about an IPA range which is unused, so > it > is > available for remapping the grant table into. On x86 there is a BAR > on > the > Xen platform PCI device which serves a similar purpose. > > IIRC somebody (perhaps David V, CCd) had proposed at some point to > make > it > so that Linux was able to pick such an IPA itself by examining the > memory > map or by some other scheme. PVH in Linux uses ballooned pages which are vmap()'d into a virtually contiguous region. See xlated_setup_gnttab_pages(). >>> >>> So somewhat more concrete than a proposal then ;-) >>> >>> I don't see anything there which would be a problem on ARM, so we >>> should >>> probably go that route there too (at least for ACPI, if not globally >>> for >>> all ARM guests). >> >> IIRC we talked about it few months ago and you said that using balloon >> page will split in 4K the 1G/2M mapping done in the stage-2 p2m. > > Did I? Odd because I'm also of the opinion that alloc_ballooned_pages > should operate in chunks of 2M at the hypercall layer and keep any > resulting spare 4K pages on a free list to use for future such allocations. > > IOW it should avoid such shattering where it can. You can also (soon) enable memory hotplug and get to hotplug new (empty) memory sections to avoid having to release any frames back to Xen. David ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Tue, 2015-08-11 at 16:11 +0100, Julien Grall wrote: > On 11/08/15 15:59, Ian Campbell wrote: > > On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote: > > > On 11/08/15 15:12, Ian Campbell wrote: > > > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > > > > > > > > > [...] > > > > > 3. Dom0 gets grant table and event channel irq information > > > > > --- > > > > > As said above, we assign the hypervisor_id be "XenVMM" to tell > > > > > Dom0 > > > > > that > > > > > it runs on Xen hypervisor. > > > > > > > > > > For grant table, add two new HVM_PARAMs: > > > > > HVM_PARAM_GNTTAB_START_ADDRESS > > > > > and HVM_PARAM_GNTTAB_SIZE. > > > > > > > > The reason we expose this range is essentially to allow OS authors > > > > to > > > > take > > > > a short cut by telling them about an IPA range which is unused, so > > > > it > > > > is > > > > available for remapping the grant table into. On x86 there is a BAR > > > > on > > > > the > > > > Xen platform PCI device which serves a similar purpose. > > > > > > > > IIRC somebody (perhaps David V, CCd) had proposed at some point to > > > > make > > > > it > > > > so that Linux was able to pick such an IPA itself by examining the > > > > memory > > > > map or by some other scheme. > > > > > > PVH in Linux uses ballooned pages which are vmap()'d into a virtually > > > contiguous region. > > > > > > See xlated_setup_gnttab_pages(). > > > > So somewhat more concrete than a proposal then ;-) > > > > I don't see anything there which would be a problem on ARM, so we > > should > > probably go that route there too (at least for ACPI, if not globally > > for > > all ARM guests). > > IIRC we talked about it few months ago and you said that using balloon > page will split in 4K the 1G/2M mapping done in the stage-2 p2m. Did I? Odd because I'm also of the opinion that alloc_ballooned_pages should operate in chunks of 2M at the hypercall layer and keep any resulting spare 4K pages on a free list to use for future such allocations. IOW it should avoid such shattering where it can. > Note that for DOM0 we never give back the ballooned page to Xen. This is > because we have to respect the 1:1 mapping when DOM0 want to get back > the page. > > Regards, > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 15:59, Ian Campbell wrote: > On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote: >> On 11/08/15 15:12, Ian Campbell wrote: >>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >>> [...] 3. Dom0 gets grant table and event channel irq information --- As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that it runs on Xen hypervisor. For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS and HVM_PARAM_GNTTAB_SIZE. >>> >>> The reason we expose this range is essentially to allow OS authors to >>> take >>> a short cut by telling them about an IPA range which is unused, so it >>> is >>> available for remapping the grant table into. On x86 there is a BAR on >>> the >>> Xen platform PCI device which serves a similar purpose. >>> >>> IIRC somebody (perhaps David V, CCd) had proposed at some point to make >>> it >>> so that Linux was able to pick such an IPA itself by examining the >>> memory >>> map or by some other scheme. >> >> PVH in Linux uses ballooned pages which are vmap()'d into a virtually >> contiguous region. >> >> See xlated_setup_gnttab_pages(). > > So somewhat more concrete than a proposal then ;-) > > I don't see anything there which would be a problem on ARM, so we should > probably go that route there too (at least for ACPI, if not globally for > all ARM guests). IIRC we talked about it few months ago and you said that using balloon page will split in 4K the 1G/2M mapping done in the stage-2 p2m. Note that for DOM0 we never give back the ballooned page to Xen. This is because we have to respect the 1:1 mapping when DOM0 want to get back the page. Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 15:59, Ian Campbell wrote: > On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote: >> On 11/08/15 15:12, Ian Campbell wrote: >>> On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >>> [...] 3. Dom0 gets grant table and event channel irq information --- As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that it runs on Xen hypervisor. For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS and HVM_PARAM_GNTTAB_SIZE. >>> >>> The reason we expose this range is essentially to allow OS authors to >>> take >>> a short cut by telling them about an IPA range which is unused, so it >>> is >>> available for remapping the grant table into. On x86 there is a BAR on >>> the >>> Xen platform PCI device which serves a similar purpose. >>> >>> IIRC somebody (perhaps David V, CCd) had proposed at some point to make >>> it >>> so that Linux was able to pick such an IPA itself by examining the >>> memory >>> map or by some other scheme. >> >> PVH in Linux uses ballooned pages which are vmap()'d into a virtually >> contiguous region. >> >> See xlated_setup_gnttab_pages(). > > So somewhat more concrete than a proposal then ;-) > > I don't see anything there which would be a problem on ARM, so we should > probably go that route there too (at least for ACPI, if not globally for > all ARM guests). If someone does this please move xlated_setup_gnttab_pages() into drivers/xen/xlate_mmu.c, and not copy it into an arm specific file. David ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Tue, 2015-08-11 at 15:51 +0100, David Vrabel wrote: > On 11/08/15 15:12, Ian Campbell wrote: > > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > > > > > [...] > > > 3. Dom0 gets grant table and event channel irq information > > > --- > > > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 > > > that > > > it runs on Xen hypervisor. > > > > > > For grant table, add two new HVM_PARAMs: > > > HVM_PARAM_GNTTAB_START_ADDRESS > > > and HVM_PARAM_GNTTAB_SIZE. > > > > The reason we expose this range is essentially to allow OS authors to > > take > > a short cut by telling them about an IPA range which is unused, so it > > is > > available for remapping the grant table into. On x86 there is a BAR on > > the > > Xen platform PCI device which serves a similar purpose. > > > > IIRC somebody (perhaps David V, CCd) had proposed at some point to make > > it > > so that Linux was able to pick such an IPA itself by examining the > > memory > > map or by some other scheme. > > PVH in Linux uses ballooned pages which are vmap()'d into a virtually > contiguous region. > > See xlated_setup_gnttab_pages(). So somewhat more concrete than a proposal then ;-) I don't see anything there which would be a problem on ARM, so we should probably go that route there too (at least for ACPI, if not globally for all ARM guests). Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 15:12, Ian Campbell wrote: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: >> > [...] >> 3. Dom0 gets grant table and event channel irq information >> --- >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >> it runs on Xen hypervisor. >> >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >> and HVM_PARAM_GNTTAB_SIZE. > > The reason we expose this range is essentially to allow OS authors to take > a short cut by telling them about an IPA range which is unused, so it is > available for remapping the grant table into. On x86 there is a BAR on the > Xen platform PCI device which serves a similar purpose. > > IIRC somebody (perhaps David V, CCd) had proposed at some point to make it > so that Linux was able to pick such an IPA itself by examining the memory > map or by some other scheme. PVH in Linux uses ballooned pages which are vmap()'d into a virtually contiguous region. See xlated_setup_gnttab_pages(). David ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Tue, 2015-08-11 at 15:19 +0100, Ian Campbell wrote: > On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > > This document is going to explain the design details of Xen booting > > with > > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are > > welcome. > > Some small subsets of this seem like they might overlap with what will be > required for PVH on x86 (a new x86 guest mode not dissimilar to the sole > ARM guest mode). If so then it would be preferable IMHO if PVH x86 could > use the same interfaces. > > I've trimmed the quotes to just those bits and CCd some of the PVH people > (Boris and Roger[0]) in case they have any thoughts. > > Actually, having done the trimming there is only one such bit: > > [...] > > 4. Map MMIO regions > > --- > > Register a bus_notifier for platform and amba bus in Linux. Previously PCI was included in this scheme, which is why I thought of PVH, having missed that PCI wasn't mentioned here now. Is it handled some other way now or is that just an accidental omission? > > Add a new > > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the > > device is newly added, then call hypercall XENMEM_add_to_physmap to map > > the mmio regions. > > Ian. > > [0] Roger is away for a week or so, but I'm expect feedback to be of the > "we could use one extra field" type rather than "this needs to be done > some > totally different way for x86/PVH" (in which case we wouldn't want to > share > the interface anyway I suppose) so need to block on awaiting that > feedback. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > This document is going to explain the design details of Xen booting with > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are > welcome. Some small subsets of this seem like they might overlap with what will be required for PVH on x86 (a new x86 guest mode not dissimilar to the sole ARM guest mode). If so then it would be preferable IMHO if PVH x86 could use the same interfaces. I've trimmed the quotes to just those bits and CCd some of the PVH people (Boris and Roger[0]) in case they have any thoughts. Actually, having done the trimming there is only one such bit: [...] > 4. Map MMIO regions > --- > Register a bus_notifier for platform and amba bus in Linux. Add a new > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the > device is newly added, then call hypercall XENMEM_add_to_physmap to map > the mmio regions. Ian. [0] Roger is away for a week or so, but I'm expect feedback to be of the "we could use one extra field" type rather than "this needs to be done some totally different way for x86/PVH" (in which case we wouldn't want to share the interface anyway I suppose) so need to block on awaiting that feedback. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, 2015-08-07 at 10:11 +0800, Shannon Zhao wrote: > [...] > 3. Dom0 gets grant table and event channel irq information > --- > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that > it runs on Xen hypervisor. > > For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS > and HVM_PARAM_GNTTAB_SIZE. The reason we expose this range is essentially to allow OS authors to take a short cut by telling them about an IPA range which is unused, so it is available for remapping the grant table into. On x86 there is a BAR on the Xen platform PCI device which serves a similar purpose. IIRC somebody (perhaps David V, CCd) had proposed at some point to make it so that Linux was able to pick such an IPA itself by examining the memory map or by some other scheme. Any _if_ there was a viable proposal along those lines then we could use it and avoid the need for these HVM params perhaps? Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 2015/8/11 17:46, Julien Grall wrote: > On 11/08/15 03:09, Shannon Zhao wrote: >> Hi Julien, > > Hi Shannon, > >> On 2015/8/7 18:33, Julien Grall wrote: >>> Hi Shannon, >>> >>> Just some clarification questions. >>> >>> On 07/08/15 03:11, Shannon Zhao wrote: 3. Dom0 gets grant table and event channel irq information --- As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that it runs on Xen hypervisor. For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS and HVM_PARAM_GNTTAB_SIZE. For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new delivery type: val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 only) >>> >>> Can you describe the content of flag? >>> >> >> This needs definition as well. I think it could use the definition of >> xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt >> polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ. > > That would be fine for me. > When constructing Dom0 in Xen, save these values. Then Dom0 could get them through hypercall HVMOP_get_param. 4. Map MMIO regions --- Register a bus_notifier for platform and amba bus in Linux. Add a new XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the device is newly added, then call hypercall XENMEM_add_to_physmap to map the mmio regions. 5. Route device interrupts to Dom0 -- Route all the SPI interrupts to Dom0 before Dom0 booting. >>> >>> Not all the SPI will be routed to DOM0. Some are used by Xen and should >>> never be used by any guest. I have in mind the UART and SMMU interrupts. >>> >>> You will have to find away to skip them nicely. Note that not all the >>> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). >>> >> To uart, we can get the interrupt information from SPCR table and hide >> it from Dom0. > > Can you clarify your meaning of "hide from DOM0"? Did you mean avoid to > route the SPI to DOM0? > Yes. >> IIUC, currently Xen (as well as Linux) doesn't support use SMMU when >> booting with ACPI. When it supports, it could read the interrupts >> information from IORT table and Hide them from Dom0. > > Well for Xen we don't even have ACPI supported upstream ;). For Linux > there is some on-going work. Anyway, this is not important right now. > Yeah, that could be done after this patchset upstream. Thanks, -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 11/08/15 03:09, Shannon Zhao wrote: > Hi Julien, Hi Shannon, > On 2015/8/7 18:33, Julien Grall wrote: >> Hi Shannon, >> >> Just some clarification questions. >> >> On 07/08/15 03:11, Shannon Zhao wrote: >>> 3. Dom0 gets grant table and event channel irq information >>> --- >>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >>> it runs on Xen hypervisor. >>> >>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >>> and HVM_PARAM_GNTTAB_SIZE. >>> >>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new >>> delivery type: >>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 >>> only) >> >> Can you describe the content of flag? >> > > This needs definition as well. I think it could use the definition of > xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt > polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ. That would be fine for me. >>> When constructing Dom0 in Xen, save these values. Then Dom0 could get >>> them through hypercall HVMOP_get_param. >>> >>> 4. Map MMIO regions >>> --- >>> Register a bus_notifier for platform and amba bus in Linux. Add a new >>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >>> device is newly added, then call hypercall XENMEM_add_to_physmap to map >>> the mmio regions. >>> >>> 5. Route device interrupts to Dom0 >>> -- >>> Route all the SPI interrupts to Dom0 before Dom0 booting. >> >> Not all the SPI will be routed to DOM0. Some are used by Xen and should >> never be used by any guest. I have in mind the UART and SMMU interrupts. >> >> You will have to find away to skip them nicely. Note that not all the >> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). >> > To uart, we can get the interrupt information from SPCR table and hide > it from Dom0. Can you clarify your meaning of "hide from DOM0"? Did you mean avoid to route the SPI to DOM0? > IIUC, currently Xen (as well as Linux) doesn't support use SMMU when > booting with ACPI. When it supports, it could read the interrupts > information from IORT table and Hide them from Dom0. Well for Xen we don't even have ACPI supported upstream ;). For Linux there is some on-going work. Anyway, this is not important right now. -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi Julien, On 2015/8/7 18:33, Julien Grall wrote: > Hi Shannon, > > Just some clarification questions. > > On 07/08/15 03:11, Shannon Zhao wrote: >> 3. Dom0 gets grant table and event channel irq information >> --- >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >> it runs on Xen hypervisor. >> >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >> and HVM_PARAM_GNTTAB_SIZE. >> >> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new >> delivery type: >> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 >> only) > > Can you describe the content of flag? > This needs definition as well. I think it could use the definition of xenv table. Bit 0 stands interrupt mode and bit 1 stands interrupt polarity. And explain it in the comment of HVM_PARAM_CALLBACK_IRQ. >> When constructing Dom0 in Xen, save these values. Then Dom0 could get >> them through hypercall HVMOP_get_param. >> >> 4. Map MMIO regions >> --- >> Register a bus_notifier for platform and amba bus in Linux. Add a new >> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >> device is newly added, then call hypercall XENMEM_add_to_physmap to map >> the mmio regions. >> >> 5. Route device interrupts to Dom0 >> -- >> Route all the SPI interrupts to Dom0 before Dom0 booting. > > Not all the SPI will be routed to DOM0. Some are used by Xen and should > never be used by any guest. I have in mind the UART and SMMU interrupts. > > You will have to find away to skip them nicely. Note that not all the > IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). > To uart, we can get the interrupt information from SPCR table and hide it from Dom0. IIUC, currently Xen (as well as Linux) doesn't support use SMMU when booting with ACPI. When it supports, it could read the interrupts information from IORT table and Hide them from Dom0. -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On 07/08/15 11:37, Christoffer Dall wrote: > On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall wrote: >> Hi Shannon, >> >> Just some clarification questions. >> >> On 07/08/15 03:11, Shannon Zhao wrote: >>> 3. Dom0 gets grant table and event channel irq information >>> --- >>> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >>> it runs on Xen hypervisor. >>> >>> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >>> and HVM_PARAM_GNTTAB_SIZE. >>> >>> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new >>> delivery type: >>> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 >>> only) >> >> Can you describe the content of flag? >> >>> When constructing Dom0 in Xen, save these values. Then Dom0 could get >>> them through hypercall HVMOP_get_param. >>> >>> 4. Map MMIO regions >>> --- >>> Register a bus_notifier for platform and amba bus in Linux. Add a new >>> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >>> device is newly added, then call hypercall XENMEM_add_to_physmap to map >>> the mmio regions. >>> >>> 5. Route device interrupts to Dom0 >>> -- >>> Route all the SPI interrupts to Dom0 before Dom0 booting. >> >> Not all the SPI will be routed to DOM0. Some are used by Xen and should >> never be used by any guest. I have in mind the UART and SMMU interrupts. >> >> You will have to find away to skip them nicely. Note that not all the >> IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). >> > Just to clarify; Xen should map all SPIs which are not reserved for > Xen to Dom0, right? Right. -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, 7 Aug 2015, Christoffer Dall wrote: > On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall wrote: > > Hi Shannon, > > > > Just some clarification questions. > > > > On 07/08/15 03:11, Shannon Zhao wrote: > >> 3. Dom0 gets grant table and event channel irq information > >> --- > >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that > >> it runs on Xen hypervisor. > >> > >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS > >> and HVM_PARAM_GNTTAB_SIZE. > >> > >> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new > >> delivery type: > >> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 > >> only) > > > > Can you describe the content of flag? > > > >> When constructing Dom0 in Xen, save these values. Then Dom0 could get > >> them through hypercall HVMOP_get_param. > >> > >> 4. Map MMIO regions > >> --- > >> Register a bus_notifier for platform and amba bus in Linux. Add a new > >> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the > >> device is newly added, then call hypercall XENMEM_add_to_physmap to map > >> the mmio regions. > >> > >> 5. Route device interrupts to Dom0 > >> -- > >> Route all the SPI interrupts to Dom0 before Dom0 booting. > > > > Not all the SPI will be routed to DOM0. Some are used by Xen and should > > never be used by any guest. I have in mind the UART and SMMU interrupts. > > > > You will have to find away to skip them nicely. Note that not all the > > IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). > > > Just to clarify; Xen should map all SPIs which are not reserved for > Xen to Dom0, right? Yes, all SPIs that Xen does not use for itself. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, Aug 7, 2015 at 12:33 PM, Julien Grall wrote: > Hi Shannon, > > Just some clarification questions. > > On 07/08/15 03:11, Shannon Zhao wrote: >> 3. Dom0 gets grant table and event channel irq information >> --- >> As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that >> it runs on Xen hypervisor. >> >> For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS >> and HVM_PARAM_GNTTAB_SIZE. >> >> For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new >> delivery type: >> val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 >> only) > > Can you describe the content of flag? > >> When constructing Dom0 in Xen, save these values. Then Dom0 could get >> them through hypercall HVMOP_get_param. >> >> 4. Map MMIO regions >> --- >> Register a bus_notifier for platform and amba bus in Linux. Add a new >> XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the >> device is newly added, then call hypercall XENMEM_add_to_physmap to map >> the mmio regions. >> >> 5. Route device interrupts to Dom0 >> -- >> Route all the SPI interrupts to Dom0 before Dom0 booting. > > Not all the SPI will be routed to DOM0. Some are used by Xen and should > never be used by any guest. I have in mind the UART and SMMU interrupts. > > You will have to find away to skip them nicely. Note that not all the > IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). > Just to clarify; Xen should map all SPIs which are not reserved for Xen to Dom0, right? -Christoffer ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
Hi Shannon, Just some clarification questions. On 07/08/15 03:11, Shannon Zhao wrote: > 3. Dom0 gets grant table and event channel irq information > --- > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that > it runs on Xen hypervisor. > > For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS > and HVM_PARAM_GNTTAB_SIZE. > > For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new > delivery type: > val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 > only) Can you describe the content of flag? > When constructing Dom0 in Xen, save these values. Then Dom0 could get > them through hypercall HVMOP_get_param. > > 4. Map MMIO regions > --- > Register a bus_notifier for platform and amba bus in Linux. Add a new > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the > device is newly added, then call hypercall XENMEM_add_to_physmap to map > the mmio regions. > > 5. Route device interrupts to Dom0 > -- > Route all the SPI interrupts to Dom0 before Dom0 booting. Not all the SPI will be routed to DOM0. Some are used by Xen and should never be used by any guest. I have in mind the UART and SMMU interrupts. You will have to find away to skip them nicely. Note that not all the IRQs used by Xen are properly registered when we build DOM0 (see the SMMU). Regards, -- Julien Grall ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
On Fri, 7 Aug 2015, Shannon Zhao wrote: > This document is going to explain the design details of Xen booting with > ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are > welcome. > > To Xen itself booting with ACPI, this is similar to Linux kernel except > that Xen doesn't parse DSDT table. So I'll skip this part and focus on > how Xen prepares ACPI tables for Dom0 and how Xen passes them to Dom0. > > 1. Copy and change some EFI and ACPI tables > --- > a) Copy EFI_SYSTEM_TABLE and change the value of FirmwareVendor, > VendorGuid, VendorTable, ConfigurationTable. These changes are not very > special and it just assign values to these members. > > b) Create EFI_MEMORY_DESCRIPTOR table. This will add memory start and > size information of Dom0. And Dom0 will get the memory information > through this EFI table. > > c) Copy FADT table. Change the value of arm_boot_flags to enable PSCI > and HVC. Let the hypervisor_id be "XenVMM" in order to tell Dom0 that it > runs on Xen hypervisor, then Dom0 could through HVM_PARAM to get some > informations for booting necessity, such as grant table start address > and size. Change header revison, length and checksum as well. > > d) Copy GTDT table. Set non_secure_el2_interrupt and > non_secure_el2_flags to 0 to mask EL2 timer for Dom0. > > e) Copy MADT table. According to the value of dom0_max_vcpus to change > the number GICC entries. > > f) Create STAO table. This table is a new added one that's used to > define a list of ACPI namespace names that are to be ignored by the OSPM > in Dom0. Currently we use it to tell OSPM should ignore UART defined in > SPCR table. > > g) Copy XSDT table. Add a new table entry for STAO and change other > table's entries. > > h) Change the value of xsdt_physical_address in RSDP table. > > i) The rest of tables are not copied or changed. They are reused > including DSDT, SPCR, etc. > > All these tables will be copied to Dom0 memory except that the reused > tables(DSDT, SPCR, etc) will be mapped to Dom0. > > 2. Create minimal DT to pass required information to Dom0 > -- > The minimal DT mainly passes Dom0 bootargs, address and size of initrd > (if available), address and size of uefi system table, address and size > of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. > > An example of the minimal DT: > / { > #address-cells = <2>; > #size-cells = <1>; > chosen { > bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c09 > root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; > linux,initrd-start = <0x>; > linux,initrd-end = <0x>; > linux,uefi-system-table = <0x>; > linux,uefi-mmap-start = <0x>; > linux,uefi-mmap-size = <0x>; > linux,uefi-mmap-desc-size = <0x>; > linux,uefi-mmap-desc-ver = <0x>; > }; > }; > > For details loook at > https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt > > 3. Dom0 gets grant table and event channel irq information > --- > As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that > it runs on Xen hypervisor. > > For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS > and HVM_PARAM_GNTTAB_SIZE. > > For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new > delivery type: > val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 > only) > > When constructing Dom0 in Xen, save these values. Then Dom0 could get > them through hypercall HVMOP_get_param. > > 4. Map MMIO regions > --- > Register a bus_notifier for platform and amba bus in Linux. Add a new > XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the > device is newly added, then call hypercall XENMEM_add_to_physmap to map > the mmio regions. > > 5. Route device interrupts to Dom0 > -- > Route all the SPI interrupts to Dom0 before Dom0 booting. > > > Look forward to your comments. If you think it has no problem, giving > your ack would be a big help. Then the patchset could move on. :) Indeed. Acked-by: Stefano Stabellini It would be nice to have an Ack from Jan too. He should be back in a couple of days. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] Design doc of adding ACPI support for arm64 on Xen - version 2
This document is going to explain the design details of Xen booting with ACPI on ARM. Maybe parts of it may not be appropriate. Any comments are welcome. To Xen itself booting with ACPI, this is similar to Linux kernel except that Xen doesn't parse DSDT table. So I'll skip this part and focus on how Xen prepares ACPI tables for Dom0 and how Xen passes them to Dom0. 1. Copy and change some EFI and ACPI tables --- a) Copy EFI_SYSTEM_TABLE and change the value of FirmwareVendor, VendorGuid, VendorTable, ConfigurationTable. These changes are not very special and it just assign values to these members. b) Create EFI_MEMORY_DESCRIPTOR table. This will add memory start and size information of Dom0. And Dom0 will get the memory information through this EFI table. c) Copy FADT table. Change the value of arm_boot_flags to enable PSCI and HVC. Let the hypervisor_id be "XenVMM" in order to tell Dom0 that it runs on Xen hypervisor, then Dom0 could through HVM_PARAM to get some informations for booting necessity, such as grant table start address and size. Change header revison, length and checksum as well. d) Copy GTDT table. Set non_secure_el2_interrupt and non_secure_el2_flags to 0 to mask EL2 timer for Dom0. e) Copy MADT table. According to the value of dom0_max_vcpus to change the number GICC entries. f) Create STAO table. This table is a new added one that's used to define a list of ACPI namespace names that are to be ignored by the OSPM in Dom0. Currently we use it to tell OSPM should ignore UART defined in SPCR table. g) Copy XSDT table. Add a new table entry for STAO and change other table's entries. h) Change the value of xsdt_physical_address in RSDP table. i) The rest of tables are not copied or changed. They are reused including DSDT, SPCR, etc. All these tables will be copied to Dom0 memory except that the reused tables(DSDT, SPCR, etc) will be mapped to Dom0. 2. Create minimal DT to pass required information to Dom0 -- The minimal DT mainly passes Dom0 bootargs, address and size of initrd (if available), address and size of uefi system table, address and size of uefi memory table, uefi-mmap-desc-size and uefi-mmap-desc-ver. An example of the minimal DT: / { #address-cells = <2>; #size-cells = <1>; chosen { bootargs = "kernel=Image console=hvc0 earlycon=pl011,0x1c09 root=/dev/vda2 rw rootfstype=ext4 init=/bin/sh acpi=force"; linux,initrd-start = <0x>; linux,initrd-end = <0x>; linux,uefi-system-table = <0x>; linux,uefi-mmap-start = <0x>; linux,uefi-mmap-size = <0x>; linux,uefi-mmap-desc-size = <0x>; linux,uefi-mmap-desc-ver = <0x>; }; }; For details loook at https://github.com/torvalds/linux/blob/master/Documentation/arm/uefi.txt 3. Dom0 gets grant table and event channel irq information --- As said above, we assign the hypervisor_id be "XenVMM" to tell Dom0 that it runs on Xen hypervisor. For grant table, add two new HVM_PARAMs: HVM_PARAM_GNTTAB_START_ADDRESS and HVM_PARAM_GNTTAB_SIZE. For event channel irq, reuse HVM_PARAM_CALLBACK_IRQ and add a new delivery type: val[63:56] == 3: val[15:8] is flag: val[7:0] is a PPI (ARM and ARM64 only) When constructing Dom0 in Xen, save these values. Then Dom0 could get them through hypercall HVMOP_get_param. 4. Map MMIO regions --- Register a bus_notifier for platform and amba bus in Linux. Add a new XENMAPSPACE "XENMAPSPACE_dev_mmio". Within the register, check if the device is newly added, then call hypercall XENMEM_add_to_physmap to map the mmio regions. 5. Route device interrupts to Dom0 -- Route all the SPI interrupts to Dom0 before Dom0 booting. Look forward to your comments. If you think it has no problem, giving your ack would be a big help. Then the patchset could move on. :) Thanks, -- Shannon ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel