Re: Xen data from meta-virtualization layer
Hi Julien, Am 2022-02-05 00:29, schrieb Julien Grall: [..] But not a very user friendly one, though. I guess the first UART is disabled/removed by Xen? I haven't looked at how it is handled. Can't we search for other uarts with the same interrupt and disable these, too? Maybe conditionally by the SoC compatible? The problem sounds quite similar to the one we had on sunxi. Although the UART was on the same page rather than sharing interrupts. Xen has per-platform hook to prevent a device been assigned to dom0. For an example, you could look at: https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c Nice. At least, this is working now ;) I'll send a patch in the next few days. I guess it's safe to assume that we can always remove both UARTs on the LS1028A (probably on most layerscapes). The most common use case is to use the first UART for Xen. You could run Xen without console (?), then you'd miss the possibility to map the DUART. Or there could be a new driver for the LPUART on the LS1028A. In this case, the DUART wouldn't be used by Xen either. But I think we should start simple and just remove the DUART altogether via that hook. -michael
Re: Xen data from meta-virtualization layer
Hi Stefano, Am 2022-02-04 22:11, schrieb Stefano Stabellini: On Fri, 4 Feb 2022, Michael Walle wrote: > In regards to the reserved-memory regions, maybe we are not seeing them > because Leo posted the host device tree, not the one passed at runtime > from u-boot to Linux? > > If so, Leo, could you please boot Linux on native (no Xen) and get the > device tree from there at runtime using dtc -I fs -O dts > /proc/device-tree ? > > > However, the name of the reserved-memory region created by u-boot seems > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the > Linux kernel tree either. > > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen > throws errors in regards to GIC/ITS initialization. On other hardware > Xen can use and virtualize GICv3 and ITS just fine. Could you please > explain what is different about sAL28 and how Xen/Linux is expected to > use the lpi_rd_table reserved-memory region? I actually stumbled across this thread after trying out Xen myself. I'm using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel and vanilla Xen. So far I've discovered, that xen complains that it cannot route IRQ64 to dom0. That is because on the LS1028A there is a dual UART (two 16550 with one shared interrupt) and xen takes the first UART and then tries to map the interrupt of the second UART to linux. For now, I don't know how this is solved correctly. As a quick hack, I removed the second uart node from the device tree. This is an interesting problem. Removing the second UART is a good workaround for now as there is no obvious solution I think. But not a very user friendly one, though. I guess the first UART is disabled/removed by Xen? I haven't looked at how it is handled. Can't we search for other uarts with the same interrupt and disable these, too? Maybe conditionally by the SoC compatible? But what is more severe is that the iommu isn't set up correctly. I'm getting the following faults: (XEN) smmu: /soc/iommu@500: Unexpected global fault, this could be serious (XEN) smmu: /soc/iommu@500: GFSR 0x8002, GFSYNR0 0x, GFSYNR1 0x042a, GFSYNR2 0x If I decode it correctly, the streamid should be 0x2a which would be one of the PCI devices on the internal root complex. Probably the network card. Yes there is DMA transaction with an "unknown" StreamID. I think the StreamID is 0x42a. It means that there is a DMA master on the board with StreamID 0x42a that is either: - not described in device tree - described in device tree with a different StreamID - the right StreamID is described device tree, but it is not picked up by Xen See below. This is the first developer experience with Xen, so please bear with me :) It seems that Xen doesn't add the master to the IOMMU. To me it seems that only devices with a 'iommus' dt property are added. But in case of PCI devices the parent only has a iommu-map property. And it makes me wonder why Leo has an almost working setup. Maybe I'm missing some patches though. Xen 4.16 is able to parse StreamID in the "iommus" property and also "mmu-masters" property. But It is not able to parse the "iommu-map" property yet. So if 0x42a is described in device tree using "iommu-map" then the error makes sense. A simple solution is to replace iommu-map with iommus in device tree. I'm not sure this is so easy, because they are dynamically assigned by the bootloader. Sure for now I could do that I guess, but iommu=0 works as well ;) I now got Xen and Linux booting and see the same problems with the GIC ITS, that is that the enetc interrupts aren't delivered to the dom0 linux. I've also applied the patch in this thread and I'm seeing the same as Leo. Full boot log is here [1]. I noticed the following. [0.168544] pci :00:00.0: Failed to add - passthrough or MSI/MSI-X might fail! Not sure if it should work nonetheless. It is possible that someone in CC to this email might already have a patch to introduce parsing of iommu-map in Xen. I guess they've used the old mmu-masters property. Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't use TF-A and runs without it. Nonetheless, I've booted the board with the bl31 from NXP and it doesn't help either. There is still a difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot. I just found GIC setup code in the bl31. I'll also have a look there. -michael [1] https://pastebin.com/raw/XMjE3BvG
Re: Xen data from meta-virtualization layer
Hi all,
> + Zhiqiang Hou
>
> On Tue, 24 Nov 2020, Leo Krueger wrote:
> > > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > > Hi,
> > >
> > > I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > > before...) but then had to add the following node to my device tree
> > >
> > > gic_lpi_base: syscon@0x8000 {
> > > compatible = "gic-lpi-base";
> > > >>
> > > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > > >> could you clarify which flavor/version of Linux you are using?
> > > >
> > > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> > >
> > > Do you have a link to the Linux tree? Is there any additional patches on
> > > top of
> > > vanilla?
> >
> > Linux tree is found here:
> > https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> > (up to the latest commit in that branch)
FWIW, I'm the developer of the support for this board both in the
mentioned branch as well as upstream.
If you don't need graphics you shouldn't really be using the branch
above, but the latest vanilla kernel release.
> [...]
>
> > > Looking at the DT changes in [0], it looks like the node is not a child
> > > of gic@.
> > > So I think Xen will map the region to Dom0.
> > >
> > > There are two things that I can notice:
> > >1) This region is RAM, but I can't find any reserve node. Is there any
> > > specific
> > > code in Linux to reserve it?
> > >2) The implementation in U-boot seems to suggest that the firmware will
> > > configure the LPIs and then enable it. If that's the case, then Xen needs
> > > to
> > > re-use the table in the DT rather than allocating a new one.
>
> That Linux tree has no mentions of gic-lpi-base. That means that
> gic-lpi-base is only used in u-boot, not in Linux. In particular the
> most relevant commit is af288cb291da3abef6be0875527729296f7de7a0.
That node was horrible hack from NXP for u-boot and was removed in
a84cea06bb8fff69810a890ac0e4b47ea5726512.
> In regards to the reserved-memory regions, maybe we are not seeing them
> because Leo posted the host device tree, not the one passed at runtime
> from u-boot to Linux?
>
> If so, Leo, could you please boot Linux on native (no Xen) and get the
> device tree from there at runtime using dtc -I fs -O dts
> /proc/device-tree ?
>
>
> However, the name of the reserved-memory region created by u-boot seems
> to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> Linux kernel tree either.
>
> Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> throws errors in regards to GIC/ITS initialization. On other hardware
> Xen can use and virtualize GICv3 and ITS just fine. Could you please
> explain what is different about sAL28 and how Xen/Linux is expected to
> use the lpi_rd_table reserved-memory region?
I actually stumbled across this thread after trying out Xen myself. I'm
using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel
and vanilla Xen.
So far I've discovered, that xen complains that it cannot route IRQ64 to
dom0. That is because on the LS1028A there is a dual UART (two 16550
with one shared interrupt) and xen takes the first UART and then tries
to map the interrupt of the second UART to linux. For now, I don't know
how this is solved correctly. As a quick hack, I removed the second uart
node from the device tree.
But what is more severe is that the iommu isn't set up correctly. I'm
getting the following faults:
(XEN) smmu: /soc/iommu@500: Unexpected global fault, this could be serious
(XEN) smmu: /soc/iommu@500: GFSR 0x8002, GFSYNR0 0x,
GFSYNR1 0x042a, GFSYNR2 0x
If I decode it correctly, the streamid should be 0x2a which would be one
of the PCI devices on the internal root complex. Probably the network
card.
This is the first developer experience with Xen, so please bear with me
:) It seems that Xen doesn't add the master to the IOMMU. To me it seems
that only devices with a 'iommus' dt property are added. But in case of
PCI devices the parent only has a iommu-map property.
And it makes me wonder why Leo has an almost working setup. Maybe I'm
missing some patches though.
-michael
