Re: Xen data from meta-virtualization layer
On 04/02/2022 23:29, Julien Grall wrote: This message is harmless. This is printed because Xen on Arm doesn't hypercall the hypercall to add a PCI device. On Arm, I meant "doesn't need the hypercall...". we don't need it yet (it might be necessary for PCI passthrough) and MSI/MSI-X are handled/registered the same way as on bare-metal (for your case through the ITS) Not sure if it should work nonetheless. I looked through the log and couldn't spot anything obvious. However, skimming through Linux, I noticed there are some changes in the ITS for freescale (now NXP) such as: drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c Is it something that might be used on the board you are using? If the answer is yes, then my suggestion would be to look how this is meant to interact with the ITS. It might be possible that we are missing some pieces in Xen to properly support it. Another tree you might want to look at is https://source.codeaurora.org/external/imx/imx-xen. It contains a bunch of patches for NXP SoC that was never upstreamed. It is possible that someone in CC to this email might already have a patch to introduce parsing of iommu-map in Xen. Pass. I don't have any and couldn't find any submission on the ML. I guess they've used the old mmu-masters property. Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't use TF-A and runs without it. Nonetheless, I've booted the board with the bl31 from NXP and it doesn't help either. There is still a difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot. I just found GIC setup code in the bl31. I'll also have a look there. -michael [1] https://pastebin.com/raw/XMjE3BvG Cheers, -- Julien Grall
Re: Xen data from meta-virtualization layer
Hi Julien, Am 2022-02-05 00:29, schrieb Julien Grall: [..] But not a very user friendly one, though. I guess the first UART is disabled/removed by Xen? I haven't looked at how it is handled. Can't we search for other uarts with the same interrupt and disable these, too? Maybe conditionally by the SoC compatible? The problem sounds quite similar to the one we had on sunxi. Although the UART was on the same page rather than sharing interrupts. Xen has per-platform hook to prevent a device been assigned to dom0. For an example, you could look at: https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c Nice. At least, this is working now ;) I'll send a patch in the next few days. I guess it's safe to assume that we can always remove both UARTs on the LS1028A (probably on most layerscapes). The most common use case is to use the first UART for Xen. You could run Xen without console (?), then you'd miss the possibility to map the DUART. Or there could be a new driver for the LPUART on the LS1028A. In this case, the DUART wouldn't be used by Xen either. But I think we should start simple and just remove the DUART altogether via that hook. -michael
Re: Xen data from meta-virtualization layer
Hi Michael, On Fri, 4 Feb 2022 at 22:42, Michael Walle wrote: > Am 2022-02-04 22:11, schrieb Stefano Stabellini: > > On Fri, 4 Feb 2022, Michael Walle wrote: > >> > In regards to the reserved-memory regions, maybe we are not seeing them > >> > because Leo posted the host device tree, not the one passed at runtime > >> > from u-boot to Linux? > >> > > >> > If so, Leo, could you please boot Linux on native (no Xen) and get the > >> > device tree from there at runtime using dtc -I fs -O dts > >> > /proc/device-tree ? > >> > > >> > > >> > However, the name of the reserved-memory region created by u-boot seems > >> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the > >> > Linux kernel tree either. > >> > > >> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen > >> > throws errors in regards to GIC/ITS initialization. On other hardware > >> > Xen can use and virtualize GICv3 and ITS just fine. Could you please > >> > explain what is different about sAL28 and how Xen/Linux is expected to > >> > use the lpi_rd_table reserved-memory region? > >> > >> I actually stumbled across this thread after trying out Xen myself. > >> I'm > >> using lastest vanilla u-boot (with pending PSCI patches), vanilla > >> kernel > >> and vanilla Xen. > >> > >> So far I've discovered, that xen complains that it cannot route IRQ64 > >> to > >> dom0. That is because on the LS1028A there is a dual UART (two 16550 > >> with one shared interrupt) and xen takes the first UART and then tries > >> to map the interrupt of the second UART to linux. For now, I don't > >> know > >> how this is solved correctly. As a quick hack, I removed the second > >> uart > >> node from the device tree. > > > > This is an interesting problem. Removing the second UART is a good > > workaround for now as there is no obvious solution I think. > > But not a very user friendly one, though. I guess the first UART > is disabled/removed by Xen? I haven't looked at how it is handled. > > Can't we search for other uarts with the same interrupt and disable > these, too? Maybe conditionally by the SoC compatible? The problem sounds quite similar to the one we had on sunxi. Although the UART was on the same page rather than sharing interrupts. Xen has per-platform hook to prevent a device been assigned to dom0. For an example, you could look at: https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c > >> This is the first developer experience with Xen, so please bear with > >> me > >> :) It seems that Xen doesn't add the master to the IOMMU. To me it > >> seems > >> that only devices with a 'iommus' dt property are added. But in case > >> of > >> PCI devices the parent only has a iommu-map property. > >> > >> And it makes me wonder why Leo has an almost working setup. Maybe I'm > >> missing some patches though. > > > > Xen 4.16 is able to parse StreamID in the "iommus" property and also > > "mmu-masters" property. But It is not able to parse the "iommu-map" > > property yet. So if 0x42a is described in device tree using "iommu-map" > > then the error makes sense. > > > > A simple solution is to replace iommu-map with iommus in device tree. > > I'm not sure this is so easy, because they are dynamically assigned > by the bootloader. Sure for now I could do that I guess, but iommu=0 > works as well ;) > > I now got Xen and Linux booting and see the same problems with the > GIC ITS, that is that the enetc interrupts aren't delivered to the > dom0 linux. I've also applied the patch in this thread and I'm > seeing the same as Leo. Full boot log is here [1]. > > I noticed the following. > [0.168544] pci :00:00.0: Failed to add - passthrough or > MSI/MSI-X might fail! This message is harmless. This is printed because Xen on Arm doesn't hypercall the hypercall to add a PCI device. On Arm, we don't need it yet (it might be necessary for PCI passthrough) and MSI/MSI-X are handled/registered the same way as on bare-metal (for your case through the ITS) > > Not sure if it should work nonetheless. I looked through the log and couldn't spot anything obvious. However, skimming through Linux, I noticed there are some changes in the ITS for freescale (now NXP) such as: drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c Is it something that might be used on the board you are using? If the answer is yes, then my suggestion would be to look how this is meant to interact with the ITS. It might be possible that we are missing some pieces in Xen to properly support it. > > > It is possible that someone in CC to this email might already have a > > patch to introduce parsing of iommu-map in Xen. Pass. I don't have any and couldn't find any submission on the ML. > > I guess they've used the old mmu-masters property. > > Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't > use TF-A and runs without it. Nonetheless, I've booted the board with > the bl31 from NXP and it doesn't help either. There is sti
Re: Xen data from meta-virtualization layer
Hi Stefano, Am 2022-02-04 22:11, schrieb Stefano Stabellini: On Fri, 4 Feb 2022, Michael Walle wrote: > In regards to the reserved-memory regions, maybe we are not seeing them > because Leo posted the host device tree, not the one passed at runtime > from u-boot to Linux? > > If so, Leo, could you please boot Linux on native (no Xen) and get the > device tree from there at runtime using dtc -I fs -O dts > /proc/device-tree ? > > > However, the name of the reserved-memory region created by u-boot seems > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the > Linux kernel tree either. > > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen > throws errors in regards to GIC/ITS initialization. On other hardware > Xen can use and virtualize GICv3 and ITS just fine. Could you please > explain what is different about sAL28 and how Xen/Linux is expected to > use the lpi_rd_table reserved-memory region? I actually stumbled across this thread after trying out Xen myself. I'm using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel and vanilla Xen. So far I've discovered, that xen complains that it cannot route IRQ64 to dom0. That is because on the LS1028A there is a dual UART (two 16550 with one shared interrupt) and xen takes the first UART and then tries to map the interrupt of the second UART to linux. For now, I don't know how this is solved correctly. As a quick hack, I removed the second uart node from the device tree. This is an interesting problem. Removing the second UART is a good workaround for now as there is no obvious solution I think. But not a very user friendly one, though. I guess the first UART is disabled/removed by Xen? I haven't looked at how it is handled. Can't we search for other uarts with the same interrupt and disable these, too? Maybe conditionally by the SoC compatible? But what is more severe is that the iommu isn't set up correctly. I'm getting the following faults: (XEN) smmu: /soc/iommu@500: Unexpected global fault, this could be serious (XEN) smmu: /soc/iommu@500: GFSR 0x8002, GFSYNR0 0x, GFSYNR1 0x042a, GFSYNR2 0x If I decode it correctly, the streamid should be 0x2a which would be one of the PCI devices on the internal root complex. Probably the network card. Yes there is DMA transaction with an "unknown" StreamID. I think the StreamID is 0x42a. It means that there is a DMA master on the board with StreamID 0x42a that is either: - not described in device tree - described in device tree with a different StreamID - the right StreamID is described device tree, but it is not picked up by Xen See below. This is the first developer experience with Xen, so please bear with me :) It seems that Xen doesn't add the master to the IOMMU. To me it seems that only devices with a 'iommus' dt property are added. But in case of PCI devices the parent only has a iommu-map property. And it makes me wonder why Leo has an almost working setup. Maybe I'm missing some patches though. Xen 4.16 is able to parse StreamID in the "iommus" property and also "mmu-masters" property. But It is not able to parse the "iommu-map" property yet. So if 0x42a is described in device tree using "iommu-map" then the error makes sense. A simple solution is to replace iommu-map with iommus in device tree. I'm not sure this is so easy, because they are dynamically assigned by the bootloader. Sure for now I could do that I guess, but iommu=0 works as well ;) I now got Xen and Linux booting and see the same problems with the GIC ITS, that is that the enetc interrupts aren't delivered to the dom0 linux. I've also applied the patch in this thread and I'm seeing the same as Leo. Full boot log is here [1]. I noticed the following. [0.168544] pci :00:00.0: Failed to add - passthrough or MSI/MSI-X might fail! Not sure if it should work nonetheless. It is possible that someone in CC to this email might already have a patch to introduce parsing of iommu-map in Xen. I guess they've used the old mmu-masters property. Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't use TF-A and runs without it. Nonetheless, I've booted the board with the bl31 from NXP and it doesn't help either. There is still a difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot. I just found GIC setup code in the bl31. I'll also have a look there. -michael [1] https://pastebin.com/raw/XMjE3BvG
Re: Xen data from meta-virtualization layer
On Fri, 4 Feb 2022, Michael Walle wrote: > > In regards to the reserved-memory regions, maybe we are not seeing them > > because Leo posted the host device tree, not the one passed at runtime > > from u-boot to Linux? > > > > If so, Leo, could you please boot Linux on native (no Xen) and get the > > device tree from there at runtime using dtc -I fs -O dts > > /proc/device-tree ? > > > > > > However, the name of the reserved-memory region created by u-boot seems > > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the > > Linux kernel tree either. > > > > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen > > throws errors in regards to GIC/ITS initialization. On other hardware > > Xen can use and virtualize GICv3 and ITS just fine. Could you please > > explain what is different about sAL28 and how Xen/Linux is expected to > > use the lpi_rd_table reserved-memory region? > > I actually stumbled across this thread after trying out Xen myself. I'm > using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel > and vanilla Xen. > > So far I've discovered, that xen complains that it cannot route IRQ64 to > dom0. That is because on the LS1028A there is a dual UART (two 16550 > with one shared interrupt) and xen takes the first UART and then tries > to map the interrupt of the second UART to linux. For now, I don't know > how this is solved correctly. As a quick hack, I removed the second uart > node from the device tree. This is an interesting problem. Removing the second UART is a good workaround for now as there is no obvious solution I think. > But what is more severe is that the iommu isn't set up correctly. I'm > getting the following faults: > > (XEN) smmu: /soc/iommu@500: Unexpected global fault, this could be serious > (XEN) smmu: /soc/iommu@500: GFSR 0x8002, GFSYNR0 0x, > GFSYNR1 0x042a, GFSYNR2 0x > > If I decode it correctly, the streamid should be 0x2a which would be one > of the PCI devices on the internal root complex. Probably the network > card. Yes there is DMA transaction with an "unknown" StreamID. I think the StreamID is 0x42a. It means that there is a DMA master on the board with StreamID 0x42a that is either: - not described in device tree - described in device tree with a different StreamID - the right StreamID is described device tree, but it is not picked up by Xen > This is the first developer experience with Xen, so please bear with me > :) It seems that Xen doesn't add the master to the IOMMU. To me it seems > that only devices with a 'iommus' dt property are added. But in case of > PCI devices the parent only has a iommu-map property. > > And it makes me wonder why Leo has an almost working setup. Maybe I'm > missing some patches though. Xen 4.16 is able to parse StreamID in the "iommus" property and also "mmu-masters" property. But It is not able to parse the "iommu-map" property yet. So if 0x42a is described in device tree using "iommu-map" then the error makes sense. A simple solution is to replace iommu-map with iommus in device tree. It is possible that someone in CC to this email might already have a patch to introduce parsing of iommu-map in Xen.
Re: Xen data from meta-virtualization layer
Hi all,
> + Zhiqiang Hou
>
> On Tue, 24 Nov 2020, Leo Krueger wrote:
> > > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > > Hi,
> > >
> > > I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > > before...) but then had to add the following node to my device tree
> > >
> > > gic_lpi_base: syscon@0x8000 {
> > > compatible = "gic-lpi-base";
> > > >>
> > > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > > >> could you clarify which flavor/version of Linux you are using?
> > > >
> > > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> > >
> > > Do you have a link to the Linux tree? Is there any additional patches on
> > > top of
> > > vanilla?
> >
> > Linux tree is found here:
> > https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> > (up to the latest commit in that branch)
FWIW, I'm the developer of the support for this board both in the
mentioned branch as well as upstream.
If you don't need graphics you shouldn't really be using the branch
above, but the latest vanilla kernel release.
> [...]
>
> > > Looking at the DT changes in [0], it looks like the node is not a child
> > > of gic@.
> > > So I think Xen will map the region to Dom0.
> > >
> > > There are two things that I can notice:
> > >1) This region is RAM, but I can't find any reserve node. Is there any
> > > specific
> > > code in Linux to reserve it?
> > >2) The implementation in U-boot seems to suggest that the firmware will
> > > configure the LPIs and then enable it. If that's the case, then Xen needs
> > > to
> > > re-use the table in the DT rather than allocating a new one.
>
> That Linux tree has no mentions of gic-lpi-base. That means that
> gic-lpi-base is only used in u-boot, not in Linux. In particular the
> most relevant commit is af288cb291da3abef6be0875527729296f7de7a0.
That node was horrible hack from NXP for u-boot and was removed in
a84cea06bb8fff69810a890ac0e4b47ea5726512.
> In regards to the reserved-memory regions, maybe we are not seeing them
> because Leo posted the host device tree, not the one passed at runtime
> from u-boot to Linux?
>
> If so, Leo, could you please boot Linux on native (no Xen) and get the
> device tree from there at runtime using dtc -I fs -O dts
> /proc/device-tree ?
>
>
> However, the name of the reserved-memory region created by u-boot seems
> to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> Linux kernel tree either.
>
> Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> throws errors in regards to GIC/ITS initialization. On other hardware
> Xen can use and virtualize GICv3 and ITS just fine. Could you please
> explain what is different about sAL28 and how Xen/Linux is expected to
> use the lpi_rd_table reserved-memory region?
I actually stumbled across this thread after trying out Xen myself. I'm
using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel
and vanilla Xen.
So far I've discovered, that xen complains that it cannot route IRQ64 to
dom0. That is because on the LS1028A there is a dual UART (two 16550
with one shared interrupt) and xen takes the first UART and then tries
to map the interrupt of the second UART to linux. For now, I don't know
how this is solved correctly. As a quick hack, I removed the second uart
node from the device tree.
But what is more severe is that the iommu isn't set up correctly. I'm
getting the following faults:
(XEN) smmu: /soc/iommu@500: Unexpected global fault, this could be serious
(XEN) smmu: /soc/iommu@500: GFSR 0x8002, GFSYNR0 0x,
GFSYNR1 0x042a, GFSYNR2 0x
If I decode it correctly, the streamid should be 0x2a which would be one
of the PCI devices on the internal root complex. Probably the network
card.
This is the first developer experience with Xen, so please bear with me
:) It seems that Xen doesn't add the master to the IOMMU. To me it seems
that only devices with a 'iommus' dt property are added. But in case of
PCI devices the parent only has a iommu-map property.
And it makes me wonder why Leo has an almost working setup. Maybe I'm
missing some patches though.
-michael
Re: AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
+ Zhiqiang Hou
On Tue, 24 Nov 2020, Leo Krueger wrote:
> > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > Hi,
> >
> > I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > before...) but then had to add the following node to my device tree
> >
> > gic_lpi_base: syscon@0x8000 {
> > compatible = "gic-lpi-base";
> > >>
> > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > >> could you clarify which flavor/version of Linux you are using?
> > >
> > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> >
> > Do you have a link to the Linux tree? Is there any additional patches on
> > top of
> > vanilla?
>
> Linux tree is found here:
> https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> (up to the latest commit in that branch)
[...]
> > Looking at the DT changes in [0], it looks like the node is not a child of
> > gic@.
> > So I think Xen will map the region to Dom0.
> >
> > There are two things that I can notice:
> >1) This region is RAM, but I can't find any reserve node. Is there any
> > specific
> > code in Linux to reserve it?
> >2) The implementation in U-boot seems to suggest that the firmware will
> > configure the LPIs and then enable it. If that's the case, then Xen needs to
> > re-use the table in the DT rather than allocating a new one.
That Linux tree has no mentions of gic-lpi-base. That means that
gic-lpi-base is only used in u-boot, not in Linux. In particular the
most relevant commit is af288cb291da3abef6be0875527729296f7de7a0.
In regards to the reserved-memory regions, maybe we are not seeing them
because Leo posted the host device tree, not the one passed at runtime
from u-boot to Linux?
If so, Leo, could you please boot Linux on native (no Xen) and get the
device tree from there at runtime using dtc -I fs -O dts
/proc/device-tree ?
However, the name of the reserved-memory region created by u-boot seems
to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
Linux kernel tree either.
Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
throws errors in regards to GIC/ITS initialization. On other hardware
Xen can use and virtualize GICv3 and ITS just fine. Could you please
explain what is different about sAL28 and how Xen/Linux is expected to
use the lpi_rd_table reserved-memory region?
AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Hi,
> -Ursprüngliche Nachricht-
> Von: Julien Grall
> Gesendet: Montag, 23. November 2020 19:27
> An: Leo Krueger ; Stefano Stabellini
>
> Cc: Peng Fan ; bru...@xilinx.com; Cornelia Bruelhart
> ; oleksandr_andrushche...@epam.com; xen-
> de...@lists.xenproject.org; bertrand.marq...@arm.com
> Betreff: Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
>
>
>
> On 22/11/2020 22:55, Leo Krueger wrote:
> > Hi Julien,
>
> Hi Leo,
>
> >
> > finally I could try out what you suggested, please find my answers inline.
>
> Thank you for sending the logs!
>
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Julien Grall
> >> Gesendet: Mittwoch, 18. November 2020 13:24
> >> An: Stefano Stabellini ; Leo Krueger
> >>
> >> Cc: Peng Fan ; bru...@xilinx.com; Cornelia
> >> Bruelhart ;
> >> oleksandr_andrushche...@epam.com; xen- de...@lists.xenproject.org;
> >> bertrand.marq...@arm.com
> >> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> >>
> >> Hi,
> >>
> >> On 17/11/2020 23:53, Stefano Stabellini wrote:
> >>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> >>> recent experience with GICv3 ITS than me and might be able to help.
> >>> I am attaching the device tree Leo sent a few days ago for reference.
> >>>
> >>>
> >>> Typically when you can set the ethernet link up and no packets are
> >>> exchanged it is because of a missing interrupt. In this case a
> >>> missing MSI.
> >>>
> >>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> >>> recently. It is expected to work correctly with MSIs in Dom0, right?
> >>
> >> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to
> >> boot Dom0. I haven't seen any failure on recent Xen. We are testing
> >> 4.11 and onwards on Thunder-X.
> >>
> >> However, it may be possible that some more work is necessary for
> >> other hardware (e.g. workaround, missing code...). See more below.
> >>
> >>>
> >>>
> >>>
> >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> >>>> Hi,
> >>>>
> >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >>>> before...) but then had to add the following node to my device tree
> >>>>
> >>>> gic_lpi_base: syscon@0x8000 {
> >>>> compatible = "gic-lpi-base";
> >>
> >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> >> could you clarify which flavor/version of Linux you are using?
> >
> > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
>
> Do you have a link to the Linux tree? Is there any additional patches on top
> of
> vanilla?
Linux tree is found here:
https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
(up to the latest commit in that branch)
>
> > While searching around the Internet for any solution, I came across [0]
> which contained the gic-lpi-base node.
> > So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
>
> I am slightly confused to how this would help. Xen and, AFAICT, Linux don't
> understand gic-lpi-base. Do you have modification in your Linux to use it?
I have no idea, to be honest. Maybe it is about the memory being reserved due
to that node or something like that?
>
> Looking at the DT changes in [0], it looks like the node is not a child of
> gic@.
> So I think Xen will map the region to Dom0.
>
> There are two things that I can notice:
>1) This region is RAM, but I can't find any reserve node. Is there any
> specific
> code in Linux to reserve it?
>2) The implementation in U-boot seems to suggest that the firmware will
> configure the LPIs and then enable it. If that's the case, then Xen needs to
> re-use the table in the DT rather than allocating a new one.
> However, I would have expected an error message in the log:
>
> "GICv3: CPUx: Cannot initialize LPIs"
>
> At least Xen should not expose gic-lpi-base to the kernel, but I will wait on
> more details about the Linux kernel used before commenting more.
>
> I would also be interested to know more details about the failure when gic-
> lpi-base is not added in your DT. In particular, I am interested to understand
> why Xen would not expose the ITS as we d
AW: Xen data from meta-virtualization layer
Hi, Thanks for your effort! > -Ursprüngliche Nachricht- > Von: Julien Grall > Gesendet: Montag, 23. November 2020 19:42 > An: Rahul Singh ; Leo Krueger > > Cc: Stefano Stabellini ; Peng Fan > ; bru...@xilinx.com; Cornelia Bruelhart > ; oleksandr_andrushche...@epam.com; xen- > de...@lists.xenproject.org; Bertrand Marquis > > Betreff: Re: Xen data from meta-virtualization layer > > > > On 23/11/2020 11:41, Rahul Singh wrote: > > Hello , > > Hi Rahul, > > >> On 22 Nov 2020, at 10:55 pm, Leo Krueger wrote: > >> root@kontron-sal28:~# ip link set up dev gbe0 > >> (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 0017000c > >> 0001 > >> (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0005 > > >> [ 34.034598] Atheros 8031 ethernet :00:00.3:05: attached PHY driver > [Atheros 8031 ethernet] (mii_bus:phy_addr=:00:00.3:05, irq=POLL) > >> [ 34.04] 8021q: adding VLAN 0 to HW filter on device gbe0 > >> [ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready > >> root@kontron-sal28:~# [ 35.041951] fsl_enetc :00:00.0 gbe0: Link is > Down > >> [ 38.114426] fsl_enetc :00:00.0 gbe0: Link is Up - 1Gbps/Full - flow > control off > >> [ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes > ready > >> > >> Does that tell you anything? > >> > > > > I just checked the logs shared, what I found out that there’s is an error > while booting to configure the MSI for the PCI device because of that there > will be case that Device Id generate out-of-band is not mapped correctly to > ITS device table created while initialising the MSI for the device. > > I might be wrong let someone else also comments on this. > > I think there might be multiple issues. You spotted one below :). > > > [0.173964] OF: /soc/pcie@1f000: Invalid msi-map translation - no > match for rid 0xf8 on (null) > > Leo, just to confirm, this error message is not spotted when booting Linux on > baremetal? In fact it is: [0.160077] OF: /soc/pcie@1f000: Invalid msi-map translation - no match for rid 0xf8 on (null) But everything works as expected here: 110: 34288 0 ITS-MSI 1 Edge gbe0-rxtx0 111: 0 6196 ITS-MSI 2 Edge gbe0-rxtx1 > > Cheers, Best wishes > > -- > Julien Grall
Re: Xen data from meta-virtualization layer
On 23/11/2020 11:41, Rahul Singh wrote: Hello , Hi Rahul, On 22 Nov 2020, at 10:55 pm, Leo Krueger wrote: root@kontron-sal28:~# ip link set up dev gbe0 (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 0017000c 0001 (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0005 [ 34.034598] Atheros 8031 ethernet :00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=:00:00.3:05, irq=POLL) [ 34.04] 8021q: adding VLAN 0 to HW filter on device gbe0 [ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready root@kontron-sal28:~# [ 35.041951] fsl_enetc :00:00.0 gbe0: Link is Down [ 38.114426] fsl_enetc :00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off [ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready Does that tell you anything? I just checked the logs shared, what I found out that there’s is an error while booting to configure the MSI for the PCI device because of that there will be case that Device Id generate out-of-band is not mapped correctly to ITS device table created while initialising the MSI for the device. I might be wrong let someone else also comments on this. I think there might be multiple issues. You spotted one below :). [0.173964] OF: /soc/pcie@1f000: Invalid msi-map translation - no match for rid 0xf8 on (null) Leo, just to confirm, this error message is not spotted when booting Linux on baremetal? Cheers, -- Julien Grall
Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
On 22/11/2020 22:55, Leo Krueger wrote:
Hi Julien,
Hi Leo,
finally I could try out what you suggested, please find my answers inline.
Thank you for sending the logs!
-Ursprüngliche Nachricht-
Von: Julien Grall
Gesendet: Mittwoch, 18. November 2020 13:24
An: Stefano Stabellini ; Leo Krueger
Cc: Peng Fan ; bru...@xilinx.com; Cornelia Bruelhart
; oleksandr_andrushche...@epam.com; xen-
de...@lists.xenproject.org; bertrand.marq...@arm.com
Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
Hi,
On 17/11/2020 23:53, Stefano Stabellini wrote:
Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.
Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.
Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?
OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
onwards on Thunder-X.
However, it may be possible that some more work is necessary for other
hardware (e.g. workaround, missing code...). See more below.
On Tue, 17 Nov 2020, Leo Krueger wrote:
Hi,
I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
before...) but then had to add the following node to my device tree
gic_lpi_base: syscon@0x8000 {
compatible = "gic-lpi-base";
I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
you clarify which flavor/version of Linux you are using?
It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
Do you have a link to the Linux tree? Is there any additional patches on
top of vanilla?
While searching around the Internet for any solution, I came across [0] which
contained the gic-lpi-base node.
So I just tried adding it (quite desperate I know) and voila, it at least
brought me one step further (XEN exposing the ITS)...
I am slightly confused to how this would help. Xen and, AFAICT, Linux
don't understand gic-lpi-base. Do you have modification in your Linux to
use it?
Looking at the DT changes in [0], it looks like the node is not a child
of gic@. So I think Xen will map the region to Dom0.
There are two things that I can notice:
1) This region is RAM, but I can't find any reserve node. Is there
any specific code in Linux to reserve it?
2) The implementation in U-boot seems to suggest that the firmware
will configure the LPIs and then enable it. If that's the case, then Xen
needs to re-use the table in the DT rather than allocating a new one.
However, I would have expected an error message in the log:
"GICv3: CPUx: Cannot initialize LPIs"
At least Xen should not expose gic-lpi-base to the kernel, but I will
wait on more details about the Linux kernel used before commenting more.
I would also be interested to know more details about the failure when
gic-lpi-base is not added in your DT. In particular, I am interested to
understand why Xen would not expose the ITS as we don't parse that node.
[...]
For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know,
quite ugly in parts).
No worries, debug patches are not meant to be nice to read ;).
Find attached the boot log and an output of "xl dmesg" which is truncated due
to the large amount of messages.
When enabling the network interface (gbe0), the following output is visible:
root@kontron-sal28:~# ip link set up dev gbe0
(XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 0017000c 0001
(XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0005
0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt
by writing in the property table (access are not trapped to Xen) and
then requested to invalidate the cache state.
[ 34.034598] Atheros 8031 ethernet :00:00.3:05: attached PHY driver
[Atheros 8031 ethernet] (mii_bus:phy_addr=:00:00.3:05, irq=POLL)
[ 34.04] 8021q: adding VLAN 0 to HW filter on device gbe0
[ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
root@kontron-sal28:~# [ 35.041951] fsl_enetc :00:00.0 gbe0: Link is Down
[ 38.114426] fsl_enetc :00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
control off
[ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
Does that tell you anything?
It is at least a good sign because it means Linux is able to
initialize/talk to the vITS.
I would lean towards one (or multiple) issue with pITS and/or the
device-tree exposed to Linux. I am not entirely what exactly... I think
having more detai
Re: Xen data from meta-virtualization layer
Hello ,
> On 22 Nov 2020, at 10:55 pm, Leo Krueger wrote:
>
> Hi Julien,
>
> finally I could try out what you suggested, please find my answers inline.
>
>> -Ursprüngliche Nachricht-
>> Von: Julien Grall
>> Gesendet: Mittwoch, 18. November 2020 13:24
>> An: Stefano Stabellini ; Leo Krueger
>>
>> Cc: Peng Fan ; bru...@xilinx.com; Cornelia Bruelhart
>> ; oleksandr_andrushche...@epam.com; xen-
>> de...@lists.xenproject.org; bertrand.marq...@arm.com
>> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>>
>> Hi,
>>
>> On 17/11/2020 23:53, Stefano Stabellini wrote:
>>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
>>> recent experience with GICv3 ITS than me and might be able to help.
>>> I am attaching the device tree Leo sent a few days ago for reference.
>>>
>>>
>>> Typically when you can set the ethernet link up and no packets are
>>> exchanged it is because of a missing interrupt. In this case a missing
>>> MSI.
>>>
>>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
>>> recently. It is expected to work correctly with MSIs in Dom0, right?
>>
>> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
>> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
>> onwards on Thunder-X.
>>
>> However, it may be possible that some more work is necessary for other
>> hardware (e.g. workaround, missing code...). See more below.
>>
>>>
>>>
>>>
>>> On Tue, 17 Nov 2020, Leo Krueger wrote:
>>>> Hi,
>>>>
>>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>>>> before...) but then had to add the following node to my device tree
>>>>
>>>>gic_lpi_base: syscon@0x8000 {
>>>>compatible = "gic-lpi-base";
>>
>> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
>> you clarify which flavor/version of Linux you are using?
>
> It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> While searching around the Internet for any solution, I came across [0] which
> contained the gic-lpi-base node.
> So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
>
>>
>>>>reg = <0x0 0x8000 0x0 0x10>;
>>>>max-gic-redistributors = <2>;
>>>>};
>>>>
>>>> to somehow change something in regard to the ITS and MSI/MSI-X
>>>>
>>>> (XEN) GICv3 initialization:
>>>> (XEN) gic_dist_addr=0x000600
>>>> (XEN) gic_maintenance_irq=25
>>>> (XEN) gic_rdist_stride=0
>>>> (XEN) gic_rdist_regions=1
>>>> (XEN) redistributor regions:
>>>> (XEN) - region 0: 0x000604 - 0x000608
>>>> (XEN) GICv3: using at most 57344 LPIs on the host.
>>>> (XEN) GICv3: 288 lines, (IID 0001143b).
>>>> (XEN) GICv3: Found ITS @0x602
>>>> (XEN) using non-cacheable ITS command queue
>>>> (XEN) GICv3: CPU0: Found redistributor in region 0 @4001c000
>>>>
>>>> [0.00] GICv3: Distributor has no Range Selector support
>>>> [0.00] GICv3: no VLPI support, no direct LPI support
>>>> [0.00] ITS [mem 0x0602-0x0603]
>>>> [0.00] ITS@0x0602: allocated 65536 Devices
>> @dc88 (flat, esz 8, psz 64K, shr 1)
>>>> [0.00] ITS@0x0602: allocated 32768 Interrupt
>> Collections @dc82 (flat, esz 2, psz 64K, shr 1)
>>>> [0.00] GIC: using LPI property table @0xdc83
>>>> [0.00] GICv3: CPU0: found redistributor 0 region
>> 0:0x0604
>>>> [0.00] CPU0: using LPI pending table @0xdc84
>>>> ...
>>>> [0.040080] Platform MSI: gic-its domain created
>>>> [0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>>>> [0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>>>>
>>>>
>>>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
>> might fail!" log messages for some PCI devices, but at least the on-board
>> ethernet ports (fsl_enetc ) are initialized.
>>>> I can set the li
AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Hi Julien,
finally I could try out what you suggested, please find my answers inline.
> -Ursprüngliche Nachricht-
> Von: Julien Grall
> Gesendet: Mittwoch, 18. November 2020 13:24
> An: Stefano Stabellini ; Leo Krueger
>
> Cc: Peng Fan ; bru...@xilinx.com; Cornelia Bruelhart
> ; oleksandr_andrushche...@epam.com; xen-
> de...@lists.xenproject.org; bertrand.marq...@arm.com
> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>
> Hi,
>
> On 17/11/2020 23:53, Stefano Stabellini wrote:
> > Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> > recent experience with GICv3 ITS than me and might be able to help.
> > I am attaching the device tree Leo sent a few days ago for reference.
> >
> >
> > Typically when you can set the ethernet link up and no packets are
> > exchanged it is because of a missing interrupt. In this case a missing
> > MSI.
> >
> > Bertrand, I believe you tried the GIC ITS driver with PCI devices
> > recently. It is expected to work correctly with MSIs in Dom0, right?
>
> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
> onwards on Thunder-X.
>
> However, it may be possible that some more work is necessary for other
> hardware (e.g. workaround, missing code...). See more below.
>
> >
> >
> >
> > On Tue, 17 Nov 2020, Leo Krueger wrote:
> >> Hi,
> >>
> >> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >> before...) but then had to add the following node to my device tree
> >>
> >>gic_lpi_base: syscon@0x8000 {
> >>compatible = "gic-lpi-base";
>
> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
> you clarify which flavor/version of Linux you are using?
It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
While searching around the Internet for any solution, I came across [0] which
contained the gic-lpi-base node.
So I just tried adding it (quite desperate I know) and voila, it at least
brought me one step further (XEN exposing the ITS)...
>
> >>reg = <0x0 0x8000 0x0 0x10>;
> >>max-gic-redistributors = <2>;
> >>};
> >>
> >> to somehow change something in regard to the ITS and MSI/MSI-X
> >>
> >> (XEN) GICv3 initialization:
> >> (XEN) gic_dist_addr=0x000600
> >> (XEN) gic_maintenance_irq=25
> >> (XEN) gic_rdist_stride=0
> >> (XEN) gic_rdist_regions=1
> >> (XEN) redistributor regions:
> >> (XEN) - region 0: 0x000604 - 0x000608
> >> (XEN) GICv3: using at most 57344 LPIs on the host.
> >> (XEN) GICv3: 288 lines, (IID 0001143b).
> >> (XEN) GICv3: Found ITS @0x602
> >> (XEN) using non-cacheable ITS command queue
> >> (XEN) GICv3: CPU0: Found redistributor in region 0 @4001c000
> >>
> >> [0.00] GICv3: Distributor has no Range Selector support
> >> [0.00] GICv3: no VLPI support, no direct LPI support
> >> [0.00] ITS [mem 0x0602-0x0603]
> >> [0.00] ITS@0x0602: allocated 65536 Devices
> @dc88 (flat, esz 8, psz 64K, shr 1)
> >> [0.00] ITS@0x0602: allocated 32768 Interrupt
> Collections @dc82 (flat, esz 2, psz 64K, shr 1)
> >> [0.00] GIC: using LPI property table @0xdc83
> >> [0.00] GICv3: CPU0: found redistributor 0 region
> 0:0x0604
> >> [0.00] CPU0: using LPI pending table @0xdc84
> >> ...
> >> [0.040080] Platform MSI: gic-its domain created
> >> [0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> >> [0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
> >>
> >>
> >> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
> might fail!" log messages for some PCI devices, but at least the on-board
> ethernet ports (fsl_enetc ) are initialized.
> >> I can set the link up and a link is successfully established.
>
> This message is normal. Xen on Arm is not yet aware of PCI devices and
> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
>
> However, this is not an issue because the virtual ITS implementation will
> allow dom0 to configure any devices.
>
> >>
> >> But (!) I cannot receive or transmit anything (no error message...)
Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
Hi,
On 17/11/2020 23:53, Stefano Stabellini wrote:
Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.
Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.
Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?
OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
onwards on Thunder-X.
However, it may be possible that some more work is necessary for other
hardware (e.g. workaround, missing code...). See more below.
On Tue, 17 Nov 2020, Leo Krueger wrote:
Hi,
I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...)
but then had to add the following node to my device tree
gic_lpi_base: syscon@0x8000 {
compatible = "gic-lpi-base";
I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
could you clarify which flavor/version of Linux you are using?
reg = <0x0 0x8000 0x0 0x10>;
max-gic-redistributors = <2>;
};
to somehow change something in regard to the ITS and MSI/MSI-X
(XEN) GICv3 initialization:
(XEN) gic_dist_addr=0x000600
(XEN) gic_maintenance_irq=25
(XEN) gic_rdist_stride=0
(XEN) gic_rdist_regions=1
(XEN) redistributor regions:
(XEN) - region 0: 0x000604 - 0x000608
(XEN) GICv3: using at most 57344 LPIs on the host.
(XEN) GICv3: 288 lines, (IID 0001143b).
(XEN) GICv3: Found ITS @0x602
(XEN) using non-cacheable ITS command queue
(XEN) GICv3: CPU0: Found redistributor in region 0 @4001c000
[0.00] GICv3: Distributor has no Range Selector support
[0.00] GICv3: no VLPI support, no direct LPI support
[0.00] ITS [mem 0x0602-0x0603]
[0.00] ITS@0x0602: allocated 65536 Devices @dc88 (flat,
esz 8, psz 64K, shr 1)
[0.00] ITS@0x0602: allocated 32768 Interrupt Collections
@dc82 (flat, esz 2, psz 64K, shr 1)
[0.00] GIC: using LPI property table @0xdc83
[0.00] GICv3: CPU0: found redistributor 0 region 0:0x0604
[0.00] CPU0: using LPI pending table @0xdc84
...
[0.040080] Platform MSI: gic-its domain created
[0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
[0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might
fail!" log messages for some PCI devices, but at least the on-board ethernet ports
(fsl_enetc ) are initialized.
I can set the link up and a link is successfully established.
This message is normal. Xen on Arm is not yet aware of PCI devices and
therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
However, this is not an issue because the virtual ITS implementation
will allow dom0 to configure any devices.
But (!) I cannot receive or transmit anything (no error message...) and there
seem to be no interrupts:
29: 0 ITS-MSI 1 Edge gbe0-rxtx0
32: 0 ITS-MSI 8192 Edge ptp_qoriq
(from /proc/interrupts).
Any idea on this one? I keep digging and checking whether my device tree needs
some additional fixes.
Can you apply patch [1] and provide the logs? This will dump more
information about the command received by the vITS and the one send to
the host ITS.
Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
some of the messages.
[...]
[ 0.00] GICv3: Distributor has no Range Selector support
[ 0.00] GICv3: no VLPI support, no direct LPI support
[ 0.00] GICv3: CPU0: found redistributor 0 region
0:0x0604
"no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
support.
VLPI is a feature that was introduced in GICv4 to directly inject LPI in
the guest. So this is normal to see this message when running on Xen
because we are going to only expose a GICv3 to a domain until at least
we support nested virt.
However, you were right about that Xen didn't expose the ITS because the
following lines were missing:
[0.00] ITS@0x0602: allocated 65536 Devices @dc88
(flat, esz 8, psz 64K, shr 1)
Cheers,
[1]
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9558bad96ac3..8a0a02308e74 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
const void *its_cmd)
/* No ITS commands from an interrupt handler (at the moment). */
ASSERT(!in_irq());
+printk(XENLO
Re: Xen data from meta-virtualization layer
Hello Stefano,
> On 17 Nov 2020, at 11:53 pm, Stefano Stabellini
> wrote:
>
> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> recent experience with GICv3 ITS than me and might be able to help.
> I am attaching the device tree Leo sent a few days ago for reference.
>
>
> Typically when you can set the ethernet link up and no packets are
> exchanged it is because of a missing interrupt. In this case a missing
> MSI.
>
> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> recently. It is expected to work correctly with MSIs in Dom0, right?
>
Yes we are using the XEN GIC ITS driver and MSI interrupts is working fine in
DOM0.
20:112 0 0 0 ITS-MSI 1572864 Edge eth0
21:441 0 0 0 ITS-MSI 3670016 Edge
eth1
22: 4286 0 0 0 ITS-MSI 4194304 Edge
xhci_hcd
Regards,
Rahul
>
> On Tue, 17 Nov 2020, Leo Krueger wrote:
>> Hi,
>>
>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>> before...) but then had to add the following node to my device tree
>>
>> gic_lpi_base: syscon@0x8000 {
>> compatible = "gic-lpi-base";
>> reg = <0x0 0x8000 0x0 0x10>;
>> max-gic-redistributors = <2>;
>> };
>>
>> to somehow change something in regard to the ITS and MSI/MSI-X
>>
>> (XEN) GICv3 initialization:
>> (XEN) gic_dist_addr=0x000600
>> (XEN) gic_maintenance_irq=25
>> (XEN) gic_rdist_stride=0
>> (XEN) gic_rdist_regions=1
>> (XEN) redistributor regions:
>> (XEN) - region 0: 0x000604 - 0x000608
>> (XEN) GICv3: using at most 57344 LPIs on the host.
>> (XEN) GICv3: 288 lines, (IID 0001143b).
>> (XEN) GICv3: Found ITS @0x602
>> (XEN) using non-cacheable ITS command queue
>> (XEN) GICv3: CPU0: Found redistributor in region 0 @4001c000
>>
>> [0.00] GICv3: Distributor has no Range Selector support
>> [0.00] GICv3: no VLPI support, no direct LPI support
>> [0.00] ITS [mem 0x0602-0x0603]
>> [0.00] ITS@0x0602: allocated 65536 Devices @dc88
>> (flat, esz 8, psz 64K, shr 1)
>> [0.00] ITS@0x0602: allocated 32768 Interrupt Collections
>> @dc82 (flat, esz 2, psz 64K, shr 1)
>> [0.00] GIC: using LPI property table @0xdc83
>> [0.00] GICv3: CPU0: found redistributor 0 region 0:0x0604
>> [0.00] CPU0: using LPI pending table @0xdc84
>> ...
>> [0.040080] Platform MSI: gic-its domain created
>> [0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>> [0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>>
>>
>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
>> might fail!" log messages for some PCI devices, but at least the on-board
>> ethernet ports (fsl_enetc ) are initialized.
>> I can set the link up and a link is successfully established.
>>
>> But (!) I cannot receive or transmit anything (no error message...) and
>> there seem to be no interrupts:
>>
>> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
>> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>>
>> (from /proc/interrupts).
>>
>> Any idea on this one? I keep digging and checking whether my device tree
>> needs some additional fixes.
>>
>> Kind regards,
>> Leo
>>
>> --
>> Leo Krüger, M.Sc.
>> Senior Systems Engineer Distributed Systems
>> Intelligent Digital Cabin
>>
>> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
>> Hein-Saß-Weg 22
>> 21129 Hamburg
>>
>> +49 (0) 40 248 595-154
>>
>> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter
>>
>> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
>> Sitz der Gesellschaft / Legal Domicile: Hamburg
>> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
>> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR
>> Andreas Rieckhof
>> Geschäftsführung / Board of Management: Roland Gerhards
>>
>> Disclaimer:
>> This e-mail may contain confidential and/or privileged information. If you
>> are not the intended recipient (or have
>> received this mail in error), please notify the sender immediately and
>> destroy this e-mail. Any unauthorised copying,
&g
Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.
Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.
Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?
On Tue, 17 Nov 2020, Leo Krueger wrote:
> Hi,
>
> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> before...) but then had to add the following node to my device tree
>
> gic_lpi_base: syscon@0x8000 {
> compatible = "gic-lpi-base";
> reg = <0x0 0x8000 0x0 0x10>;
> max-gic-redistributors = <2>;
> };
>
> to somehow change something in regard to the ITS and MSI/MSI-X
>
> (XEN) GICv3 initialization:
> (XEN) gic_dist_addr=0x000600
> (XEN) gic_maintenance_irq=25
> (XEN) gic_rdist_stride=0
> (XEN) gic_rdist_regions=1
> (XEN) redistributor regions:
> (XEN) - region 0: 0x000604 - 0x000608
> (XEN) GICv3: using at most 57344 LPIs on the host.
> (XEN) GICv3: 288 lines, (IID 0001143b).
> (XEN) GICv3: Found ITS @0x602
> (XEN) using non-cacheable ITS command queue
> (XEN) GICv3: CPU0: Found redistributor in region 0 @4001c000
>
> [0.00] GICv3: Distributor has no Range Selector support
> [0.00] GICv3: no VLPI support, no direct LPI support
> [0.00] ITS [mem 0x0602-0x0603]
> [0.00] ITS@0x0602: allocated 65536 Devices @dc88
> (flat, esz 8, psz 64K, shr 1)
> [0.00] ITS@0x0602: allocated 32768 Interrupt Collections
> @dc82 (flat, esz 2, psz 64K, shr 1)
> [0.00] GIC: using LPI property table @0xdc83
> [0.00] GICv3: CPU0: found redistributor 0 region 0:0x0604
> [0.00] CPU0: using LPI pending table @0xdc84
> ...
> [0.040080] Platform MSI: gic-its domain created
> [0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> [0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>
>
> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
> might fail!" log messages for some PCI devices, but at least the on-board
> ethernet ports (fsl_enetc ) are initialized.
> I can set the link up and a link is successfully established.
>
> But (!) I cannot receive or transmit anything (no error message...) and there
> seem to be no interrupts:
>
> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>
> (from /proc/interrupts).
>
> Any idea on this one? I keep digging and checking whether my device tree
> needs some additional fixes.
>
> Kind regards,
> Leo
>
> --
> Leo Krüger, M.Sc.
> Senior Systems Engineer Distributed Systems
> Intelligent Digital Cabin
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Hein-Saß-Weg 22
> 21129 Hamburg
>
> +49 (0) 40 248 595-154
>
> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Sitz der Gesellschaft / Legal Domicile: Hamburg
> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR
> Andreas Rieckhof
> Geschäftsführung / Board of Management: Roland Gerhards
>
> Disclaimer:
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have
> received this mail in error), please notify the sender immediately and
> destroy this e-mail. Any unauthorised copying,
> disclosure or distribution of the material in this e-mail is strictly
> forbidden.
>
> > -Ursprüngliche Nachricht-
> > Von: Stefano Stabellini
> > Gesendet: Dienstag, 17. November 2020 01:59
> > An: Leo Krueger
> > Cc: Peng Fan ; Stefano Stabellini
> > ; bru...@xilinx.com; Cornelia Bruelhart
> >
> > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
> >
> > Replies inline below
> >
> >
> > On Sun, 15 Nov 2020, Leo Krueger wrote:
> > > Hi Peng, hi Stefano,
> > >
> > >
> > >
> > > sorry for the long silence…
> > >
> > >
> > >
> > > I tried the change suggested (and hope I didn’t do anything wrong
> > > while doing so…) on top of XEN 4.13.2 (b
