Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello, Rahul!
On 11/6/20 4:41 PM, Rahul Singh wrote:
Hello Oleksandr,
On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko
wrote:
Hi, Rahul!
On 11/6/20 3:58 PM, Rahul Singh wrote:
Hello Oleksandr,
On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko
wrote:
Hello, Rahul!
On 11/6/20 2:48 PM, Rahul Singh wrote:
Hello Oleksandr,
On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko
wrote:
Hi,
On 11/2/20 11:55 AM, Bertrand Marquis wrote:
Hi,
On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote:
Hi, Julien!
On 10/30/20 7:18 PM, Julien Grall wrote:
Hi Oleksandr,
On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
On 10/20/20 6:25 PM, Rahul Singh wrote:
Add support for ARM architected SMMUv3 implementations. It is based on
the Linux SMMUv3 driver.
Major differences between the Linux driver are as follows:
1. Only Stage-2 translation is supported as compared to the Linux driver
that supports both Stage-1 and Stage-2 translations.
First of all thank you for the efforts!
I tried the patch with QEMU and would like to know if my understanding correct
that this combination will not work as of now:
(XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
(XEN) Data Abort Trap. Syndrome=0x1940010
(XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000
(XEN) 0TH[0x0] = 0xb8468f7f
[snip]
If this is expected then is there any plan to make QEMU work as well?
I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
Just for clarication, you are trying to boot Xen on QEMU, right?
Exactly
You might be able to use the stage-1 page-tables to isolate each device in Xen.
However, I don't think you will be able to share the P2M because the
page-tables layout between stage-1 and stage-2 is different.
So, it is even more work then
Overall it would make more sense to spend some time adding proper support in
Qemu then trying to modify the driver to support Qemu right now.
We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
implementation, so it could allow testing different setups and configurations
with QEMU.
I would recommend to get the SMMU supporting supporting stage-2 page-tables.
You mean in QEMU?
See before.
Regardless that, I think Xen should be able to say the SMMU is not supported
rather than crashing.
Yes, that would be nice
Fully agree and we will look into that.
Anything you could share so that we could quickly reproduce your setup would be
more then great.
Nothing special,
qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine
virt,gic-version=2 \
-machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic
user,hostfwd=tcp:127.0.0.1:-:22 \
-nographic -serial mon:stdio [..snip..]
I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN
is able to say SMMU translation is not supported. As XEN supports Stage-2
translation and QEMU supports Stage-1 only.
(XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration
(false)
(XEN) SMMUv3: /smmuv3@905: no translation support!
(XEN) I/O virtualisation disabled
Only difference I observed is that you have to add option "-machine
virt,iommu=smmuv3 “ when launching the QEMU.
I do use the option
I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and
while launching the QEMU.
I also observed the same error what you observed if I am not using the
"-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this
might be case for you also but anyways you have use the options it might be other
issue.
Hm, probably that was on my side as now I can see:
(XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration
(false)
(XEN) SMMUv3: /smmuv3@905: no translation support!
(XEN) I/O virtualisation disabled
(XEN)
(XEN)
(XEN) Panic on CPU 0:
(XEN) Couldn't configure correctly all the IOMMUs.
(XEN)
(XEN)
(XEN) Manual reset required ('noreboot' specified)
So, sorry for the noise, I might have misconfigured something it seems
When you say "Xen is booting" do you mean you see the same panic?
Yes I observe the same.
We have to decide now if for SMMUv3 there is no translation support do we have
to print the logs and move forward or as above return error to iommu_setup
that will cal panic().
I would allow Xen to boot further
Regards,
Rahul
Thank you,
Oleksandr
Please let me know if it also works for you.
Well, I should have reported that earlier that I do not use the staging Xen at
the moment,
it is 4.14.0. So, can this be a problem with that Xen version?
I don’t think so this is the problem with the XEN version.
Anyways, if it works with the staging
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Oleksandr, > On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko > wrote: > > Hi, Rahul! > > On 11/6/20 3:58 PM, Rahul Singh wrote: >> Hello Oleksandr, >> >>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko >>> wrote: >>> >>> Hello, Rahul! >>> >>> On 11/6/20 2:48 PM, Rahul Singh wrote: Hello Oleksandr, > On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko > wrote: > > Hi, > > On 11/2/20 11:55 AM, Bertrand Marquis wrote: >> Hi, >> >>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko >>> wrote: >>> >>> Hi, Julien! >>> >>> On 10/30/20 7:18 PM, Julien Grall wrote: Hi Oleksandr, On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: > On 10/20/20 6:25 PM, Rahul Singh wrote: >> Add support for ARM architected SMMUv3 implementations. It is based >> on >> the Linux SMMUv3 driver. >> >> Major differences between the Linux driver are as follows: >> 1. Only Stage-2 translation is supported as compared to the Linux >> driver >> that supports both Stage-1 and Stage-2 translations. > First of all thank you for the efforts! > > I tried the patch with QEMU and would like to know if my > understanding correct > > that this combination will not work as of now: > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > (XEN) Data Abort Trap. Syndrome=0x1940010 > (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR > 0xb8469000 > (XEN) 0TH[0x0] = 0xb8468f7f > > [snip] > > If this is expected then is there any plan to make QEMU work as well? > > I see [1] says that "Only stage 1 and AArch64 PTW are supported." on > QEMU side. Just for clarication, you are trying to boot Xen on QEMU, right? >>> Exactly You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different. >>> So, it is even more work then >> Overall it would make more sense to spend some time adding proper >> support in Qemu then trying to modify the driver to support Qemu right >> now. >> > We are interested in QEMU/SMMUv3 as a flexible platform for PCI > passthrough > > implementation, so it could allow testing different setups and > configurations with QEMU. I would recommend to get the SMMU supporting supporting stage-2 page-tables. >>> You mean in QEMU? >> See before. >> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing. >>> Yes, that would be nice >> Fully agree and we will look into that. >> >> Anything you could share so that we could quickly reproduce your setup >> would be more then great. > Nothing special, > > qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine > virt,gic-version=2 \ > > -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic > user,hostfwd=tcp:127.0.0.1:-:22 \ > > -nographic -serial mon:stdio [..snip..] > > I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only. (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration (false) (XEN) SMMUv3: /smmuv3@905: no translation support! (XEN) I/O virtualisation disabled Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU. >>> I do use the option >> I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and >> while launching the QEMU. >> I also observed the same error what you observed if I am not using the >> "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought >> this might be case for you also but anyways you have use the options it >> might be other issue. > > Hm, probably that was on my side as now I can see: > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration > (false) > (XEN) SMMUv3: /smmuv3@905: no translation support! > (XEN) I/O virtualisation disabled > (XEN) > (XEN) > (XEN) Panic on CPU 0: > (XEN) Couldn't configure correctly all the IOMMUs. > (XEN)
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, Rahul!
On 11/6/20 3:58 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko
>> wrote:
>>
>> Hello, Rahul!
>>
>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>> Hello Oleksandr,
>>>
On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko
wrote:
Hi,
On 11/2/20 11:55 AM, Bertrand Marquis wrote:
> Hi,
>
>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko
>> wrote:
>>
>> Hi, Julien!
>>
>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>> Hi Oleksandr,
>>>
>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
On 10/20/20 6:25 PM, Rahul Singh wrote:
> Add support for ARM architected SMMUv3 implementations. It is based on
> the Linux SMMUv3 driver.
>
> Major differences between the Linux driver are as follows:
> 1. Only Stage-2 translation is supported as compared to the Linux
> driver
> that supports both Stage-1 and Stage-2 translations.
First of all thank you for the efforts!
I tried the patch with QEMU and would like to know if my understanding
correct
that this combination will not work as of now:
(XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
(XEN) Data Abort Trap. Syndrome=0x1940010
(XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR
0xb8469000
(XEN) 0TH[0x0] = 0xb8468f7f
[snip]
If this is expected then is there any plan to make QEMU work as well?
I see [1] says that "Only stage 1 and AArch64 PTW are supported." on
QEMU side.
>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>> Exactly
>>> You might be able to use the stage-1 page-tables to isolate each device
>>> in Xen. However, I don't think you will be able to share the P2M
>>> because the page-tables layout between stage-1 and stage-2 is different.
>> So, it is even more work then
> Overall it would make more sense to spend some time adding proper support
> in Qemu then trying to modify the driver to support Qemu right now.
>
We are interested in QEMU/SMMUv3 as a flexible platform for PCI
passthrough
implementation, so it could allow testing different setups and
configurations with QEMU.
>>> I would recommend to get the SMMU supporting supporting stage-2
>>> page-tables.
>> You mean in QEMU?
> See before.
>
>>> Regardless that, I think Xen should be able to say the SMMU is not
>>> supported rather than crashing.
>> Yes, that would be nice
> Fully agree and we will look into that.
>
> Anything you could share so that we could quickly reproduce your setup
> would be more then great.
Nothing special,
qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine
virt,gic-version=2 \
-machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic
user,hostfwd=tcp:127.0.0.1:-:22 \
-nographic -serial mon:stdio [..snip..]
I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and
>>> XEN is able to say SMMU translation is not supported. As XEN supports
>>> Stage-2 translation and QEMU supports Stage-1 only.
>>>
>>>
>>> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
>>> (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration
>>> (false)
>>> (XEN) SMMUv3: /smmuv3@905: no translation support!
>>> (XEN) I/O virtualisation disabled
>>>
>>> Only difference I observed is that you have to add option "-machine
>>> virt,iommu=smmuv3 “ when launching the QEMU.
>> I do use the option
> I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and
> while launching the QEMU.
> I also observed the same error what you observed if I am not using the
> "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought
> this might be case for you also but anyways you have use the options it might
> be other issue.
Hm, probably that was on my side as now I can see:
(XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration
(false)
(XEN) SMMUv3: /smmuv3@905: no translation support!
(XEN) I/O virtualisation disabled
(XEN)
(XEN)
(XEN) Panic on CPU 0:
(XEN) Couldn't configure correctly all the IOMMUs.
(XEN)
(XEN)
(XEN) Manual reset required ('noreboot' specified)
So, sorry for the noise, I might have misconfigured something it seems
When you say "Xen is booting" do you mean you see the same panic?
Thank you,
Oleksandr
>
>>> Please let me know if it also works
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Oleksandr, > On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko > wrote: > > Hello, Rahul! > > On 11/6/20 2:48 PM, Rahul Singh wrote: >> Hello Oleksandr, >> >>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko >>> wrote: >>> >>> Hi, >>> >>> On 11/2/20 11:55 AM, Bertrand Marquis wrote: Hi, > On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko > wrote: > > Hi, Julien! > > On 10/30/20 7:18 PM, Julien Grall wrote: >> Hi Oleksandr, >> >> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: >>> On 10/20/20 6:25 PM, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. >>> First of all thank you for the efforts! >>> >>> I tried the patch with QEMU and would like to know if my understanding >>> correct >>> >>> that this combination will not work as of now: >>> >>> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >>> (XEN) Data Abort Trap. Syndrome=0x1940010 >>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR >>> 0xb8469000 >>> (XEN) 0TH[0x0] = 0xb8468f7f >>> >>> [snip] >>> >>> If this is expected then is there any plan to make QEMU work as well? >>> >>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on >>> QEMU side. >> Just for clarication, you are trying to boot Xen on QEMU, right? > Exactly >> You might be able to use the stage-1 page-tables to isolate each device >> in Xen. However, I don't think you will be able to share the P2M because >> the page-tables layout between stage-1 and stage-2 is different. > So, it is even more work then Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now. >>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI >>> passthrough >>> >>> implementation, so it could allow testing different setups and >>> configurations with QEMU. >> I would recommend to get the SMMU supporting supporting stage-2 >> page-tables. > You mean in QEMU? See before. >> Regardless that, I think Xen should be able to say the SMMU is not >> supported rather than crashing. > Yes, that would be nice Fully agree and we will look into that. Anything you could share so that we could quickly reproduce your setup would be more then great. >>> Nothing special, >>> >>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine >>> virt,gic-version=2 \ >>> >>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic >>> user,hostfwd=tcp:127.0.0.1:-:22 \ >>> >>> -nographic -serial mon:stdio [..snip..] >>> >>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 >> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and >> XEN is able to say SMMU translation is not supported. As XEN supports >> Stage-2 translation and QEMU supports Stage-1 only. >> >> >> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >> (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration >> (false) >> (XEN) SMMUv3: /smmuv3@905: no translation support! >> (XEN) I/O virtualisation disabled >> >> Only difference I observed is that you have to add option "-machine >> virt,iommu=smmuv3 “ when launching the QEMU. > I do use the option I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and while launching the QEMU. I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue. >> >> Please let me know if it also works for you. > > Well, I should have reported that earlier that I do not use the staging Xen > at the moment, > > it is 4.14.0. So, can this be a problem with that Xen version? I don’t think so this is the problem with the XEN version. > > Anyways, if it works with the staging then everything looks ok > > Thank you, > > Oleksandr > >> Regards Bertrand >> Cheers, >> > Thank you, > > Oleksandr Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello, Rahul! On 11/6/20 2:48 PM, Rahul Singh wrote: > Hello Oleksandr, > >> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko >> wrote: >> >> Hi, >> >> On 11/2/20 11:55 AM, Bertrand Marquis wrote: >>> Hi, >>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote: Hi, Julien! On 10/30/20 7:18 PM, Julien Grall wrote: > Hi Oleksandr, > > On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: >> On 10/20/20 6:25 PM, Rahul Singh wrote: >>> Add support for ARM architected SMMUv3 implementations. It is based on >>> the Linux SMMUv3 driver. >>> >>> Major differences between the Linux driver are as follows: >>> 1. Only Stage-2 translation is supported as compared to the Linux driver >>> that supports both Stage-1 and Stage-2 translations. >> First of all thank you for the efforts! >> >> I tried the patch with QEMU and would like to know if my understanding >> correct >> >> that this combination will not work as of now: >> >> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >> (XEN) Data Abort Trap. Syndrome=0x1940010 >> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR >> 0xb8469000 >> (XEN) 0TH[0x0] = 0xb8468f7f >> >> [snip] >> >> If this is expected then is there any plan to make QEMU work as well? >> >> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on >> QEMU side. > Just for clarication, you are trying to boot Xen on QEMU, right? Exactly > You might be able to use the stage-1 page-tables to isolate each device > in Xen. However, I don't think you will be able to share the P2M because > the page-tables layout between stage-1 and stage-2 is different. So, it is even more work then >>> Overall it would make more sense to spend some time adding proper support >>> in Qemu then trying to modify the driver to support Qemu right now. >>> >> We are interested in QEMU/SMMUv3 as a flexible platform for PCI >> passthrough >> >> implementation, so it could allow testing different setups and >> configurations with QEMU. > I would recommend to get the SMMU supporting supporting stage-2 > page-tables. You mean in QEMU? >>> See before. >>> > Regardless that, I think Xen should be able to say the SMMU is not > supported rather than crashing. Yes, that would be nice >>> Fully agree and we will look into that. >>> >>> Anything you could share so that we could quickly reproduce your setup >>> would be more then great. >> Nothing special, >> >> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine >> virt,gic-version=2 \ >> >> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic >> user,hostfwd=tcp:127.0.0.1:-:22 \ >> >> -nographic -serial mon:stdio [..snip..] >> >> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 > I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and > XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 > translation and QEMU supports Stage-1 only. > > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration > (false) > (XEN) SMMUv3: /smmuv3@905: no translation support! > (XEN) I/O virtualisation disabled > > Only difference I observed is that you have to add option "-machine > virt,iommu=smmuv3 “ when launching the QEMU. I do use the option > > Please let me know if it also works for you. Well, I should have reported that earlier that I do not use the staging Xen at the moment, it is 4.14.0. So, can this be a problem with that Xen version? Anyways, if it works with the staging then everything looks ok Thank you, Oleksandr > >>> Regards >>> Bertrand >>> > Cheers, > Thank you, Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Oleksandr, > On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko > wrote: > > Hi, > > On 11/2/20 11:55 AM, Bertrand Marquis wrote: >> Hi, >> >>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote: >>> >>> Hi, Julien! >>> >>> On 10/30/20 7:18 PM, Julien Grall wrote: Hi Oleksandr, On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: > On 10/20/20 6:25 PM, Rahul Singh wrote: >> Add support for ARM architected SMMUv3 implementations. It is based on >> the Linux SMMUv3 driver. >> >> Major differences between the Linux driver are as follows: >> 1. Only Stage-2 translation is supported as compared to the Linux driver >> that supports both Stage-1 and Stage-2 translations. > First of all thank you for the efforts! > > I tried the patch with QEMU and would like to know if my understanding > correct > > that this combination will not work as of now: > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > (XEN) Data Abort Trap. Syndrome=0x1940010 > (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 > (XEN) 0TH[0x0] = 0xb8468f7f > > [snip] > > If this is expected then is there any plan to make QEMU work as well? > > I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU > side. Just for clarication, you are trying to boot Xen on QEMU, right? >>> Exactly You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different. >>> So, it is even more work then >> Overall it would make more sense to spend some time adding proper support in >> Qemu then trying to modify the driver to support Qemu right now. >> > > We are interested in QEMU/SMMUv3 as a flexible platform for PCI > passthrough > > implementation, so it could allow testing different setups and > configurations with QEMU. I would recommend to get the SMMU supporting supporting stage-2 page-tables. >>> You mean in QEMU? >> See before. >> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing. >>> Yes, that would be nice >> Fully agree and we will look into that. >> >> Anything you could share so that we could quickly reproduce your setup would >> be more then great. > > Nothing special, > > qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine > virt,gic-version=2 \ > > -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic > user,hostfwd=tcp:127.0.0.1:-:22 \ > > -nographic -serial mon:stdio [..snip..] > > I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only. (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) SMMUv3: /smmuv3@905: IDR0.COHACC overridden by FW configuration (false) (XEN) SMMUv3: /smmuv3@905: no translation support! (XEN) I/O virtualisation disabled Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU. Please let me know if it also works for you. > >> >> Regards >> Bertrand >> Cheers, >>> Thank you, >>> >>> Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Oleksandr, > On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko > wrote: > > Hi, > > On 11/2/20 11:55 AM, Bertrand Marquis wrote: >> Hi, >> >>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote: >>> >>> Hi, Julien! >>> >>> On 10/30/20 7:18 PM, Julien Grall wrote: Hi Oleksandr, On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: > On 10/20/20 6:25 PM, Rahul Singh wrote: >> Add support for ARM architected SMMUv3 implementations. It is based on >> the Linux SMMUv3 driver. >> >> Major differences between the Linux driver are as follows: >> 1. Only Stage-2 translation is supported as compared to the Linux driver >> that supports both Stage-1 and Stage-2 translations. > First of all thank you for the efforts! > > I tried the patch with QEMU and would like to know if my understanding > correct > > that this combination will not work as of now: > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > (XEN) Data Abort Trap. Syndrome=0x1940010 > (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 > (XEN) 0TH[0x0] = 0xb8468f7f > > [snip] > > If this is expected then is there any plan to make QEMU work as well? > > I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU > side. Just for clarication, you are trying to boot Xen on QEMU, right? >>> Exactly You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different. >>> So, it is even more work then >> Overall it would make more sense to spend some time adding proper support in >> Qemu then trying to modify the driver to support Qemu right now. >> > > We are interested in QEMU/SMMUv3 as a flexible platform for PCI > passthrough > > implementation, so it could allow testing different setups and > configurations with QEMU. I would recommend to get the SMMU supporting supporting stage-2 page-tables. >>> You mean in QEMU? >> See before. >> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing. >>> Yes, that would be nice >> Fully agree and we will look into that. >> >> Anything you could share so that we could quickly reproduce your setup would >> be more then great. > > Nothing special, > > qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine > virt,gic-version=2 \ > > -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic > user,hostfwd=tcp:127.0.0.1:-:22 \ > > -nographic -serial mon:stdio [..snip..] > > I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 Thanks for sharing the information. I will also try to reproduce the issue and will let you know the results. > >> >> Regards >> Bertrand >> Cheers, >>> Thank you, >>> >>> Oleksandr Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, On 11/2/20 11:55 AM, Bertrand Marquis wrote: > Hi, > >> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote: >> >> Hi, Julien! >> >> On 10/30/20 7:18 PM, Julien Grall wrote: >>> Hi Oleksandr, >>> >>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: On 10/20/20 6:25 PM, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is based on > the Linux SMMUv3 driver. > > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux driver > that supports both Stage-1 and Stage-2 translations. First of all thank you for the efforts! I tried the patch with QEMU and would like to know if my understanding correct that this combination will not work as of now: (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) Data Abort Trap. Syndrome=0x1940010 (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 (XEN) 0TH[0x0] = 0xb8468f7f [snip] If this is expected then is there any plan to make QEMU work as well? I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side. >>> Just for clarication, you are trying to boot Xen on QEMU, right? >> Exactly >>> You might be able to use the stage-1 page-tables to isolate each device in >>> Xen. However, I don't think you will be able to share the P2M because the >>> page-tables layout between stage-1 and stage-2 is different. >> So, it is even more work then > Overall it would make more sense to spend some time adding proper support in > Qemu then trying to modify the driver to support Qemu right now. > We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough implementation, so it could allow testing different setups and configurations with QEMU. >>> I would recommend to get the SMMU supporting supporting stage-2 page-tables. >> You mean in QEMU? > See before. > >>> Regardless that, I think Xen should be able to say the SMMU is not >>> supported rather than crashing. >> Yes, that would be nice > Fully agree and we will look into that. > > Anything you could share so that we could quickly reproduce your setup would > be more then great. Nothing special, qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \ -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:-:22 \ -nographic -serial mon:stdio [..snip..] I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1 > > Regards > Bertrand > >>> Cheers, >>> >> Thank you, >> >> Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, > On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko wrote: > > Hi, Julien! > > On 10/30/20 7:18 PM, Julien Grall wrote: >> Hi Oleksandr, >> >> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: >>> On 10/20/20 6:25 PM, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. >>> >>> First of all thank you for the efforts! >>> >>> I tried the patch with QEMU and would like to know if my understanding >>> correct >>> >>> that this combination will not work as of now: >>> >>> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >>> (XEN) Data Abort Trap. Syndrome=0x1940010 >>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 >>> (XEN) 0TH[0x0] = 0xb8468f7f >>> >>> [snip] >>> >>> If this is expected then is there any plan to make QEMU work as well? >>> >>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU >>> side. >> >> Just for clarication, you are trying to boot Xen on QEMU, right? > Exactly >> >> You might be able to use the stage-1 page-tables to isolate each device in >> Xen. However, I don't think you will be able to share the P2M because the >> page-tables layout between stage-1 and stage-2 is different. > So, it is even more work then Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now. >> >>> >>> >>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough >>> >>> implementation, so it could allow testing different setups and >>> configurations with QEMU. >> >> I would recommend to get the SMMU supporting supporting stage-2 page-tables. > You mean in QEMU? See before. >> >> Regardless that, I think Xen should be able to say the SMMU is not supported >> rather than crashing. > Yes, that would be nice Fully agree and we will look into that. Anything you could share so that we could quickly reproduce your setup would be more then great. Regards Bertrand >> >> Cheers, >> > Thank you, > > Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, Julien! On 10/30/20 7:18 PM, Julien Grall wrote: Hi Oleksandr, On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: On 10/20/20 6:25 PM, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. First of all thank you for the efforts! I tried the patch with QEMU and would like to know if my understanding correct that this combination will not work as of now: (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) Data Abort Trap. Syndrome=0x1940010 (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 (XEN) 0TH[0x0] = 0xb8468f7f [snip] If this is expected then is there any plan to make QEMU work as well? I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side. Just for clarication, you are trying to boot Xen on QEMU, right? Exactly You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different. So, it is even more work then We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough implementation, so it could allow testing different setups and configurations with QEMU. I would recommend to get the SMMU supporting supporting stage-2 page-tables. You mean in QEMU? Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing. Yes, that would be nice Cheers, Thank you, Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, On 10/30/20 6:08 PM, Bertrand Marquis wrote: > Hi Oleksandr, > >> On 30 Oct 2020, at 15:02, Oleksandr Andrushchenko >> wrote: >> >> Hi, >> >> On 10/30/20 4:47 PM, Rahul Singh wrote: >>> Hello Oleksandr, >>> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko wrote: Hi, Rahul! On 10/20/20 6:25 PM, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is based on > the Linux SMMUv3 driver. > > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux driver > that supports both Stage-1 and Stage-2 translations. First of all thank you for the efforts! I tried the patch with QEMU and would like to know if my understanding correct that this combination will not work as of now: (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >>> I have limited knowledge about QEMU internals.As what I see from the logs, >>> fault is occurred at early driver initialisation when SMMU driver is trying >>> to probe the HW. >>> (XEN) Data Abort Trap. Syndrome=0x1940010 (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 (XEN) 0TH[0x0] = 0xb8468f7f [snip] If this is expected then is there any plan to make QEMU work as well? I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side. >>> Yes as of now only Stage-2 is supported in XEN.If we have any requirement >>> or use case that depends on Stage-1 translation we can support that also in >>> XEN. >> The use case is below: PCI passthrough and various configurations including >> SR-IOV which is possible with QEMU... > This is currently not in the list of configuration supported or that we have > planned on our side to support. > > But we would be more then happy to review any changes to enable this :-) Fair enough Unfortunately we do not have any HW with SMMUv3 at the moment, so not sure we can invest time in this at the moment > > Regards > Bertrand Thank you, Oleksandr > We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough implementation, so it could allow testing different setups and configurations with QEMU. Thank you in advance, Oleksandr [1] https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.au...@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ [patchwork[.]ozlabs[.]org] >>> Regards, >>> Rahul >> Thank you, >> >> Oleksandr >
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Oleksandr, On 30/10/2020 10:44, Oleksandr Andrushchenko wrote: On 10/20/20 6:25 PM, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. First of all thank you for the efforts! I tried the patch with QEMU and would like to know if my understanding correct that this combination will not work as of now: (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) Data Abort Trap. Syndrome=0x1940010 (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 (XEN) 0TH[0x0] = 0xb8468f7f [snip] If this is expected then is there any plan to make QEMU work as well? I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side. Just for clarication, you are trying to boot Xen on QEMU, right? You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different. We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough implementation, so it could allow testing different setups and configurations with QEMU. I would recommend to get the SMMU supporting supporting stage-2 page-tables. Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing. Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Oleksandr, > On 30 Oct 2020, at 15:02, Oleksandr Andrushchenko > wrote: > > Hi, > > On 10/30/20 4:47 PM, Rahul Singh wrote: >> Hello Oleksandr, >> >>> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko >>> wrote: >>> >>> Hi, Rahul! >>> >>> On 10/20/20 6:25 PM, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. >>> First of all thank you for the efforts! >>> >>> I tried the patch with QEMU and would like to know if my understanding >>> correct >>> >>> that this combination will not work as of now: >>> >>> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq >> I have limited knowledge about QEMU internals.As what I see from the logs, >> fault is occurred at early driver initialisation when SMMU driver is trying >> to probe the HW. >> >>> (XEN) Data Abort Trap. Syndrome=0x1940010 >>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 >>> (XEN) 0TH[0x0] = 0xb8468f7f >>> >>> [snip] >>> >>> If this is expected then is there any plan to make QEMU work as well? >>> >>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU >>> side. >> Yes as of now only Stage-2 is supported in XEN.If we have any requirement or >> use case that depends on Stage-1 translation we can support that also in XEN. > The use case is below: PCI passthrough and various configurations including > SR-IOV which is possible with QEMU... This is currently not in the list of configuration supported or that we have planned on our side to support. But we would be more then happy to review any changes to enable this :-) Regards Bertrand >> >>> >>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough >>> >>> implementation, so it could allow testing different setups and >>> configurations with QEMU. >>> >>> >>> Thank you in advance, >>> >>> Oleksandr >>> >>> [1] >>> https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.au...@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ >>> [patchwork[.]ozlabs[.]org] >> Regards, >> Rahul > > Thank you, > > Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, On 10/30/20 4:47 PM, Rahul Singh wrote: > Hello Oleksandr, > >> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko >> wrote: >> >> Hi, Rahul! >> >> On 10/20/20 6:25 PM, Rahul Singh wrote: >>> Add support for ARM architected SMMUv3 implementations. It is based on >>> the Linux SMMUv3 driver. >>> >>> Major differences between the Linux driver are as follows: >>> 1. Only Stage-2 translation is supported as compared to the Linux driver >>> that supports both Stage-1 and Stage-2 translations. >> First of all thank you for the efforts! >> >> I tried the patch with QEMU and would like to know if my understanding >> correct >> >> that this combination will not work as of now: >> >> (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq > I have limited knowledge about QEMU internals.As what I see from the logs, > fault is occurred at early driver initialisation when SMMU driver is trying > to probe the HW. > >> (XEN) Data Abort Trap. Syndrome=0x1940010 >> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 >> (XEN) 0TH[0x0] = 0xb8468f7f >> >> [snip] >> >> If this is expected then is there any plan to make QEMU work as well? >> >> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU >> side. > Yes as of now only Stage-2 is supported in XEN.If we have any requirement or > use case that depends on Stage-1 translation we can support that also in XEN. The use case is below: PCI passthrough and various configurations including SR-IOV which is possible with QEMU... > >> >> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough >> >> implementation, so it could allow testing different setups and >> configurations with QEMU. >> >> >> Thank you in advance, >> >> Oleksandr >> >> [1] >> https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.au...@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ >> [patchwork[.]ozlabs[.]org] > Regards, > Rahul Thank you, Oleksandr
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Oleksandr, > On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko > wrote: > > Hi, Rahul! > > On 10/20/20 6:25 PM, Rahul Singh wrote: >> Add support for ARM architected SMMUv3 implementations. It is based on >> the Linux SMMUv3 driver. >> >> Major differences between the Linux driver are as follows: >> 1. Only Stage-2 translation is supported as compared to the Linux driver >>that supports both Stage-1 and Stage-2 translations. > > First of all thank you for the efforts! > > I tried the patch with QEMU and would like to know if my understanding correct > > that this combination will not work as of now: > > (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq I have limited knowledge about QEMU internals.As what I see from the logs, fault is occurred at early driver initialisation when SMMU driver is trying to probe the HW. > (XEN) Data Abort Trap. Syndrome=0x1940010 > (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 > (XEN) 0TH[0x0] = 0xb8468f7f > > [snip] > > If this is expected then is there any plan to make QEMU work as well? > > I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU > side. Yes as of now only Stage-2 is supported in XEN.If we have any requirement or use case that depends on Stage-1 translation we can support that also in XEN. > > > We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough > > implementation, so it could allow testing different setups and configurations > with QEMU. > > > Thank you in advance, > > Oleksandr > > [1] > https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.au...@redhat.com/ Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 30/10/2020 11:33, Rahul Singh wrote: Hello Julien, Hi, On 30 Oct 2020, at 10:05 am, Julien Grall wrote: On 30/10/2020 09:45, Rahul Singh wrote: Hello Julien, On 30 Oct 2020, at 9:21 am, Julien Grall wrote: Hi, On 30/10/2020 08:46, Rahul Singh wrote: Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also. I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread. The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen. However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9). So can you please clarify? Cheers, Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations. After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit. Ok. That was definitely not clear from the commit message. Please make this clearer in the commit message. Ok. I will make this clearer in the commit message. Anway, it means we need to do a full review of the code (rather than a light one) because of the hybrid model. I am still a bit puzzle to why it would require almost of a restart of the implementation in order to sync the latest code. Does it imply that you are mostly using the old code? SMMuv3 code is divided into below parts : 1. Low-level/High level queue manipulation functions. 2. Context descriptor manipulation functions. 3. Stream table manipulation functions. 4. Interrupt handling. 5. Linux IOMMU API functions. 6. Driver initialisation functions( probe/reset ). Low-level/High-level queue manipulation functions are from the old code, rest is the new code whenever it was possible. Thanks for the details! I think it would be useful to mention that in the commit message. I started with porting the latest code but there are many dependencies for the queue manipulation function so we decided to use the old queue manipulation function. In general, I would recommend to involve the community as soon as possible in the development process. This is quite important for big feature because it allows all the party to agree on the approach without investing significant effort. As the queue manipulation function is a big part of the code it will require a lot of effort and testing to sync with the latest code once the atomic operation is in place to use Sure, everything has a cost (including maintaining the code). This has to be a trade-off. My main concern was the maintenance of the code long term. However, as you and Bertrand stepped up for maintaining the code, then this should be less of a concern... Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 30 Oct 2020, at 10:05 am, Julien Grall wrote: > > > > On 30/10/2020 09:45, Rahul Singh wrote: >> Hello Julien, >>> On 30 Oct 2020, at 9:21 am, Julien Grall wrote: >>> >>> Hi, >>> >>> On 30/10/2020 08:46, Rahul Singh wrote: Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also. >>> >>> I would like to seek some clarifications on the code because there seem to >>> be conflicting information provided in this thread. >>> >>> The patch (the baseline commit is provided) and the discussion with >>> Bertrand suggests that you took a snapshot of the code last year and >>> adapted for Xen. >>> >>> However, here you suggest that you took an hybrid approach where part of >>> the code is based from last year and other part is based from the latest >>> code (I assume v5.9). >>> >>> So can you please clarify? >>> >>> Cheers, >> Approach I took is to first merge the code from the commit ( Jul 2, 2019 >> 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic >> operation is used in SMMUv3 code for command queue operations. >> After that I fixed the other code( not related to command queue >> operations.) from the latest code so that no bug is introduced in XEN >> because of using the last year commit. > > Ok. That was definitely not clear from the commit message. Please make this > clearer in the commit message. > Ok. I will make this clearer in the commit message. > Anway, it means we need to do a full review of the code (rather than a light > one) because of the hybrid model. > > I am still a bit puzzle to why it would require almost of a restart of the > implementation in order to sync the latest code. Does it imply that you are > mostly using the old code? > SMMuv3 code is divided into below parts : 1. Low-level/High level queue manipulation functions. 2. Context descriptor manipulation functions. 3. Stream table manipulation functions. 4. Interrupt handling. 5. Linux IOMMU API functions. 6. Driver initialisation functions( probe/reset ). Low-level/High-level queue manipulation functions are from the old code, rest is the new code whenever it was possible. I started with porting the latest code but there are many dependencies for the queue manipulation function so we decided to use the old queue manipulation function. As the queue manipulation function is a big part of the code it will require a lot of effort and testing to sync with the latest code once the atomic operation is in place to use. Once atomic operation is available in XEN we have merge the below commit from Linux to XEN to make XEN in sync with Linux code. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/iommu/arm-smmu-v3.c?h=v5.8=587e6c10a7ce89a5924fdbeff2ec524fbd6a124b > Cheers, > > -- > Julien Grall Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, Rahul! On 10/20/20 6:25 PM, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is based on > the Linux SMMUv3 driver. > > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux driver > that supports both Stage-1 and Stage-2 translations. First of all thank you for the efforts! I tried the patch with QEMU and would like to know if my understanding correct that this combination will not work as of now: (XEN) SMMUv3: /smmuv3@905: SMMUv3: DT value = eventq (XEN) Data Abort Trap. Syndrome=0x1940010 (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0xb8469000 (XEN) 0TH[0x0] = 0xb8468f7f [snip] If this is expected then is there any plan to make QEMU work as well? I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side. We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough implementation, so it could allow testing different setups and configurations with QEMU. Thank you in advance, Oleksandr [1] https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.au...@redhat.com/
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 30/10/2020 09:45, Rahul Singh wrote: Hello Julien, On 30 Oct 2020, at 9:21 am, Julien Grall wrote: Hi, On 30/10/2020 08:46, Rahul Singh wrote: Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also. I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread. The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen. However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9). So can you please clarify? Cheers, Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations. After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit. Ok. That was definitely not clear from the commit message. Please make this clearer in the commit message. Anway, it means we need to do a full review of the code (rather than a light one) because of the hybrid model. I am still a bit puzzle to why it would require almost of a restart of the implementation in order to sync the latest code. Does it imply that you are mostly using the old code? Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 30 Oct 2020, at 9:21 am, Julien Grall wrote: > > Hi, > > On 30/10/2020 08:46, Rahul Singh wrote: >> Ok Yes when I ported the driver I port the command queue operation from the >> previous commit where atomic operations is not used and rest all the code is >> from the latest code. I will again make sure that any bug that is fixed in >> Linux should be fixed in XEN also. > > I would like to seek some clarifications on the code because there seem to be > conflicting information provided in this thread. > > The patch (the baseline commit is provided) and the discussion with Bertrand > suggests that you took a snapshot of the code last year and adapted for Xen. > > However, here you suggest that you took an hybrid approach where part of the > code is based from last year and other part is based from the latest code (I > assume v5.9). > > So can you please clarify? > > Cheers, Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations. After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit. > > -- > Julien Grall Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, On 30/10/2020 08:46, Rahul Singh wrote: Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also. I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread. The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen. However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9). So can you please clarify? Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Stefano, > On 29 Oct 2020, at 8:17 pm, Stefano Stabellini wrote: > > On Thu, 29 Oct 2020, Bertrand Marquis wrote: >>> On 28 Oct 2020, at 19:12, Julien Grall wrote: >>> On 26/10/2020 11:03, Rahul Singh wrote: Hello Julien, > On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: > On 23/10/2020 15:27, Rahul Singh wrote: >> Hello Julien, >>> On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: >>> On 23/10/2020 12:35, Rahul Singh wrote: Hello, > On 23 Oct 2020, at 1:02 am, Stefano Stabellini > wrote: > > On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is > based on > the Linux SMMUv3 driver. > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux > driver > that supports both Stage-1 and Stage-2 translations. > 2. Use P2M page table instead of creating one as SMMUv3 has the > capability to share the page tables with the CPU. > 3. Tasklets is used in place of threaded IRQ's in Linux for event > queue > and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? >>> >>> Yes you are right because of long running operations Linux is using >>> the >>> threaded IRQs. >>> >>> SMMUv3 reports fault/events bases on memory-based circular buffer >>> queues not >>> based on the register. As per my understanding, it is >>> time-consuming to >>> process the memory based queues in interrupt context because of >>> that Linux >>> is using threaded IRQ to process the faults/events from SMMU. >>> >>> I didn’t find any other solution in XEN in place of tasklet to >>> defer the >>> work, that’s why I used tasklet in XEN in replacement of threaded >>> IRQs. If >>> we do all work in interrupt context we will make XEN less >>> responsive. >> >> So we need to make sure that Xen continue to receives interrupts, >> but we also >> need to make sure that a vCPU bound to the pCPU is also responsive. >> >>> >>> If you know another solution in XEN that will be used to defer the >>> work in >>> the interrupt please let me know I will try to use that. >> >> One of my work colleague encountered a similar problem recently. He >> had a long >> running tasklet and wanted to be broken down in smaller chunk. >> >> We decided to use a timer to reschedule the taslket in the future. >> This allows >> the scheduler to run other loads (e.g. vCPU) for some time. >> >> This is pretty hackish but I couldn't find a better solution as >> tasklet have >> high priority. >> >> Maybe the other will have a better idea. > > Julien's suggestion is a good one. > > But I think tasklets can be configured to be called from the > idle_loop, > in which case they are not run in interrupt context? > Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. >>> >>> This depends on your tasklet. Some will run from the softirq context >>> which is usually (for Arm) on the return of an exception. >>> >> Thanks for the info. I will check and will get better understanding of >> the tasklet how it will run in XEN. > > 4. Latest version of the Linux SMMUv3 code implements the > commands queue > access functions based on atomic operations implemented in > Linux. Can you provide more details? >>> >>> I tried to port the latest version of the SMMUv3 code than I >>> observed that >>> in order to port that code I have to also port atomic operation >>> implemented >>> in Linux to XEN. As latest Linux code uses atomic operation to >>> process the >>> command queues >>> (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , >>> atomic_fetch_andnot_relaxed()) . >> >> Thank you for the explanation. I think it would be best to import >> the atomic >>
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
HI Stefano, > On 29 Oct 2020, at 20:17, Stefano Stabellini wrote: > > On Thu, 29 Oct 2020, Bertrand Marquis wrote: >>> On 28 Oct 2020, at 19:12, Julien Grall wrote: >>> On 26/10/2020 11:03, Rahul Singh wrote: Hello Julien, > On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: > On 23/10/2020 15:27, Rahul Singh wrote: >> Hello Julien, >>> On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: >>> On 23/10/2020 12:35, Rahul Singh wrote: Hello, > On 23 Oct 2020, at 1:02 am, Stefano Stabellini > wrote: > > On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is > based on > the Linux SMMUv3 driver. > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux > driver > that supports both Stage-1 and Stage-2 translations. > 2. Use P2M page table instead of creating one as SMMUv3 has the > capability to share the page tables with the CPU. > 3. Tasklets is used in place of threaded IRQ's in Linux for event > queue > and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? >>> >>> Yes you are right because of long running operations Linux is using >>> the >>> threaded IRQs. >>> >>> SMMUv3 reports fault/events bases on memory-based circular buffer >>> queues not >>> based on the register. As per my understanding, it is >>> time-consuming to >>> process the memory based queues in interrupt context because of >>> that Linux >>> is using threaded IRQ to process the faults/events from SMMU. >>> >>> I didn’t find any other solution in XEN in place of tasklet to >>> defer the >>> work, that’s why I used tasklet in XEN in replacement of threaded >>> IRQs. If >>> we do all work in interrupt context we will make XEN less >>> responsive. >> >> So we need to make sure that Xen continue to receives interrupts, >> but we also >> need to make sure that a vCPU bound to the pCPU is also responsive. >> >>> >>> If you know another solution in XEN that will be used to defer the >>> work in >>> the interrupt please let me know I will try to use that. >> >> One of my work colleague encountered a similar problem recently. He >> had a long >> running tasklet and wanted to be broken down in smaller chunk. >> >> We decided to use a timer to reschedule the taslket in the future. >> This allows >> the scheduler to run other loads (e.g. vCPU) for some time. >> >> This is pretty hackish but I couldn't find a better solution as >> tasklet have >> high priority. >> >> Maybe the other will have a better idea. > > Julien's suggestion is a good one. > > But I think tasklets can be configured to be called from the > idle_loop, > in which case they are not run in interrupt context? > Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. >>> >>> This depends on your tasklet. Some will run from the softirq context >>> which is usually (for Arm) on the return of an exception. >>> >> Thanks for the info. I will check and will get better understanding of >> the tasklet how it will run in XEN. > > 4. Latest version of the Linux SMMUv3 code implements the > commands queue > access functions based on atomic operations implemented in > Linux. Can you provide more details? >>> >>> I tried to port the latest version of the SMMUv3 code than I >>> observed that >>> in order to port that code I have to also port atomic operation >>> implemented >>> in Linux to XEN. As latest Linux code uses atomic operation to >>> process the >>> command queues >>> (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , >>> atomic_fetch_andnot_relaxed()) . >> >> Thank you for the explanation. I think it would be best to import >> the atomic >>
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On Thu, 29 Oct 2020, Bertrand Marquis wrote: > > On 28 Oct 2020, at 19:12, Julien Grall wrote: > > On 26/10/2020 11:03, Rahul Singh wrote: > >> Hello Julien, > >>> On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: > >>> On 23/10/2020 15:27, Rahul Singh wrote: > Hello Julien, > > On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: > > On 23/10/2020 12:35, Rahul Singh wrote: > >> Hello, > >>> On 23 Oct 2020, at 1:02 am, Stefano Stabellini > >>> wrote: > >>> > >>> On Thu, 22 Oct 2020, Julien Grall wrote: > >> On 20/10/2020 16:25, Rahul Singh wrote: > >>> Add support for ARM architected SMMUv3 implementations. It is > >>> based on > >>> the Linux SMMUv3 driver. > >>> Major differences between the Linux driver are as follows: > >>> 1. Only Stage-2 translation is supported as compared to the Linux > >>> driver > >>>that supports both Stage-1 and Stage-2 translations. > >>> 2. Use P2M page table instead of creating one as SMMUv3 has the > >>>capability to share the page tables with the CPU. > >>> 3. Tasklets is used in place of threaded IRQ's in Linux for event > >>> queue > >>>and priority queue IRQ handling. > >> > >> Tasklets are not a replacement for threaded IRQ. In particular, > >> they will > >> have priority over anything else (IOW nothing will run on the pCPU > >> until > >> they are done). > >> > >> Do you know why Linux is using thread. Is it because of long > >> running > >> operations? > > > > Yes you are right because of long running operations Linux is using > > the > > threaded IRQs. > > > > SMMUv3 reports fault/events bases on memory-based circular buffer > > queues not > > based on the register. As per my understanding, it is > > time-consuming to > > process the memory based queues in interrupt context because of > > that Linux > > is using threaded IRQ to process the faults/events from SMMU. > > > > I didn’t find any other solution in XEN in place of tasklet to > > defer the > > work, that’s why I used tasklet in XEN in replacement of threaded > > IRQs. If > > we do all work in interrupt context we will make XEN less > > responsive. > > So we need to make sure that Xen continue to receives interrupts, > but we also > need to make sure that a vCPU bound to the pCPU is also responsive. > > > > > If you know another solution in XEN that will be used to defer the > > work in > > the interrupt please let me know I will try to use that. > > One of my work colleague encountered a similar problem recently. He > had a long > running tasklet and wanted to be broken down in smaller chunk. > > We decided to use a timer to reschedule the taslket in the future. > This allows > the scheduler to run other loads (e.g. vCPU) for some time. > > This is pretty hackish but I couldn't find a better solution as > tasklet have > high priority. > > Maybe the other will have a better idea. > >>> > >>> Julien's suggestion is a good one. > >>> > >>> But I think tasklets can be configured to be called from the > >>> idle_loop, > >>> in which case they are not run in interrupt context? > >>> > >> Yes you are right tasklet will be scheduled from the idle_loop that > >> is not interrupt conext. > > > > This depends on your tasklet. Some will run from the softirq context > > which is usually (for Arm) on the return of an exception. > > > Thanks for the info. I will check and will get better understanding of > the tasklet how it will run in XEN. > >>> > >>> 4. Latest version of the Linux SMMUv3 code implements the > >>> commands queue > >>>access functions based on atomic operations implemented in > >>> Linux. > >> > >> Can you provide more details? > > > > I tried to port the latest version of the SMMUv3 code than I > > observed that > > in order to port that code I have to also port atomic operation > > implemented > > in Linux to XEN. As latest Linux code uses atomic operation to > > process the > > command queues > > (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , > > atomic_fetch_andnot_relaxed()) . > > Thank you for the explanation. I think it would be best to import > the atomic > helpers and use the latest code. > > This will ensure
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Julien, > On 28 Oct 2020, at 19:12, Julien Grall wrote: > > > > On 26/10/2020 11:03, Rahul Singh wrote: >> Hello Julien, > > Hi Rahul, > >>> On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: >>> >>> >>> >>> On 23/10/2020 15:27, Rahul Singh wrote: Hello Julien, > On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: > > > > On 23/10/2020 12:35, Rahul Singh wrote: >> Hello, >>> On 23 Oct 2020, at 1:02 am, Stefano Stabellini >>> wrote: >>> >>> On Thu, 22 Oct 2020, Julien Grall wrote: >> On 20/10/2020 16:25, Rahul Singh wrote: >>> Add support for ARM architected SMMUv3 implementations. It is based >>> on >>> the Linux SMMUv3 driver. >>> Major differences between the Linux driver are as follows: >>> 1. Only Stage-2 translation is supported as compared to the Linux >>> driver >>>that supports both Stage-1 and Stage-2 translations. >>> 2. Use P2M page table instead of creating one as SMMUv3 has the >>>capability to share the page tables with the CPU. >>> 3. Tasklets is used in place of threaded IRQ's in Linux for event >>> queue >>>and priority queue IRQ handling. >> >> Tasklets are not a replacement for threaded IRQ. In particular, they >> will >> have priority over anything else (IOW nothing will run on the pCPU >> until >> they are done). >> >> Do you know why Linux is using thread. Is it because of long running >> operations? > > Yes you are right because of long running operations Linux is using > the > threaded IRQs. > > SMMUv3 reports fault/events bases on memory-based circular buffer > queues not > based on the register. As per my understanding, it is time-consuming > to > process the memory based queues in interrupt context because of that > Linux > is using threaded IRQ to process the faults/events from SMMU. > > I didn’t find any other solution in XEN in place of tasklet to defer > the > work, that’s why I used tasklet in XEN in replacement of threaded > IRQs. If > we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. > > If you know another solution in XEN that will be used to defer the > work in > the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. >>> >>> Julien's suggestion is a good one. >>> >>> But I think tasklets can be configured to be called from the idle_loop, >>> in which case they are not run in interrupt context? >>> >> Yes you are right tasklet will be scheduled from the idle_loop that is >> not interrupt conext. > > This depends on your tasklet. Some will run from the softirq context > which is usually (for Arm) on the return of an exception. > Thanks for the info. I will check and will get better understanding of the tasklet how it will run in XEN. >>> >>> 4. Latest version of the Linux SMMUv3 code implements the commands >>> queue >>>access functions based on atomic operations implemented in Linux. >> >> Can you provide more details? > > I tried to port the latest version of the SMMUv3 code than I observed > that > in order to port that code I have to also port atomic operation > implemented > in Linux to XEN. As latest Linux code uses atomic operation to > process the > command queues > (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , > atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? >>> >>> I think you are right. >> Yes, I agree with
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 26/10/2020 11:03, Rahul Singh wrote: Hello Julien, Hi Rahul, On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: On 23/10/2020 15:27, Rahul Singh wrote: Hello Julien, On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: On 23/10/2020 12:35, Rahul Singh wrote: Hello, On 23 Oct 2020, at 1:02 am, Stefano Stabellini wrote: On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. Julien's suggestion is a good one. But I think tasklets can be configured to be called from the idle_loop, in which case they are not run in interrupt context? Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. This depends on your tasklet. Some will run from the softirq context which is usually (for Arm) on the return of an exception. Thanks for the info. I will check and will get better understanding of the tasklet how it will run in XEN. 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. Can you provide more details? I tried to port the latest version of the SMMUv3 code than I observed that in order to port that code I have to also port atomic operation implemented in Linux to XEN. As latest Linux code uses atomic operation to process the command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? I think you are right. Yes, I agree with you to have XEN code in sync with Linux code that's why I started with to port the Linux atomic operations to XEN then I realised that it is not straightforward to port atomic operations and it requires lots of effort and testing. Therefore I decided to port the code before the atomic operation is introduced in Linux. Hmmm... I would not have expected a lot of effort required to add the 3 atomics operations above. Are you trying to also port the LSE support at the same time? There are other atomic operations used in the SMMUv3 code apart from the 3 atomic operation I mention. I just mention 3 operation as an example. Ok. Do you have a list you could share? Yes. Please find the list that we have to port to the XEN in order to merge the latest SMMUv3 code. Thanks! If we start to port the below list we might have to port another atomic operation based on which below atomic operations are implemented. I have not spent time on how these atomic operations are implemented in detail but as per my understanding, it required an effort to port them to XEN and required a lot of testing. For the beginning, I think it is fine to implement them with a
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Julien, > On 27 Oct 2020, at 14:37, Julien Grall wrote: > > > > On 27/10/2020 14:19, Bertrand Marquis wrote: >> Hi Julien, > > Hi Bertrand, > >>> On 26 Oct 2020, at 19:05, Julien Grall wrote: >>> >>> On 26/10/2020 12:10, Ash Wilding wrote: Hi, >>> >>> Hi Ash, >>> > 1. atomic_set_release > 2. atomic_fetch_andnot_relaxed > 3. atomic_cond_read_relaxed > 4. atomic_long_cond_read_relaxed > 5. atomic_long_xor > 6. atomic_set_release > 7. atomic_cmpxchg_relaxed might be we can use atomic_cmpxchg that is >implemented in XEN need to check. > 8. atomic_dec_return_release > 9. atomic_fetch_inc_relaxed If we're going to pull in Linux's implementations of the above atomics helpers for SMMUv3, and given the majority of SMMUv3 systems are v8.1+ with LSE, perhaps this would be a good time to drop the current atomic.h in Xen completely and pull in both Linux's LL/SC and LSE helpers, >>> >>> When I originally answered to the thread, I thought about suggesting >>> importing LSE. But I felt it was too much to ask in order to merge the >>> SMMUv3 code. >>> >>> However, I would love to have support for LSE in Xen as this would solve >>> other not yet fully closed security issue with LL/SC (see XSA-295 [2]). >>> >>> Would Arm be willing to add support for LSE before merging the SMMUv3? >> We are willing to work on LSE but I cannot commit on me and my team to start >> working on this subject before the end of the year. > > That's fine. There are other way we can bridge the gap between Xen and Linux > in order to get the latest version (see more below). > >> So I think it would be good to integrate this version of SMMUv3 and we can >> then update it to the latest Linux one once LSE has been added. > > As I pointed out in my first e-mail on this thread, I am quite concerned that > we are going to (re-)introduce bugs that have been fixed in Linux. > Did you investigate that this is not going to happen? We will take the action to check changes in Linux in the driver since the version we are based on to make sure no critical fixes have been done that are needed in our code. > > However, I think we can manage to get the latest version without requiring > LSE. It should be possible to provide dumb version of most the helpers. If this is done, there would still be after a big work on switching to the newer code from Linux, testing and reviewing it. Updating the driver after those dumb versions are added would still be possible. Cheers Bertrand > >> I think it make sense in the meantime to discuss how this should be designed >> but it might be a good idea to make a specific discussion thread for that. > > Make sense. Can you start a new thread? > > Cheers, > > -- > Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 27/10/2020 14:19, Bertrand Marquis wrote: Hi Julien, Hi Bertrand, On 26 Oct 2020, at 19:05, Julien Grall wrote: On 26/10/2020 12:10, Ash Wilding wrote: Hi, Hi Ash, 1. atomic_set_release 2. atomic_fetch_andnot_relaxed 3. atomic_cond_read_relaxed 4. atomic_long_cond_read_relaxed 5. atomic_long_xor 6. atomic_set_release 7. atomic_cmpxchg_relaxed might be we can use atomic_cmpxchg that is implemented in XEN need to check. 8. atomic_dec_return_release 9. atomic_fetch_inc_relaxed If we're going to pull in Linux's implementations of the above atomics helpers for SMMUv3, and given the majority of SMMUv3 systems are v8.1+ with LSE, perhaps this would be a good time to drop the current atomic.h in Xen completely and pull in both Linux's LL/SC and LSE helpers, When I originally answered to the thread, I thought about suggesting importing LSE. But I felt it was too much to ask in order to merge the SMMUv3 code. However, I would love to have support for LSE in Xen as this would solve other not yet fully closed security issue with LL/SC (see XSA-295 [2]). Would Arm be willing to add support for LSE before merging the SMMUv3? We are willing to work on LSE but I cannot commit on me and my team to start working on this subject before the end of the year. That's fine. There are other way we can bridge the gap between Xen and Linux in order to get the latest version (see more below). So I think it would be good to integrate this version of SMMUv3 and we can then update it to the latest Linux one once LSE has been added. As I pointed out in my first e-mail on this thread, I am quite concerned that we are going to (re-)introduce bugs that have been fixed in Linux. Did you investigate that this is not going to happen? However, I think we can manage to get the latest version without requiring LSE. It should be possible to provide dumb version of most the helpers. I think it make sense in the meantime to discuss how this should be designed but it might be a good idea to make a specific discussion thread for that. Make sense. Can you start a new thread? Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Julien, > On 26 Oct 2020, at 19:05, Julien Grall wrote: > > On 26/10/2020 12:10, Ash Wilding wrote: >> Hi, > > Hi Ash, > >>> 1. atomic_set_release >>> 2. atomic_fetch_andnot_relaxed >>> 3. atomic_cond_read_relaxed >>> 4. atomic_long_cond_read_relaxed >>> 5. atomic_long_xor >>> 6. atomic_set_release >>> 7. atomic_cmpxchg_relaxed might be we can use atomic_cmpxchg that is >>>implemented in XEN need to check. >>> 8. atomic_dec_return_release >>> 9. atomic_fetch_inc_relaxed >> If we're going to pull in Linux's implementations of the above atomics >> helpers for SMMUv3, and given the majority of SMMUv3 systems are v8.1+ >> with LSE, perhaps this would be a good time to drop the current >> atomic.h in Xen completely and pull in both Linux's LL/SC and LSE >> helpers, > > When I originally answered to the thread, I thought about suggesting > importing LSE. But I felt it was too much to ask in order to merge the SMMUv3 > code. > > However, I would love to have support for LSE in Xen as this would solve > other not yet fully closed security issue with LL/SC (see XSA-295 [2]). > > Would Arm be willing to add support for LSE before merging the SMMUv3? We are willing to work on LSE but I cannot commit on me and my team to start working on this subject before the end of the year. So I think it would be good to integrate this version of SMMUv3 and we can then update it to the latest Linux one once LSE has been added. I think it make sense in the meantime to discuss how this should be designed but it might be a good idea to make a specific discussion thread for that. Cheers Bertrand > > As an alternative, it might also be possible to provide "dumb" implementation > for all the helpers even if they are stricter than necessary for the memory > ordering requirements. > > then use a new Kconfig to toggle between them? > > I would prefer to follow the same approach as Linux and allow Xen to select > at boot time which implementations to use. This would enable distro to > provide a single binary that boot on all Armv8 and still allow Xen to select > the best set of instructions. > > Xen already provides a framework to switch between two sets of instructions > at boot. This was borrowed from Linux, so I don't expect a big hurdle to get > this supported. > >> Back in 5d45ecabe3 [1] Jan mentioned we probably want to avoid relying >> on gcc atomics helpers as we can't switch between LL/SC and LSE >> atomics. > > I asked Jan to add this line in the commit message :). My concern was that > even if we provided a runtime switch (or sanity check for XSA-295), the GCC > helpers would not be able to take advantage (the code is not written by Xen > community). > > Cheers, > >> [1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5d45ecabe3 > > [2] https://xenbits.xen.org/xsa/advisory-295.html > > > -- > Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Julien, > Would Arm be willing to add support for LSE before merging the > SMMUv3? (( Taking my Arm hat off for a second and speaking independently... )) I've been toying with doing this in my own personal time but unsure how long it would take (unable to commit much time on it right now). I'll let the Arm folks speak for themselves as to whether they're able and willing do it. If not, I don't necessarily think pulling in Linux's LL/SC and LSE atomics helpers should block merging the SMMUv3 driver if everything else is OK after review; we could use Rahul's versions in the driver for now and then merge Linux's helpers later. > I would prefer to follow the same approach as Linux and allow Xen to > select at boot time which implementations to use. This would enable > distro to provide a single binary that boot on all Armv8 and still > allow Xen to select the best set of instructions. Yep good idea, agreed. Note that while Linux uses the alternatives framework for LL/SC vs LSE, it's still controlled by CONFIG_ARM64_LSE_ATOMICS, see [3]. This would give us the best of both worlds - distros can build with the Kconfig =y to enable single image with runtime detection, while expert users can still build a custom image to force use of LL/SC on v8.1+ systems should they wish. > I asked Jan to add this line in the commit message :). My concern was > that even if we provided a runtime switch (or sanity check for > XSA-295), the GCC helpers would not be able to take advantage (the > code is not written by Xen community). Ahh yes, makes sense - all the more reason for us to get explicit implementations into Xen sooner rather than later :-) Thanks, Ash. > [1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5d45ecabe3 > [2] https://xenbits.xen.org/xsa/advisory-295.html [3] https://elixir.bootlin.com/linux/latest/source/arch/arm64/include/asm/lse.h#L7
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 26/10/2020 12:10, Ash Wilding wrote: Hi, Hi Ash, 1. atomic_set_release 2. atomic_fetch_andnot_relaxed 3. atomic_cond_read_relaxed 4. atomic_long_cond_read_relaxed 5. atomic_long_xor 6. atomic_set_release 7. atomic_cmpxchg_relaxed might be we can use atomic_cmpxchg that is implemented in XEN need to check. 8. atomic_dec_return_release 9. atomic_fetch_inc_relaxed If we're going to pull in Linux's implementations of the above atomics helpers for SMMUv3, and given the majority of SMMUv3 systems are v8.1+ with LSE, perhaps this would be a good time to drop the current atomic.h in Xen completely and pull in both Linux's LL/SC and LSE helpers, When I originally answered to the thread, I thought about suggesting importing LSE. But I felt it was too much to ask in order to merge the SMMUv3 code. However, I would love to have support for LSE in Xen as this would solve other not yet fully closed security issue with LL/SC (see XSA-295 [2]). Would Arm be willing to add support for LSE before merging the SMMUv3? As an alternative, it might also be possible to provide "dumb" implementation for all the helpers even if they are stricter than necessary for the memory ordering requirements. then use a new Kconfig to toggle between them? I would prefer to follow the same approach as Linux and allow Xen to select at boot time which implementations to use. This would enable distro to provide a single binary that boot on all Armv8 and still allow Xen to select the best set of instructions. Xen already provides a framework to switch between two sets of instructions at boot. This was borrowed from Linux, so I don't expect a big hurdle to get this supported. Back in 5d45ecabe3 [1] Jan mentioned we probably want to avoid relying on gcc atomics helpers as we can't switch between LL/SC and LSE atomics. I asked Jan to add this line in the commit message :). My concern was that even if we provided a runtime switch (or sanity check for XSA-295), the GCC helpers would not be able to take advantage (the code is not written by Xen community). Cheers, [1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5d45ecabe3 [2] https://xenbits.xen.org/xsa/advisory-295.html -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi, > 1. atomic_set_release > 2. atomic_fetch_andnot_relaxed > 3. atomic_cond_read_relaxed > 4. atomic_long_cond_read_relaxed > 5. atomic_long_xor > 6. atomic_set_release > 7. atomic_cmpxchg_relaxed might be we can use atomic_cmpxchg that is >implemented in XEN need to check. > 8. atomic_dec_return_release > 9. atomic_fetch_inc_relaxed If we're going to pull in Linux's implementations of the above atomics helpers for SMMUv3, and given the majority of SMMUv3 systems are v8.1+ with LSE, perhaps this would be a good time to drop the current atomic.h in Xen completely and pull in both Linux's LL/SC and LSE helpers, then use a new Kconfig to toggle between them? Back in 5d45ecabe3 [1] Jan mentioned we probably want to avoid relying on gcc atomics helpers as we can't switch between LL/SC and LSE atomics. With the above we'd be able to drop the reference to gcc's built-in __sync_fetch_and_add() in xen/include/asm-arm/system.h by making arch_fetch_and_add() pull in the explicit implementation of the helper. Thoughts? Thanks, Ash. [1] https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=5d45ecabe3
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 23 Oct 2020, at 4:19 pm, Julien Grall wrote: > > > > On 23/10/2020 15:27, Rahul Singh wrote: >> Hello Julien, >>> On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: >>> >>> >>> >>> On 23/10/2020 12:35, Rahul Singh wrote: Hello, > On 23 Oct 2020, at 1:02 am, Stefano Stabellini > wrote: > > On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is based on > the Linux SMMUv3 driver. > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux > driver >that supports both Stage-1 and Stage-2 translations. > 2. Use P2M page table instead of creating one as SMMUv3 has the >capability to share the page tables with the CPU. > 3. Tasklets is used in place of threaded IRQ's in Linux for event > queue >and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? >>> >>> Yes you are right because of long running operations Linux is using the >>> threaded IRQs. >>> >>> SMMUv3 reports fault/events bases on memory-based circular buffer >>> queues not >>> based on the register. As per my understanding, it is time-consuming to >>> process the memory based queues in interrupt context because of that >>> Linux >>> is using threaded IRQ to process the faults/events from SMMU. >>> >>> I didn’t find any other solution in XEN in place of tasklet to defer the >>> work, that’s why I used tasklet in XEN in replacement of threaded IRQs. >>> If >>> we do all work in interrupt context we will make XEN less responsive. >> >> So we need to make sure that Xen continue to receives interrupts, but we >> also >> need to make sure that a vCPU bound to the pCPU is also responsive. >> >>> >>> If you know another solution in XEN that will be used to defer the work >>> in >>> the interrupt please let me know I will try to use that. >> >> One of my work colleague encountered a similar problem recently. He had >> a long >> running tasklet and wanted to be broken down in smaller chunk. >> >> We decided to use a timer to reschedule the taslket in the future. This >> allows >> the scheduler to run other loads (e.g. vCPU) for some time. >> >> This is pretty hackish but I couldn't find a better solution as tasklet >> have >> high priority. >> >> Maybe the other will have a better idea. > > Julien's suggestion is a good one. > > But I think tasklets can be configured to be called from the idle_loop, > in which case they are not run in interrupt context? > Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. >>> >>> This depends on your tasklet. Some will run from the softirq context which >>> is usually (for Arm) on the return of an exception. >>> >> Thanks for the info. I will check and will get better understanding of the >> tasklet how it will run in XEN. > > 4. Latest version of the Linux SMMUv3 code implements the commands > queue >access functions based on atomic operations implemented in Linux. Can you provide more details? >>> >>> I tried to port the latest version of the SMMUv3 code than I observed >>> that >>> in order to port that code I have to also port atomic operation >>> implemented >>> in Linux to XEN. As latest Linux code uses atomic operation to process >>> the >>> command queues >>> (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , >>> atomic_fetch_andnot_relaxed()) . >> >> Thank you for the explanation. I think it would be best to import the >> atomic >> helpers and use the latest code. >> >> This will ensure that we don't re-introduce bugs and also buy us some >> time >> before the Linux and Xen driver diverge again too much. >> >> Stefano, what do you think? > > I think you are right. Yes, I agree with you to have XEN code in sync with Linux code that's why I started with to port the Linux atomic operations to XEN then I realised that it is not straightforward to port atomic operations and it requires lots of effort and testing. Therefore I decided to port the code before the atomic operation is introduced in Linux. >>> >>> Hmmm... I would not have expected a lot of effort required to
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On Fri, 23 Oct 2020, Julien Grall wrote: > On 23/10/2020 01:02, Stefano Stabellini wrote: > > On Thu, 22 Oct 2020, Julien Grall wrote: > > > > > On 20/10/2020 16:25, Rahul Singh wrote: > > > > > > Add support for ARM architected SMMUv3 implementations. It is based > > > > > > on > > > > > > the Linux SMMUv3 driver. > > > > > > Major differences between the Linux driver are as follows: > > > > > > 1. Only Stage-2 translation is supported as compared to the Linux > > > > > > driver > > > > > > that supports both Stage-1 and Stage-2 translations. > > > > > > 2. Use P2M page table instead of creating one as SMMUv3 has the > > > > > > capability to share the page tables with the CPU. > > > > > > 3. Tasklets is used in place of threaded IRQ's in Linux for event > > > > > > queue > > > > > > and priority queue IRQ handling. > > > > > > > > > > Tasklets are not a replacement for threaded IRQ. In particular, they > > > > > will > > > > > have priority over anything else (IOW nothing will run on the pCPU > > > > > until > > > > > they are done). > > > > > > > > > > Do you know why Linux is using thread. Is it because of long running > > > > > operations? > > > > > > > > Yes you are right because of long running operations Linux is using the > > > > threaded IRQs. > > > > > > > > SMMUv3 reports fault/events bases on memory-based circular buffer queues > > > > not > > > > based on the register. As per my understanding, it is time-consuming to > > > > process the memory based queues in interrupt context because of that > > > > Linux > > > > is using threaded IRQ to process the faults/events from SMMU. > > > > > > > > I didn’t find any other solution in XEN in place of tasklet to defer the > > > > work, that’s why I used tasklet in XEN in replacement of threaded IRQs. > > > > If > > > > we do all work in interrupt context we will make XEN less responsive. > > > > > > So we need to make sure that Xen continue to receives interrupts, but we > > > also > > > need to make sure that a vCPU bound to the pCPU is also responsive. > > > > > > > > > > > If you know another solution in XEN that will be used to defer the work > > > > in > > > > the interrupt please let me know I will try to use that. > > > > > > One of my work colleague encountered a similar problem recently. He had a > > > long > > > running tasklet and wanted to be broken down in smaller chunk. > > > > > > We decided to use a timer to reschedule the taslket in the future. This > > > allows > > > the scheduler to run other loads (e.g. vCPU) for some time. > > > > > > This is pretty hackish but I couldn't find a better solution as tasklet > > > have > > > high priority. > > > > > > Maybe the other will have a better idea. > > > > Julien's suggestion is a good one. > > > > But I think tasklets can be configured to be called from the idle_loop, > > in which case they are not run in interrupt context? > > Tasklets can either run from the IDLE loop or from a softirq context. > > When running from a softirq context is may happen on return from receiving an > interrupt. However, interrupts will always be enabled. > > So I am not sure what concern you are trying to raise here. Not raising any concerns :-) I thought one of the previous statements in this thread implied that tasklets are run in interrupt context -- I just wanted to go into details on that point as it is relevant.
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 23/10/2020 15:27, Rahul Singh wrote: Hello Julien, On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: On 23/10/2020 12:35, Rahul Singh wrote: Hello, On 23 Oct 2020, at 1:02 am, Stefano Stabellini wrote: On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. Julien's suggestion is a good one. But I think tasklets can be configured to be called from the idle_loop, in which case they are not run in interrupt context? Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. This depends on your tasklet. Some will run from the softirq context which is usually (for Arm) on the return of an exception. Thanks for the info. I will check and will get better understanding of the tasklet how it will run in XEN. 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. Can you provide more details? I tried to port the latest version of the SMMUv3 code than I observed that in order to port that code I have to also port atomic operation implemented in Linux to XEN. As latest Linux code uses atomic operation to process the command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? I think you are right. Yes, I agree with you to have XEN code in sync with Linux code that's why I started with to port the Linux atomic operations to XEN then I realised that it is not straightforward to port atomic operations and it requires lots of effort and testing. Therefore I decided to port the code before the atomic operation is introduced in Linux. Hmmm... I would not have expected a lot of effort required to add the 3 atomics operations above. Are you trying to also port the LSE support at the same time? There are other atomic operations used in the SMMUv3 code apart from the 3 atomic operation I mention. I just mention 3 operation as an example. Ok. Do you have a list you could share? Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 23 Oct 2020, at 2:00 pm, Julien Grall wrote: > > > > On 23/10/2020 12:35, Rahul Singh wrote: >> Hello, >>> On 23 Oct 2020, at 1:02 am, Stefano Stabellini >>> wrote: >>> >>> On Thu, 22 Oct 2020, Julien Grall wrote: >> On 20/10/2020 16:25, Rahul Singh wrote: >>> Add support for ARM architected SMMUv3 implementations. It is based on >>> the Linux SMMUv3 driver. >>> Major differences between the Linux driver are as follows: >>> 1. Only Stage-2 translation is supported as compared to the Linux driver >>>that supports both Stage-1 and Stage-2 translations. >>> 2. Use P2M page table instead of creating one as SMMUv3 has the >>>capability to share the page tables with the CPU. >>> 3. Tasklets is used in place of threaded IRQ's in Linux for event queue >>>and priority queue IRQ handling. >> >> Tasklets are not a replacement for threaded IRQ. In particular, they will >> have priority over anything else (IOW nothing will run on the pCPU until >> they are done). >> >> Do you know why Linux is using thread. Is it because of long running >> operations? > > Yes you are right because of long running operations Linux is using the > threaded IRQs. > > SMMUv3 reports fault/events bases on memory-based circular buffer queues > not > based on the register. As per my understanding, it is time-consuming to > process the memory based queues in interrupt context because of that Linux > is using threaded IRQ to process the faults/events from SMMU. > > I didn’t find any other solution in XEN in place of tasklet to defer the > work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If > we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. > > If you know another solution in XEN that will be used to defer the work in > the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. >>> >>> Julien's suggestion is a good one. >>> >>> But I think tasklets can be configured to be called from the idle_loop, >>> in which case they are not run in interrupt context? >>> >> Yes you are right tasklet will be scheduled from the idle_loop that is not >> interrupt conext. > > This depends on your tasklet. Some will run from the softirq context which is > usually (for Arm) on the return of an exception. > Thanks for the info. I will check and will get better understanding of the tasklet how it will run in XEN. >>> >>> 4. Latest version of the Linux SMMUv3 code implements the commands queue >>>access functions based on atomic operations implemented in Linux. >> >> Can you provide more details? > > I tried to port the latest version of the SMMUv3 code than I observed that > in order to port that code I have to also port atomic operation > implemented > in Linux to XEN. As latest Linux code uses atomic operation to process the > command queues > (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , > atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? >>> >>> I think you are right. >> Yes, I agree with you to have XEN code in sync with Linux code that's why I >> started with to port the Linux atomic operations to XEN then I realised >> that it is not straightforward to port atomic operations and it requires >> lots of effort and testing. Therefore I decided to port the code before the >> atomic operation is introduced in Linux. > > Hmmm... I would not have expected a lot of effort required to add the 3 > atomics operations above. Are you trying to also port the LSE support at the > same time? There are other atomic operations used in the SMMUv3 code apart from the 3 atomic operation I mention. I just mention 3 operation as an example. I tried to port at that time but when I start porting I realised that one atomic operation depend on another one so I decided not to proceed further. > > Cheers, > > -- > Julien
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 23/10/2020 12:35, Rahul Singh wrote: Hello, On 23 Oct 2020, at 1:02 am, Stefano Stabellini wrote: On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. Julien's suggestion is a good one. But I think tasklets can be configured to be called from the idle_loop, in which case they are not run in interrupt context? Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. This depends on your tasklet. Some will run from the softirq context which is usually (for Arm) on the return of an exception. 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. Can you provide more details? I tried to port the latest version of the SMMUv3 code than I observed that in order to port that code I have to also port atomic operation implemented in Linux to XEN. As latest Linux code uses atomic operation to process the command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? I think you are right. Yes, I agree with you to have XEN code in sync with Linux code that's why I started with to port the Linux atomic operations to XEN then I realised that it is not straightforward to port atomic operations and it requires lots of effort and testing. Therefore I decided to port the code before the atomic operation is introduced in Linux. Hmmm... I would not have expected a lot of effort required to add the 3 atomics operations above. Are you trying to also port the LSE support at the same time? Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Stefano, On 23/10/2020 01:02, Stefano Stabellini wrote: On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. Julien's suggestion is a good one. But I think tasklets can be configured to be called from the idle_loop, in which case they are not run in interrupt context? Tasklets can either run from the IDLE loop or from a softirq context. When running from a softirq context is may happen on return from receiving an interrupt. However, interrupts will always be enabled. So I am not sure what concern you are trying to raise here. Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 22 Oct 2020, at 9:32 am, Julien Grall wrote: > > > > On 21/10/2020 12:25, Rahul Singh wrote: >> Hello Julien, > > Hi Rahul, > >>> On 20 Oct 2020, at 6:03 pm, Julien Grall wrote: >>> >>> Hi Rahul, >>> >>> Thank you for the contribution. Lets make sure this attempt to SMMUv3 >>> support in Xen will be more successful than the other one :). >> Yes sure. >>> >>> I haven't reviewed the code yet, but I wanted to provide feedback on the >>> commit message. >>> >>> On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. >>> >>> Tasklets are not a replacement for threaded IRQ. In particular, they will >>> have priority over anything else (IOW nothing will run on the pCPU until >>> they are done). >>> >>> Do you know why Linux is using thread. Is it because of long running >>> operations? >> Yes you are right because of long running operations Linux is using the >> threaded IRQs. >> SMMUv3 reports fault/events bases on memory-based circular buffer queues not >> based on the register. As per my understanding, it is time-consuming to >> process the memory based queues in interrupt context because of that Linux >> is using threaded IRQ to process the faults/events from SMMU. >> I didn’t find any other solution in XEN in place of tasklet to defer the >> work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If >> we do all work in interrupt context we will make XEN less responsive. > > So we need to make sure that Xen continue to receives interrupts, but we also > need to make sure that a vCPU bound to the pCPU is also responsive. Yes I agree. > >> If you know another solution in XEN that will be used to defer the work in >> the interrupt please let me know I will try to use that. > > One of my work colleague encountered a similar problem recently. He had a > long running tasklet and wanted to be broken down in smaller chunk. > > We decided to use a timer to reschedule the taslket in the future. This > allows the scheduler to run other loads (e.g. vCPU) for some time. > > This is pretty hackish but I couldn't find a better solution as tasklet have > high priority. > > Maybe the other will have a better idea. Let me try to use the timer and will share my findings. > 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. >>> >>> Can you provide more details? >> I tried to port the latest version of the SMMUv3 code than I observed that >> in order to port that code I have to also port atomic operation implemented >> in Linux to XEN. As latest Linux code uses atomic operation to process the >> command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , >> atomic_fetch_andnot_relaxed()) . > > Thank you for the explanation. I think it would be best to import the atomic > helpers and use the latest code. > > This will ensure that we don't re-introduce bugs and also buy us some time > before the Linux and Xen driver diverge again too much. > > Stefano, what do you think? > >>> Atomic functions used by the commands queue access functions is not implemented in XEN therefore we decided to port the earlier version of the code. Once the proper atomic operations will be available in XEN the driver can be updated. Signed-off-by: Rahul Singh --- xen/drivers/passthrough/Kconfig | 10 + xen/drivers/passthrough/arm/Makefile |1 + xen/drivers/passthrough/arm/smmu-v3.c | 2847 + 3 files changed, 2858 insertions(+) >>> >>> This is quite significant patch to review. Is there any way to get it split >>> (maybe a verbatim Linux copy + Xen modification)? >> Yes, I understand this is a quite significant patch to review let me think >> to get it split. If it is ok for you to review this patch and provide your >> comments then it will great for us. > I will try to have a look next week. Thanks in advance ☺️ > > Cheers, > > -- > Julien Grall Regards, Rahul
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello, > On 23 Oct 2020, at 1:02 am, Stefano Stabellini wrote: > > On Thu, 22 Oct 2020, Julien Grall wrote: On 20/10/2020 16:25, Rahul Singh wrote: > Add support for ARM architected SMMUv3 implementations. It is based on > the Linux SMMUv3 driver. > Major differences between the Linux driver are as follows: > 1. Only Stage-2 translation is supported as compared to the Linux driver >that supports both Stage-1 and Stage-2 translations. > 2. Use P2M page table instead of creating one as SMMUv3 has the >capability to share the page tables with the CPU. > 3. Tasklets is used in place of threaded IRQ's in Linux for event queue >and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? >>> >>> Yes you are right because of long running operations Linux is using the >>> threaded IRQs. >>> >>> SMMUv3 reports fault/events bases on memory-based circular buffer queues not >>> based on the register. As per my understanding, it is time-consuming to >>> process the memory based queues in interrupt context because of that Linux >>> is using threaded IRQ to process the faults/events from SMMU. >>> >>> I didn’t find any other solution in XEN in place of tasklet to defer the >>> work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If >>> we do all work in interrupt context we will make XEN less responsive. >> >> So we need to make sure that Xen continue to receives interrupts, but we also >> need to make sure that a vCPU bound to the pCPU is also responsive. >> >>> >>> If you know another solution in XEN that will be used to defer the work in >>> the interrupt please let me know I will try to use that. >> >> One of my work colleague encountered a similar problem recently. He had a >> long >> running tasklet and wanted to be broken down in smaller chunk. >> >> We decided to use a timer to reschedule the taslket in the future. This >> allows >> the scheduler to run other loads (e.g. vCPU) for some time. >> >> This is pretty hackish but I couldn't find a better solution as tasklet have >> high priority. >> >> Maybe the other will have a better idea. > > Julien's suggestion is a good one. > > But I think tasklets can be configured to be called from the idle_loop, > in which case they are not run in interrupt context? > Yes you are right tasklet will be scheduled from the idle_loop that is not interrupt conext. > Still, tasklets run until completion in Xen, which could take too long. > The code has to voluntarily release control of the execution flow once > it realizes it has been running for too long. The rescheduling via a > timer works. > > > Now, to brainstorm other possible alternatives, for hypercalls we have > been using hypercall continuations. Continuations is a way to break a > hypercall implementation that takes too long into multiple execution > chunks. It works by calling into itself again: making the same hypercall > again with updated arguments, so that the scheduler has a chance to do > other operations in between, including running other tasklets and > softirqs. > > That works well because the source of the work is a guest request, > specifically a hypercall. However, in the case of the SMMU driver, there > is no hypercall. The Xen driver has to do work in response to an > interrupt and the work is not tied to one particular domain. > > So I don't think the hypercall continuation model could work here. The > timer seems to be the best option. > Yes, I agree with you as the source of the work is not a guest request in the case of SMMU I think we can not use the hyper call continuation. As suggested I will try to use the timer to schedule the work and will share the findings. > > > 4. Latest version of the Linux SMMUv3 code implements the commands queue >access functions based on atomic operations implemented in Linux. Can you provide more details? >>> >>> I tried to port the latest version of the SMMUv3 code than I observed that >>> in order to port that code I have to also port atomic operation implemented >>> in Linux to XEN. As latest Linux code uses atomic operation to process the >>> command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , >>> atomic_fetch_andnot_relaxed()) . >> >> Thank you for the explanation. I think it would be best to import the atomic >> helpers and use the latest code. >> >> This will ensure that we don't re-introduce bugs and also buy us some time >> before the Linux and Xen driver diverge again too much. >> >> Stefano, what do you think? > > I think you are right. Yes, I agree with you to have XEN code in sync with Linux code that's why I started with to port the Linux atomic
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On Thu, 22 Oct 2020, Julien Grall wrote: > > > On 20/10/2020 16:25, Rahul Singh wrote: > > > > Add support for ARM architected SMMUv3 implementations. It is based on > > > > the Linux SMMUv3 driver. > > > > Major differences between the Linux driver are as follows: > > > > 1. Only Stage-2 translation is supported as compared to the Linux driver > > > > that supports both Stage-1 and Stage-2 translations. > > > > 2. Use P2M page table instead of creating one as SMMUv3 has the > > > > capability to share the page tables with the CPU. > > > > 3. Tasklets is used in place of threaded IRQ's in Linux for event queue > > > > and priority queue IRQ handling. > > > > > > Tasklets are not a replacement for threaded IRQ. In particular, they will > > > have priority over anything else (IOW nothing will run on the pCPU until > > > they are done). > > > > > > Do you know why Linux is using thread. Is it because of long running > > > operations? > > > > Yes you are right because of long running operations Linux is using the > > threaded IRQs. > > > > SMMUv3 reports fault/events bases on memory-based circular buffer queues not > > based on the register. As per my understanding, it is time-consuming to > > process the memory based queues in interrupt context because of that Linux > > is using threaded IRQ to process the faults/events from SMMU. > > > > I didn’t find any other solution in XEN in place of tasklet to defer the > > work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If > > we do all work in interrupt context we will make XEN less responsive. > > So we need to make sure that Xen continue to receives interrupts, but we also > need to make sure that a vCPU bound to the pCPU is also responsive. > > > > > If you know another solution in XEN that will be used to defer the work in > > the interrupt please let me know I will try to use that. > > One of my work colleague encountered a similar problem recently. He had a long > running tasklet and wanted to be broken down in smaller chunk. > > We decided to use a timer to reschedule the taslket in the future. This allows > the scheduler to run other loads (e.g. vCPU) for some time. > > This is pretty hackish but I couldn't find a better solution as tasklet have > high priority. > > Maybe the other will have a better idea. Julien's suggestion is a good one. But I think tasklets can be configured to be called from the idle_loop, in which case they are not run in interrupt context? Still, tasklets run until completion in Xen, which could take too long. The code has to voluntarily release control of the execution flow once it realizes it has been running for too long. The rescheduling via a timer works. Now, to brainstorm other possible alternatives, for hypercalls we have been using hypercall continuations. Continuations is a way to break a hypercall implementation that takes too long into multiple execution chunks. It works by calling into itself again: making the same hypercall again with updated arguments, so that the scheduler has a chance to do other operations in between, including running other tasklets and softirqs. That works well because the source of the work is a guest request, specifically a hypercall. However, in the case of the SMMU driver, there is no hypercall. The Xen driver has to do work in response to an interrupt and the work is not tied to one particular domain. So I don't think the hypercall continuation model could work here. The timer seems to be the best option. > > > > 4. Latest version of the Linux SMMUv3 code implements the commands queue > > > > access functions based on atomic operations implemented in Linux. > > > > > > Can you provide more details? > > > > I tried to port the latest version of the SMMUv3 code than I observed that > > in order to port that code I have to also port atomic operation implemented > > in Linux to XEN. As latest Linux code uses atomic operation to process the > > command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , > > atomic_fetch_andnot_relaxed()) . > > Thank you for the explanation. I think it would be best to import the atomic > helpers and use the latest code. > > This will ensure that we don't re-introduce bugs and also buy us some time > before the Linux and Xen driver diverge again too much. > > Stefano, what do you think? I think you are right.
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
On 21/10/2020 12:25, Rahul Singh wrote: Hello Julien, Hi Rahul, On 20 Oct 2020, at 6:03 pm, Julien Grall wrote: Hi Rahul, Thank you for the contribution. Lets make sure this attempt to SMMUv3 support in Xen will be more successful than the other one :). Yes sure. I haven't reviewed the code yet, but I wanted to provide feedback on the commit message. On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. So we need to make sure that Xen continue to receives interrupts, but we also need to make sure that a vCPU bound to the pCPU is also responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. One of my work colleague encountered a similar problem recently. He had a long running tasklet and wanted to be broken down in smaller chunk. We decided to use a timer to reschedule the taslket in the future. This allows the scheduler to run other loads (e.g. vCPU) for some time. This is pretty hackish but I couldn't find a better solution as tasklet have high priority. Maybe the other will have a better idea. 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. Can you provide more details? I tried to port the latest version of the SMMUv3 code than I observed that in order to port that code I have to also port atomic operation implemented in Linux to XEN. As latest Linux code uses atomic operation to process the command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , atomic_fetch_andnot_relaxed()) . Thank you for the explanation. I think it would be best to import the atomic helpers and use the latest code. This will ensure that we don't re-introduce bugs and also buy us some time before the Linux and Xen driver diverge again too much. Stefano, what do you think? Atomic functions used by the commands queue access functions is not implemented in XEN therefore we decided to port the earlier version of the code. Once the proper atomic operations will be available in XEN the driver can be updated. Signed-off-by: Rahul Singh --- xen/drivers/passthrough/Kconfig | 10 + xen/drivers/passthrough/arm/Makefile |1 + xen/drivers/passthrough/arm/smmu-v3.c | 2847 + 3 files changed, 2858 insertions(+) This is quite significant patch to review. Is there any way to get it split (maybe a verbatim Linux copy + Xen modification)? Yes, I understand this is a quite significant patch to review let me think to get it split. If it is ok for you to review this patch and provide your comments then it will great for us. I will try to have a look next week. Cheers, -- Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hello Julien, > On 20 Oct 2020, at 6:03 pm, Julien Grall wrote: > > Hi Rahul, > > Thank you for the contribution. Lets make sure this attempt to SMMUv3 support > in Xen will be more successful than the other one :). Yes sure. > > I haven't reviewed the code yet, but I wanted to provide feedback on the > commit message. > > On 20/10/2020 16:25, Rahul Singh wrote: >> Add support for ARM architected SMMUv3 implementations. It is based on >> the Linux SMMUv3 driver. >> Major differences between the Linux driver are as follows: >> 1. Only Stage-2 translation is supported as compared to the Linux driver >>that supports both Stage-1 and Stage-2 translations. >> 2. Use P2M page table instead of creating one as SMMUv3 has the >>capability to share the page tables with the CPU. >> 3. Tasklets is used in place of threaded IRQ's in Linux for event queue >>and priority queue IRQ handling. > > Tasklets are not a replacement for threaded IRQ. In particular, they will > have priority over anything else (IOW nothing will run on the pCPU until they > are done). > > Do you know why Linux is using thread. Is it because of long running > operations? Yes you are right because of long running operations Linux is using the threaded IRQs. SMMUv3 reports fault/events bases on memory-based circular buffer queues not based on the register. As per my understanding, it is time-consuming to process the memory based queues in interrupt context because of that Linux is using threaded IRQ to process the faults/events from SMMU. I didn’t find any other solution in XEN in place of tasklet to defer the work, that’s why I used tasklet in XEN in replacement of threaded IRQs. If we do all work in interrupt context we will make XEN less responsive. If you know another solution in XEN that will be used to defer the work in the interrupt please let me know I will try to use that. >> 4. Latest version of the Linux SMMUv3 code implements the commands queue >>access functions based on atomic operations implemented in Linux. > > Can you provide more details? I tried to port the latest version of the SMMUv3 code than I observed that in order to port that code I have to also port atomic operation implemented in Linux to XEN. As latest Linux code uses atomic operation to process the command queues (atomic_cond_read_relaxed(),atomic_long_cond_read_relaxed() , atomic_fetch_andnot_relaxed()) . > >>Atomic functions used by the commands queue access functions is not >>implemented in XEN therefore we decided to port the earlier version >>of the code. Once the proper atomic operations will be available in XEN >>the driver can be updated. >> Signed-off-by: Rahul Singh >> --- >> xen/drivers/passthrough/Kconfig | 10 + >> xen/drivers/passthrough/arm/Makefile |1 + >> xen/drivers/passthrough/arm/smmu-v3.c | 2847 + >> 3 files changed, 2858 insertions(+) > > This is quite significant patch to review. Is there any way to get it split > (maybe a verbatim Linux copy + Xen modification)? Yes, I understand this is a quite significant patch to review let me think to get it split. If it is ok for you to review this patch and provide your comments then it will great for us. > > Cheers, > > -- > Julien Grall
Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver
Hi Rahul, Thank you for the contribution. Lets make sure this attempt to SMMUv3 support in Xen will be more successful than the other one :). I haven't reviewed the code yet, but I wanted to provide feedback on the commit message. On 20/10/2020 16:25, Rahul Singh wrote: Add support for ARM architected SMMUv3 implementations. It is based on the Linux SMMUv3 driver. Major differences between the Linux driver are as follows: 1. Only Stage-2 translation is supported as compared to the Linux driver that supports both Stage-1 and Stage-2 translations. 2. Use P2M page table instead of creating one as SMMUv3 has the capability to share the page tables with the CPU. 3. Tasklets is used in place of threaded IRQ's in Linux for event queue and priority queue IRQ handling. Tasklets are not a replacement for threaded IRQ. In particular, they will have priority over anything else (IOW nothing will run on the pCPU until they are done). Do you know why Linux is using thread. Is it because of long running operations? 4. Latest version of the Linux SMMUv3 code implements the commands queue access functions based on atomic operations implemented in Linux. Can you provide more details? Atomic functions used by the commands queue access functions is not implemented in XEN therefore we decided to port the earlier version of the code. Once the proper atomic operations will be available in XEN the driver can be updated. Signed-off-by: Rahul Singh --- xen/drivers/passthrough/Kconfig | 10 + xen/drivers/passthrough/arm/Makefile |1 + xen/drivers/passthrough/arm/smmu-v3.c | 2847 + 3 files changed, 2858 insertions(+) This is quite significant patch to review. Is there any way to get it split (maybe a verbatim Linux copy + Xen modification)? Cheers, -- Julien Grall
