RE: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Sricharan]

Hi Will, Rob,

>-Original Message-
>From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] 
>On Behalf Of Will Deacon
>Sent: Thursday, February 02, 2017 9:41 PM
>To: Sricharan <sricha...@codeaurora.org>
>Cc: linux-arm-ker...@lists.infradead.org; 'Rob Clark' <robdcl...@gmail.com>; 
>'Joerg Roedel' <j...@8bytes.org>; iommu@lists.linux-
>foundation.org
>Subject: Re: [PATCH 0/5] Implement SMMU passthrough using the default domain
>
>On Thu, Feb 02, 2017 at 09:15:19PM +0530, Sricharan wrote:
>> Hi Rob,
>>
>> >-Original Message-
>> >From: linux-arm-kernel 
>> >[mailto:linux-arm-kernel-boun...@lists.infradead.org] On Behalf Of Rob Clark
>> >Sent: Thursday, February 02, 2017 8:33 PM
>> >To: Joerg Roedel <j...@8bytes.org>
>> >Cc: Will Deacon <will.dea...@arm.com>; iommu@lists.linux-foundation.org; 
>> >Sricharan <sricha...@codeaurora.org>; linux-arm-
>> >ker...@lists.infradead.org
>> >Subject: Re: [PATCH 0/5] Implement SMMU passthrough using the default domain
>> >
>> >On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel <j...@8bytes.org> wrote:
>> >> On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
>> >>> Thanks for this series. We had a case with the GPU.
>> >>> The GPU's iommu was setup by kernel and the GPU
>> >>> also does dynamic updates for on-the-fly switching between
>> >>> process pagetables.  GPU driver was not using DMA domain and
>> >>> the GPU's firmware was always expecting to run out  of contextbank
>> >>>  '0' (although not correct) , which was not the case after the DMA domain
>> >>> was made default  as '0' was getting allocated for DMA domain and
>> >>> there were concerns about reusing the DMA domain as well.
>> >>> Now with this series, looks there is an way out of that that can be 
>> >>> tried.
>> >>>
>> >>> So should the default domain not be per device specific selectable ?
>> >>
>> >> Note that iommu-drivers can request direct-mapping for any given device
>> >> on its initializtion. This is used on x86 for devices that need a 1-1
>> >> mapping for some reason.
>> >>
>> >> Also device drivers can use the iommu-api and assign their own domain to
>> >> a device, which allows them to manage the dma address space on their
>> >> own.
>> >
>> >Part of the problem is that dev->archdata.dma_ops gets wired up to
>> >iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
>> >are not exposed to drivers, forcing us to use dma-mapping API
>> >(dma_map_sg, etc) for cache operations.
>> >
>> >Possibly we should just expose cache op's to drivers bypass this abuse
>> >of dma-mapping.
>> >
>> [1], with this, when the default domain in not DOMAIN_DMA, then
>> dev->archdata.dma_ops is not set to iommu_dma_ops , instead remains
>> to be swiotlb_ops. Is that not correct for gpu's unmanaged domain case ?
>>
>> https://www.spinics.net/lists/arm-kernel/msg556209.html
>>
>> >btw, Will, we definitely want this to *not* rely on kcmdline for the
>> >gpu with it's own private iommu case..
>>
>> Ya, that changes behavior for all devices and some might want
>> DMA_DOMAIN and some UNMANAGED.
>
>My patch changes the *default* domain. When would this ever be UNMANAGED?
>
>If you're using an UNMANAGED domain, I think you need to take care of the
>DMA ops yourself. There are likely missing functions to do that, but they
>should be added in a separate series.

Sorry, i should have said DOMAIN_DMA or DOMAIN_IDENTITY is set as
*default* domains from your patchset. So i have not tested this series
yet. But the requirement that i thought for the gpu was
* a context bank (specially '0') should not be reserved for DOMAIN_DMA
   * dma_ops should not be set to iommu_dma_ops for the gpu device.

I was seeing that both of that are satisfied in this series, if we choose
IDENTITY_DOMAIN as default and later gpu attaches it own
*UNMANAGED* domain.

The only thing i was thinking was, since we are changing the default domain
for all device from command line, some device which require DOMAIN_DMA
and some which require DOMAIN_IDENTITY as default was not possible.

Regards,
 Sricharan


___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Rob Clark]

On Thu, Feb 2, 2017 at 10:12 AM, Will Deacon  wrote:
> On Thu, Feb 02, 2017 at 10:02:50AM -0500, Rob Clark wrote:
>> On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel  wrote:
>> > On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
>> >> Thanks for this series. We had a case with the GPU.
>> >> The GPU's iommu was setup by kernel and the GPU
>> >> also does dynamic updates for on-the-fly switching between
>> >> process pagetables.  GPU driver was not using DMA domain and
>> >> the GPU's firmware was always expecting to run out  of contextbank
>> >>  '0' (although not correct) , which was not the case after the DMA domain
>> >> was made default  as '0' was getting allocated for DMA domain and
>> >> there were concerns about reusing the DMA domain as well.
>> >> Now with this series, looks there is an way out of that that can be tried.
>> >>
>> >> So should the default domain not be per device specific selectable ?
>> >
>> > Note that iommu-drivers can request direct-mapping for any given device
>> > on its initializtion. This is used on x86 for devices that need a 1-1
>> > mapping for some reason.
>> >
>> > Also device drivers can use the iommu-api and assign their own domain to
>> > a device, which allows them to manage the dma address space on their
>> > own.
>>
>> Part of the problem is that dev->archdata.dma_ops gets wired up to
>> iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
>> are not exposed to drivers, forcing us to use dma-mapping API
>> (dma_map_sg, etc) for cache operations.
>>
>> Possibly we should just expose cache op's to drivers bypass this abuse
>> of dma-mapping.
>>
>> btw, Will, we definitely want this to *not* rely on kcmdline for the
>> gpu with it's own private iommu case..
>
> I still need to understand the unmanaged domain case, but I don't really
> see why that's related to this series to be honest.
>

Only relation is if we were trying to solve the case of a driver that
needs to manage it's own domain with this patchset..  but that seems a
bit like "when all you have is a hammer, everything looks like a
nail"..

BR,
-R
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Will Deacon]

On Thu, Feb 02, 2017 at 09:15:19PM +0530, Sricharan wrote:
> Hi Rob,
> 
> >-Original Message-
> >From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] 
> >On Behalf Of Rob Clark
> >Sent: Thursday, February 02, 2017 8:33 PM
> >To: Joerg Roedel <j...@8bytes.org>
> >Cc: Will Deacon <will.dea...@arm.com>; iommu@lists.linux-foundation.org; 
> >Sricharan <sricha...@codeaurora.org>; linux-arm-
> >ker...@lists.infradead.org
> >Subject: Re: [PATCH 0/5] Implement SMMU passthrough using the default domain
> >
> >On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel <j...@8bytes.org> wrote:
> >> On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
> >>> Thanks for this series. We had a case with the GPU.
> >>> The GPU's iommu was setup by kernel and the GPU
> >>> also does dynamic updates for on-the-fly switching between
> >>> process pagetables.  GPU driver was not using DMA domain and
> >>> the GPU's firmware was always expecting to run out  of contextbank
> >>>  '0' (although not correct) , which was not the case after the DMA domain
> >>> was made default  as '0' was getting allocated for DMA domain and
> >>> there were concerns about reusing the DMA domain as well.
> >>> Now with this series, looks there is an way out of that that can be tried.
> >>>
> >>> So should the default domain not be per device specific selectable ?
> >>
> >> Note that iommu-drivers can request direct-mapping for any given device
> >> on its initializtion. This is used on x86 for devices that need a 1-1
> >> mapping for some reason.
> >>
> >> Also device drivers can use the iommu-api and assign their own domain to
> >> a device, which allows them to manage the dma address space on their
> >> own.
> >
> >Part of the problem is that dev->archdata.dma_ops gets wired up to
> >iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
> >are not exposed to drivers, forcing us to use dma-mapping API
> >(dma_map_sg, etc) for cache operations.
> >
> >Possibly we should just expose cache op's to drivers bypass this abuse
> >of dma-mapping.
> >
> [1], with this, when the default domain in not DOMAIN_DMA, then
> dev->archdata.dma_ops is not set to iommu_dma_ops , instead remains
> to be swiotlb_ops. Is that not correct for gpu's unmanaged domain case ?
> 
> https://www.spinics.net/lists/arm-kernel/msg556209.html
> 
> >btw, Will, we definitely want this to *not* rely on kcmdline for the
> >gpu with it's own private iommu case..
> 
> Ya, that changes behavior for all devices and some might want 
> DMA_DOMAIN and some UNMANAGED.

My patch changes the *default* domain. When would this ever be UNMANAGED?

If you're using an UNMANAGED domain, I think you need to take care of the
DMA ops yourself. There are likely missing functions to do that, but they
should be added in a separate series.

Will
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Sricharan]

Hi Rob,

>-Original Message-
>From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] 
>On Behalf Of Rob Clark
>Sent: Thursday, February 02, 2017 8:33 PM
>To: Joerg Roedel <j...@8bytes.org>
>Cc: Will Deacon <will.dea...@arm.com>; iommu@lists.linux-foundation.org; 
>Sricharan <sricha...@codeaurora.org>; linux-arm-
>ker...@lists.infradead.org
>Subject: Re: [PATCH 0/5] Implement SMMU passthrough using the default domain
>
>On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel <j...@8bytes.org> wrote:
>> On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
>>> Thanks for this series. We had a case with the GPU.
>>> The GPU's iommu was setup by kernel and the GPU
>>> also does dynamic updates for on-the-fly switching between
>>> process pagetables.  GPU driver was not using DMA domain and
>>> the GPU's firmware was always expecting to run out  of contextbank
>>>  '0' (although not correct) , which was not the case after the DMA domain
>>> was made default  as '0' was getting allocated for DMA domain and
>>> there were concerns about reusing the DMA domain as well.
>>> Now with this series, looks there is an way out of that that can be tried.
>>>
>>> So should the default domain not be per device specific selectable ?
>>
>> Note that iommu-drivers can request direct-mapping for any given device
>> on its initializtion. This is used on x86 for devices that need a 1-1
>> mapping for some reason.
>>
>> Also device drivers can use the iommu-api and assign their own domain to
>> a device, which allows them to manage the dma address space on their
>> own.
>
>Part of the problem is that dev->archdata.dma_ops gets wired up to
>iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
>are not exposed to drivers, forcing us to use dma-mapping API
>(dma_map_sg, etc) for cache operations.
>
>Possibly we should just expose cache op's to drivers bypass this abuse
>of dma-mapping.
>
[1], with this, when the default domain in not DOMAIN_DMA, then
dev->archdata.dma_ops is not set to iommu_dma_ops , instead remains
to be swiotlb_ops. Is that not correct for gpu's unmanaged domain case ?

https://www.spinics.net/lists/arm-kernel/msg556209.html

>btw, Will, we definitely want this to *not* rely on kcmdline for the
>gpu with it's own private iommu case..

Ya, that changes behavior for all devices and some might want 
DMA_DOMAIN and some UNMANAGED.

Regards,
 Sricharan

>
>BR,
>-R
>
>___
>linux-arm-kernel mailing list
>linux-arm-ker...@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Will Deacon]

On Thu, Feb 02, 2017 at 10:02:50AM -0500, Rob Clark wrote:
> On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel  wrote:
> > On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
> >> Thanks for this series. We had a case with the GPU.
> >> The GPU's iommu was setup by kernel and the GPU
> >> also does dynamic updates for on-the-fly switching between
> >> process pagetables.  GPU driver was not using DMA domain and
> >> the GPU's firmware was always expecting to run out  of contextbank
> >>  '0' (although not correct) , which was not the case after the DMA domain
> >> was made default  as '0' was getting allocated for DMA domain and
> >> there were concerns about reusing the DMA domain as well.
> >> Now with this series, looks there is an way out of that that can be tried.
> >>
> >> So should the default domain not be per device specific selectable ?
> >
> > Note that iommu-drivers can request direct-mapping for any given device
> > on its initializtion. This is used on x86 for devices that need a 1-1
> > mapping for some reason.
> >
> > Also device drivers can use the iommu-api and assign their own domain to
> > a device, which allows them to manage the dma address space on their
> > own.
> 
> Part of the problem is that dev->archdata.dma_ops gets wired up to
> iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
> are not exposed to drivers, forcing us to use dma-mapping API
> (dma_map_sg, etc) for cache operations.
> 
> Possibly we should just expose cache op's to drivers bypass this abuse
> of dma-mapping.
> 
> btw, Will, we definitely want this to *not* rely on kcmdline for the
> gpu with it's own private iommu case..

I still need to understand the unmanaged domain case, but I don't really
see why that's related to this series to be honest.

Will
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Rob Clark]

On Thu, Jan 26, 2017 at 12:18 PM, Joerg Roedel  wrote:
> On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
>> Thanks for this series. We had a case with the GPU.
>> The GPU's iommu was setup by kernel and the GPU
>> also does dynamic updates for on-the-fly switching between
>> process pagetables.  GPU driver was not using DMA domain and
>> the GPU's firmware was always expecting to run out  of contextbank
>>  '0' (although not correct) , which was not the case after the DMA domain
>> was made default  as '0' was getting allocated for DMA domain and
>> there were concerns about reusing the DMA domain as well.
>> Now with this series, looks there is an way out of that that can be tried.
>>
>> So should the default domain not be per device specific selectable ?
>
> Note that iommu-drivers can request direct-mapping for any given device
> on its initializtion. This is used on x86 for devices that need a 1-1
> mapping for some reason.
>
> Also device drivers can use the iommu-api and assign their own domain to
> a device, which allows them to manage the dma address space on their
> own.

Part of the problem is that dev->archdata.dma_ops gets wired up to
iommu_dma_ops.  Which isn't so bad on it's own, except that cache ops
are not exposed to drivers, forcing us to use dma-mapping API
(dma_map_sg, etc) for cache operations.

Possibly we should just expose cache op's to drivers bypass this abuse
of dma-mapping.

btw, Will, we definitely want this to *not* rely on kcmdline for the
gpu with it's own private iommu case..

BR,
-R
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Joerg Roedel]

On Tue, Jan 24, 2017 at 08:42:23PM +0530, Sricharan wrote:
> Thanks for this series. We had a case with the GPU.
> The GPU's iommu was setup by kernel and the GPU
> also does dynamic updates for on-the-fly switching between
> process pagetables.  GPU driver was not using DMA domain and
> the GPU's firmware was always expecting to run out  of contextbank
>  '0' (although not correct) , which was not the case after the DMA domain
> was made default  as '0' was getting allocated for DMA domain and
> there were concerns about reusing the DMA domain as well.
> Now with this series, looks there is an way out of that that can be tried.
> 
> So should the default domain not be per device specific selectable ?

Note that iommu-drivers can request direct-mapping for any given device
on its initializtion. This is used on x86 for devices that need a 1-1
mapping for some reason.

Also device drivers can use the iommu-api and assign their own domain to
a device, which allows them to manage the dma address space on their
own.


Joerg

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/5] Implement SMMU passthrough using the default domain

[Sricharan]

Hi Will,

>Hi all,
>
>A number of people have expressed interest in having the SMMU come up in
>a passthrough configuration, and then allow subsequent translation for
>things such as VFIO. Rather than do this in each SMMU driver, it's much
>cleaner to allow the default domain to be configured to be something other
>than DMA.
>
>This patch series implements a command-line option to configure the
>default domain type. Currently, it supports "dma" and "identity" which
>is sufficient for the passthrough use-case.
>
>Tested on an ARM fastmodel.
>
>All feedback welcome,
>

Thanks for this series. We had a case with the GPU.
The GPU's iommu was setup by kernel and the GPU
also does dynamic updates for on-the-fly switching between
process pagetables.  GPU driver was not using DMA domain and
the GPU's firmware was always expecting to run out  of contextbank
 '0' (although not correct) , which was not the case after the DMA domain
was made default  as '0' was getting allocated for DMA domain and
there were concerns about reusing the DMA domain as well.
Now with this series, looks there is an way out of that that can be tried.

So should the default domain not be per device specific selectable ?

Regards,
 Sricharan



>Will
>
>--->8
>
>Will Deacon (5):
>  iommu/arm-smmu: Restrict domain attributes to UNMANAGED domains
>  iommu/arm-smmu: Install bypass S2CRs for IOMMU_DOMAIN_IDENTITY domains
>  iommu/arm-smmu-v3: Install bypass STEs for IOMMU_DOMAIN_IDENTITY
>domains
>  arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA
>  iommu: Allow default domain type to be set on the kernel command line
>
> arch/arm64/mm/dma-mapping.c | 17 -
> drivers/iommu/arm-smmu-v3.c | 20 ++--
> drivers/iommu/arm-smmu.c| 26 +++---
> drivers/iommu/iommu.c   | 19 +--
> 4 files changed, 70 insertions(+), 12 deletions(-)
>
>--
>2.1.4
>
>
>___
>linux-arm-kernel mailing list
>linux-arm-ker...@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu