Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Don,

On 11/12/2016 03:05, Don Dutile wrote:
> On 12/08/2016 04:36 AM, Auger Eric wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>
>>
>> While I am respinning this series into v4, here is a tentative summary
>> of technical topics for which no consensus was reached at this point.
>>
>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>> ranges. Not discussed at/after LPC.
>> x I currently report reserved regions. Alex expressed the need to
>>   report the full usable IOVA range instead (x86 min-max range
>>   minus MSI APIC window). I think this is meaningful for ARM
>>   too where arm-smmu might not support the full 64b range.
>> x Any objection we report the usable IOVA regions instead?
>>
>> 2) Shall the kernel check collision with MSI window* when userspace
>> calls VFIO_IOMMU_MAP_DMA?
>> Joerg/Will No; Alex yes
>> *for IOVA regions consumed downstream to the IOMMU: everyone says NO
>>
>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
> Um, I'm missing context, but the only thing I recall saying no to wrt RMRR
> is that _any_ device that has an RMRR cannot be assigned to a guest.
Yes that was my understanding
> Or, are you saying, RMRR's should be exposed in the guest os?  if so, then
> you have my 'no' there.
> 
>> My current series does not expose them in iommu group sysfs.
>> I understand we can expose the RMRR regions in the iomm group sysfs
>> without necessarily supporting RMRR requiring device assignment.
> This sentence doesn't make sense to me.
> Can you try re-wording it?
> I can't tell what RMRR has to do w/device assignment, other than what I
> said above.
> Exposing RMRR's in sysfs is not an issue in general.
Sorry for the confusion. I Meant we can expose RMRR regions as part of
the reserved regions through the iommu group sysfs API without
supporting device assignment of devices that has RMRR.

Hope it clarifies

Eric
> 
>> We can also add this support later.
>>
>> Thanks
>>
>> Eric
>>
>>
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region
>>> callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>the reserved regions
>>> - at the moment ARM MSI base address/size is left unchanged compared
>>>to v2
>>> - we currently report reserved regions and not usable IOVA regions as
>>>requested by Alex
>>>
>>> RFC v1 -> v2:
>>> - fix intel_add_reserved_regions
>>> - add mutex lock/unlock in vfio_iommu_type1
>>>
>>>
>>> Eric Auger (10):
>>>iommu/dma: Allow MSI-only cookies
>>>iommu: Rename iommu_dm_regions into iommu_resv_regions
>>>iommu: Add new reserved IOMMU attributes
>>>iommu: iommu_alloc_resv_region
>>>iommu: Do not map reserved regions
>>>iommu: iommu_get_group_resv_regions
>>>iommu: Implement reserved_regions iommu-group sysfs file
>>>iommu/vt-d: Implement reserved region get/put callbacks
>>>iommu/arm-smmu: Implement reserved region get/put callbacks
>>>vfio/type1: Get MSI cookie
>>>
>>>   drivers/iommu/amd_iommu.c   |  20 +++---
>>>   drivers/iommu/arm-smmu.c|  52 +++
>>>   drivers/iommu/dma-iommu.c   | 116
>>> ++---
>>>   drivers/iommu/intel-iommu.c |  50 ++
>>>   drivers/iommu/iommu.c   | 141
>>> 
>>>   drivers/vfio/vfio_iommu_type1.c |  26 
>>>   include/linux/dma-iommu.h   |   7 ++
>>>   include/linux/iommu.h   |  49 ++
>>>   8 files changed, 391 insertions(+), 70 deletions(-)
>>>
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Don,

On 11/12/2016 03:05, Don Dutile wrote:
> On 12/08/2016 04:36 AM, Auger Eric wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>
>>
>> While I am respinning this series into v4, here is a tentative summary
>> of technical topics for which no consensus was reached at this point.
>>
>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>> ranges. Not discussed at/after LPC.
>> x I currently report reserved regions. Alex expressed the need to
>>   report the full usable IOVA range instead (x86 min-max range
>>   minus MSI APIC window). I think this is meaningful for ARM
>>   too where arm-smmu might not support the full 64b range.
>> x Any objection we report the usable IOVA regions instead?
>>
>> 2) Shall the kernel check collision with MSI window* when userspace
>> calls VFIO_IOMMU_MAP_DMA?
>> Joerg/Will No; Alex yes
>> *for IOVA regions consumed downstream to the IOMMU: everyone says NO
>>
>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
> Um, I'm missing context, but the only thing I recall saying no to wrt RMRR
> is that _any_ device that has an RMRR cannot be assigned to a guest.
Yes that was my understanding
> Or, are you saying, RMRR's should be exposed in the guest os?  if so, then
> you have my 'no' there.
> 
>> My current series does not expose them in iommu group sysfs.
>> I understand we can expose the RMRR regions in the iomm group sysfs
>> without necessarily supporting RMRR requiring device assignment.
> This sentence doesn't make sense to me.
> Can you try re-wording it?
> I can't tell what RMRR has to do w/device assignment, other than what I
> said above.
> Exposing RMRR's in sysfs is not an issue in general.
Sorry for the confusion. I Meant we can expose RMRR regions as part of
the reserved regions through the iommu group sysfs API without
supporting device assignment of devices that has RMRR.

Hope it clarifies

Eric
> 
>> We can also add this support later.
>>
>> Thanks
>>
>> Eric
>>
>>
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region
>>> callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>the reserved regions
>>> - at the moment ARM MSI base address/size is left unchanged compared
>>>to v2
>>> - we currently report reserved regions and not usable IOVA regions as
>>>requested by Alex
>>>
>>> RFC v1 -> v2:
>>> - fix intel_add_reserved_regions
>>> - add mutex lock/unlock in vfio_iommu_type1
>>>
>>>
>>> Eric Auger (10):
>>>iommu/dma: Allow MSI-only cookies
>>>iommu: Rename iommu_dm_regions into iommu_resv_regions
>>>iommu: Add new reserved IOMMU attributes
>>>iommu: iommu_alloc_resv_region
>>>iommu: Do not map reserved regions
>>>iommu: iommu_get_group_resv_regions
>>>iommu: Implement reserved_regions iommu-group sysfs file
>>>iommu/vt-d: Implement reserved region get/put callbacks
>>>iommu/arm-smmu: Implement reserved region get/put callbacks
>>>vfio/type1: Get MSI cookie
>>>
>>>   drivers/iommu/amd_iommu.c   |  20 +++---
>>>   drivers/iommu/arm-smmu.c|  52 +++
>>>   drivers/iommu/dma-iommu.c   | 116
>>> ++---
>>>   drivers/iommu/intel-iommu.c |  50 ++
>>>   drivers/iommu/iommu.c   | 141
>>> 
>>>   drivers/vfio/vfio_iommu_type1.c |  26 
>>>   include/linux/dma-iommu.h   |   7 ++
>>>   include/linux/iommu.h   |  49 ++
>>>   8 files changed, 391 insertions(+), 70 deletions(-)
>>>
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Don Dutile]

On 12/08/2016 04:36 AM, Auger Eric wrote:

Hi,

On 15/11/2016 14:09, Eric Auger wrote:

Following LPC discussions, we now report reserved regions through
iommu-group sysfs reserved_regions attribute file.



While I am respinning this series into v4, here is a tentative summary
of technical topics for which no consensus was reached at this point.

1) Shall we report the usable IOVA range instead of reserved IOVA
ranges. Not discussed at/after LPC.
x I currently report reserved regions. Alex expressed the need to
  report the full usable IOVA range instead (x86 min-max range
  minus MSI APIC window). I think this is meaningful for ARM
  too where arm-smmu might not support the full 64b range.
x Any objection we report the usable IOVA regions instead?

2) Shall the kernel check collision with MSI window* when userspace
calls VFIO_IOMMU_MAP_DMA?
Joerg/Will No; Alex yes
*for IOVA regions consumed downstream to the IOMMU: everyone says NO

3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no

Um, I'm missing context, but the only thing I recall saying no to wrt RMRR
is that _any_ device that has an RMRR cannot be assigned to a guest.
Or, are you saying, RMRR's should be exposed in the guest os?  if so, then
you have my 'no' there.


My current series does not expose them in iommu group sysfs.
I understand we can expose the RMRR regions in the iomm group sysfs
without necessarily supporting RMRR requiring device assignment.

This sentence doesn't make sense to me.
Can you try re-wording it?
I can't tell what RMRR has to do w/device assignment, other than what I said 
above.
Exposing RMRR's in sysfs is not an issue in general.


We can also add this support later.

Thanks

Eric




Reserved regions are populated through the IOMMU get_resv_region callback
(former get_dm_regions), now implemented by amd-iommu, intel-iommu and
arm-smmu.

The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
IOMMU_RESV_NOMAP reserved region.

arm-smmu reports the MSI window (arbitrarily located at 0x800 and
1MB large) and the PCI host bridge windows.

The series integrates a not officially posted patch from Robin:
"iommu/dma: Allow MSI-only cookies".

This series currently does not address IRQ safety assessment.

Best Regards

Eric

Git: complete series available at
https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3

History:
RFC v2 -> v3:
- switch to an iommu-group sysfs API
- use new dummy allocator provided by Robin
- dummy allocator initialized by vfio-iommu-type1 after enumerating
   the reserved regions
- at the moment ARM MSI base address/size is left unchanged compared
   to v2
- we currently report reserved regions and not usable IOVA regions as
   requested by Alex

RFC v1 -> v2:
- fix intel_add_reserved_regions
- add mutex lock/unlock in vfio_iommu_type1


Eric Auger (10):
   iommu/dma: Allow MSI-only cookies
   iommu: Rename iommu_dm_regions into iommu_resv_regions
   iommu: Add new reserved IOMMU attributes
   iommu: iommu_alloc_resv_region
   iommu: Do not map reserved regions
   iommu: iommu_get_group_resv_regions
   iommu: Implement reserved_regions iommu-group sysfs file
   iommu/vt-d: Implement reserved region get/put callbacks
   iommu/arm-smmu: Implement reserved region get/put callbacks
   vfio/type1: Get MSI cookie

  drivers/iommu/amd_iommu.c   |  20 +++---
  drivers/iommu/arm-smmu.c|  52 +++
  drivers/iommu/dma-iommu.c   | 116 ++---
  drivers/iommu/intel-iommu.c |  50 ++
  drivers/iommu/iommu.c   | 141 
  drivers/vfio/vfio_iommu_type1.c |  26 
  include/linux/dma-iommu.h   |   7 ++
  include/linux/iommu.h   |  49 ++
  8 files changed, 391 insertions(+), 70 deletions(-)

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Don Dutile]

On 12/08/2016 04:36 AM, Auger Eric wrote:

Hi,

On 15/11/2016 14:09, Eric Auger wrote:

Following LPC discussions, we now report reserved regions through
iommu-group sysfs reserved_regions attribute file.



While I am respinning this series into v4, here is a tentative summary
of technical topics for which no consensus was reached at this point.

1) Shall we report the usable IOVA range instead of reserved IOVA
ranges. Not discussed at/after LPC.
x I currently report reserved regions. Alex expressed the need to
  report the full usable IOVA range instead (x86 min-max range
  minus MSI APIC window). I think this is meaningful for ARM
  too where arm-smmu might not support the full 64b range.
x Any objection we report the usable IOVA regions instead?

2) Shall the kernel check collision with MSI window* when userspace
calls VFIO_IOMMU_MAP_DMA?
Joerg/Will No; Alex yes
*for IOVA regions consumed downstream to the IOMMU: everyone says NO

3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no

Um, I'm missing context, but the only thing I recall saying no to wrt RMRR
is that _any_ device that has an RMRR cannot be assigned to a guest.
Or, are you saying, RMRR's should be exposed in the guest os?  if so, then
you have my 'no' there.


My current series does not expose them in iommu group sysfs.
I understand we can expose the RMRR regions in the iomm group sysfs
without necessarily supporting RMRR requiring device assignment.

This sentence doesn't make sense to me.
Can you try re-wording it?
I can't tell what RMRR has to do w/device assignment, other than what I said 
above.
Exposing RMRR's in sysfs is not an issue in general.


We can also add this support later.

Thanks

Eric




Reserved regions are populated through the IOMMU get_resv_region callback
(former get_dm_regions), now implemented by amd-iommu, intel-iommu and
arm-smmu.

The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
IOMMU_RESV_NOMAP reserved region.

arm-smmu reports the MSI window (arbitrarily located at 0x800 and
1MB large) and the PCI host bridge windows.

The series integrates a not officially posted patch from Robin:
"iommu/dma: Allow MSI-only cookies".

This series currently does not address IRQ safety assessment.

Best Regards

Eric

Git: complete series available at
https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3

History:
RFC v2 -> v3:
- switch to an iommu-group sysfs API
- use new dummy allocator provided by Robin
- dummy allocator initialized by vfio-iommu-type1 after enumerating
   the reserved regions
- at the moment ARM MSI base address/size is left unchanged compared
   to v2
- we currently report reserved regions and not usable IOVA regions as
   requested by Alex

RFC v1 -> v2:
- fix intel_add_reserved_regions
- add mutex lock/unlock in vfio_iommu_type1


Eric Auger (10):
   iommu/dma: Allow MSI-only cookies
   iommu: Rename iommu_dm_regions into iommu_resv_regions
   iommu: Add new reserved IOMMU attributes
   iommu: iommu_alloc_resv_region
   iommu: Do not map reserved regions
   iommu: iommu_get_group_resv_regions
   iommu: Implement reserved_regions iommu-group sysfs file
   iommu/vt-d: Implement reserved region get/put callbacks
   iommu/arm-smmu: Implement reserved region get/put callbacks
   vfio/type1: Get MSI cookie

  drivers/iommu/amd_iommu.c   |  20 +++---
  drivers/iommu/arm-smmu.c|  52 +++
  drivers/iommu/dma-iommu.c   | 116 ++---
  drivers/iommu/intel-iommu.c |  50 ++
  drivers/iommu/iommu.c   | 141 
  drivers/vfio/vfio_iommu_type1.c |  26 
  include/linux/dma-iommu.h   |   7 ++
  include/linux/iommu.h   |  49 ++
  8 files changed, 391 insertions(+), 70 deletions(-)

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 17:01, Alex Williamson wrote:
> On Thu, 8 Dec 2016 13:14:04 +
> Robin Murphy  wrote:
>> On 08/12/16 09:36, Auger Eric wrote:
>>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>>My current series does not expose them in iommu group sysfs.
>>>I understand we can expose the RMRR regions in the iomm group sysfs
>>>without necessarily supporting RMRR requiring device assignment.
>>>We can also add this support later.  
>>
>> As you say, reporting them doesn't necessitate allowing device
>> assignment, and it's information which can already be easily grovelled
>> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
>> need to hide them, but the x86 folks can have the final word on that.
> 
> Eric and I talked about this and I don't see the value in identifying
> an RMRR as anything other than a reserved range for a device.  It's not
> userspace's job to maintain an identify mapped range for the device,
> and it can't be trusted to do so anyway.  It does throw a kink in the
> machinery though as an RMRR is a reserved memory range unique to a
> device.  It doesn't really fit into a monolithic /sys/class/iommu view
> of global reserved regions as an RMRR is only relevant to the device
> paths affected.

I think we're in violent agreement then - to clarify, I was thinking in
terms of patch 7 of this series, where everything relevant to a
particular group would be exposed as just an opaque "don't use this
address range" regardless of the internal type.

I'm less convinced the kernel has any need to provide its own 'global'
view of reservations which strictly are always at least per-IOMMU, if
not per-root-complex, even when all the instances do share the same
address by design. The group-based interface fits the reality neatly,
and userspace can easily iterate all the groups if it wants to consider
everything. Plus if it doesn't want to, then it needn't bother reserving
anything which doesn't apply to the group(s) it's going to bind to VFIO.

Robin.

> Another kink is that sometimes we know what the RMRR is for, know that
> it's irrelevant for our use case, and ignore it.  This is true for USB
> and Intel graphics use cases of RMRRs.
> 
> Also, aside from the above mentioned cases, devices with RMRRs are
> currently excluded from participating in the IOMMU API by the
> intel-iommu driver and I expect this to continue in the general case
> regardless of whether the ranges are more easily exposed to userspace.
> ARM may have to deal with mangling a guest memory map due to lack of
> any standard layout, de facto or otherwise, but for x86 I don't think
> it's worth the migration and hotplug implications.  Thanks,
> 
> Alex
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 17:01, Alex Williamson wrote:
> On Thu, 8 Dec 2016 13:14:04 +
> Robin Murphy  wrote:
>> On 08/12/16 09:36, Auger Eric wrote:
>>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>>My current series does not expose them in iommu group sysfs.
>>>I understand we can expose the RMRR regions in the iomm group sysfs
>>>without necessarily supporting RMRR requiring device assignment.
>>>We can also add this support later.  
>>
>> As you say, reporting them doesn't necessitate allowing device
>> assignment, and it's information which can already be easily grovelled
>> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
>> need to hide them, but the x86 folks can have the final word on that.
> 
> Eric and I talked about this and I don't see the value in identifying
> an RMRR as anything other than a reserved range for a device.  It's not
> userspace's job to maintain an identify mapped range for the device,
> and it can't be trusted to do so anyway.  It does throw a kink in the
> machinery though as an RMRR is a reserved memory range unique to a
> device.  It doesn't really fit into a monolithic /sys/class/iommu view
> of global reserved regions as an RMRR is only relevant to the device
> paths affected.

I think we're in violent agreement then - to clarify, I was thinking in
terms of patch 7 of this series, where everything relevant to a
particular group would be exposed as just an opaque "don't use this
address range" regardless of the internal type.

I'm less convinced the kernel has any need to provide its own 'global'
view of reservations which strictly are always at least per-IOMMU, if
not per-root-complex, even when all the instances do share the same
address by design. The group-based interface fits the reality neatly,
and userspace can easily iterate all the groups if it wants to consider
everything. Plus if it doesn't want to, then it needn't bother reserving
anything which doesn't apply to the group(s) it's going to bind to VFIO.

Robin.

> Another kink is that sometimes we know what the RMRR is for, know that
> it's irrelevant for our use case, and ignore it.  This is true for USB
> and Intel graphics use cases of RMRRs.
> 
> Also, aside from the above mentioned cases, devices with RMRRs are
> currently excluded from participating in the IOMMU API by the
> intel-iommu driver and I expect this to continue in the general case
> regardless of whether the ranges are more easily exposed to userspace.
> ARM may have to deal with mangling a guest memory map due to lack of
> any standard layout, de facto or otherwise, but for x86 I don't think
> it's worth the migration and hotplug implications.  Thanks,
> 
> Alex
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Alex Williamson]

On Thu, 8 Dec 2016 13:14:04 +
Robin Murphy  wrote:
> On 08/12/16 09:36, Auger Eric wrote:
> > 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
> >My current series does not expose them in iommu group sysfs.
> >I understand we can expose the RMRR regions in the iomm group sysfs
> >without necessarily supporting RMRR requiring device assignment.
> >We can also add this support later.  
> 
> As you say, reporting them doesn't necessitate allowing device
> assignment, and it's information which can already be easily grovelled
> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
> need to hide them, but the x86 folks can have the final word on that.

Eric and I talked about this and I don't see the value in identifying
an RMRR as anything other than a reserved range for a device.  It's not
userspace's job to maintain an identify mapped range for the device,
and it can't be trusted to do so anyway.  It does throw a kink in the
machinery though as an RMRR is a reserved memory range unique to a
device.  It doesn't really fit into a monolithic /sys/class/iommu view
of global reserved regions as an RMRR is only relevant to the device
paths affected.

Another kink is that sometimes we know what the RMRR is for, know that
it's irrelevant for our use case, and ignore it.  This is true for USB
and Intel graphics use cases of RMRRs.

Also, aside from the above mentioned cases, devices with RMRRs are
currently excluded from participating in the IOMMU API by the
intel-iommu driver and I expect this to continue in the general case
regardless of whether the ranges are more easily exposed to userspace.
ARM may have to deal with mangling a guest memory map due to lack of
any standard layout, de facto or otherwise, but for x86 I don't think
it's worth the migration and hotplug implications.  Thanks,

Alex

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Alex Williamson]

On Thu, 8 Dec 2016 13:14:04 +
Robin Murphy  wrote:
> On 08/12/16 09:36, Auger Eric wrote:
> > 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
> >My current series does not expose them in iommu group sysfs.
> >I understand we can expose the RMRR regions in the iomm group sysfs
> >without necessarily supporting RMRR requiring device assignment.
> >We can also add this support later.  
> 
> As you say, reporting them doesn't necessitate allowing device
> assignment, and it's information which can already be easily grovelled
> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
> need to hide them, but the x86 folks can have the final word on that.

Eric and I talked about this and I don't see the value in identifying
an RMRR as anything other than a reserved range for a device.  It's not
userspace's job to maintain an identify mapped range for the device,
and it can't be trusted to do so anyway.  It does throw a kink in the
machinery though as an RMRR is a reserved memory range unique to a
device.  It doesn't really fit into a monolithic /sys/class/iommu view
of global reserved regions as an RMRR is only relevant to the device
paths affected.

Another kink is that sometimes we know what the RMRR is for, know that
it's irrelevant for our use case, and ignore it.  This is true for USB
and Intel graphics use cases of RMRRs.

Also, aside from the above mentioned cases, devices with RMRRs are
currently excluded from participating in the IOMMU API by the
intel-iommu driver and I expect this to continue in the general case
regardless of whether the ranges are more easily exposed to userspace.
ARM may have to deal with mangling a guest memory map due to lack of
any standard layout, de facto or otherwise, but for x86 I don't think
it's worth the migration and hotplug implications.  Thanks,

Alex

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 13:36, Auger Eric wrote:
> Hi Robin,
> 
> On 08/12/2016 14:14, Robin Murphy wrote:
>> On 08/12/16 09:36, Auger Eric wrote:
>>> Hi,
>>>
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.
>>>
>>>
>>> While I am respinning this series into v4, here is a tentative summary
>>> of technical topics for which no consensus was reached at this point.
>>>
>>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>>>ranges. Not discussed at/after LPC.
>>>x I currently report reserved regions. Alex expressed the need to
>>>  report the full usable IOVA range instead (x86 min-max range
>>>  minus MSI APIC window). I think this is meaningful for ARM
>>>  too where arm-smmu might not support the full 64b range.
>>>x Any objection we report the usable IOVA regions instead?
>>
>> The issue with that is that we can't actually report "the usable
>> regions" at the moment, as that involves pulling together disjoint
>> properties of arbitrary hardware unrelated to the IOMMU. We'd be
>> reporting "the not-definitely-unusable regions, which may have some
>> unusable holes in them still". That seems like an ABI nightmare - I'd
>> still much rather say "here are some, but not necessarily all, regions
>> you definitely can't use", because saying "here are some regions which
>> you might be able to use most of, probably" is what we're already doing
>> today, via a single implicit region from 0 to ULONG_MAX ;)
>>
>> The address space limits are definitely useful to know, but I think it
>> would be better to expose them separately to avoid the ambiguity. At
>> worst, I guess it would be reasonable to express the limits via an
>> "out-of-range" reserved region type for 0 to $base and $top to
>> ULONG-MAX. To *safely* expose usable regions, we'd have to start out
>> with a very conservative assumption (e.g. only IOVAs matching physical
>> RAM), and only expand them once we're sure we can detect every possible
>> bit of problematic hardware in the system - that's just too limiting to
>> be useful. And if we expose something knowingly inaccurate, we risk
>> having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
>> nobody wants that...
> Makes sense to me. "out-of-range reserved region type for 0 to $base and
> $top to ULONG-MAX" can be an alternative to fulfill the requirement.
>>
>>> 2) Shall the kernel check collision with MSI window* when userspace
>>>calls VFIO_IOMMU_MAP_DMA?
>>>Joerg/Will No; Alex yes
>>>*for IOVA regions consumed downstream to the IOMMU: everyone says NO
>>
>> If we're starting off by having the SMMU drivers expose it as a fake
>> fixed region, I don't think we need to worry about this yet. We all seem
>> to agree that as long as we communicate the fixed regions to userspace,
>> it's then userspace's job to work around them. Let's come back to this
>> one once we actually get to the point of dynamically sizing and
>> allocating 'real' MSI remapping region(s).
>>
>> Ultimately, the kernel *will* police collisions either way, because an
>> underlying iommu_map() is going to fail if overlapping IOVAs are ever
>> actually used, so it's really just a question of whether to have a more
>> user-friendly failure mode.
> That's true on ARM but not on x86 where the APIC MSI region is not
> mapped I think.

Yes, but the APIC interrupt region is fixed, i.e. it falls under "IOVA
regions consumed downstream to the IOMMU" since the DMAR units are
physically incapable of remapping those addresses. I take "MSI window"
to mean specifically the thing we have to set aside and get a magic
remapping cookie for, which is also why it warrants its own special
internal type - we definitely *don't* want VFIO trying to set up a
remapping cookie on x86. We just don't let userspace know about the
difference, yet (if ever).

Robin.

>>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>>My current series does not expose them in iommu group sysfs.
>>>I understand we can expose the RMRR regions in the iomm group sysfs
>>>without necessarily supporting RMRR requiring device assignment.
>>>We can also add this support later.
>>
>> As you say, reporting them doesn't necessitate allowing device
>> assignment, and it's information which can already be easily grovelled
>> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
>> need to hide them, but the x86 folks can have the final word on that.
> agreed
> 
> Thanks
> 
> Eric
>>
>> Robin.
>>
>>> Thanks
>>>
>>> Eric
>>>
>>>

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 13:36, Auger Eric wrote:
> Hi Robin,
> 
> On 08/12/2016 14:14, Robin Murphy wrote:
>> On 08/12/16 09:36, Auger Eric wrote:
>>> Hi,
>>>
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.
>>>
>>>
>>> While I am respinning this series into v4, here is a tentative summary
>>> of technical topics for which no consensus was reached at this point.
>>>
>>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>>>ranges. Not discussed at/after LPC.
>>>x I currently report reserved regions. Alex expressed the need to
>>>  report the full usable IOVA range instead (x86 min-max range
>>>  minus MSI APIC window). I think this is meaningful for ARM
>>>  too where arm-smmu might not support the full 64b range.
>>>x Any objection we report the usable IOVA regions instead?
>>
>> The issue with that is that we can't actually report "the usable
>> regions" at the moment, as that involves pulling together disjoint
>> properties of arbitrary hardware unrelated to the IOMMU. We'd be
>> reporting "the not-definitely-unusable regions, which may have some
>> unusable holes in them still". That seems like an ABI nightmare - I'd
>> still much rather say "here are some, but not necessarily all, regions
>> you definitely can't use", because saying "here are some regions which
>> you might be able to use most of, probably" is what we're already doing
>> today, via a single implicit region from 0 to ULONG_MAX ;)
>>
>> The address space limits are definitely useful to know, but I think it
>> would be better to expose them separately to avoid the ambiguity. At
>> worst, I guess it would be reasonable to express the limits via an
>> "out-of-range" reserved region type for 0 to $base and $top to
>> ULONG-MAX. To *safely* expose usable regions, we'd have to start out
>> with a very conservative assumption (e.g. only IOVAs matching physical
>> RAM), and only expand them once we're sure we can detect every possible
>> bit of problematic hardware in the system - that's just too limiting to
>> be useful. And if we expose something knowingly inaccurate, we risk
>> having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
>> nobody wants that...
> Makes sense to me. "out-of-range reserved region type for 0 to $base and
> $top to ULONG-MAX" can be an alternative to fulfill the requirement.
>>
>>> 2) Shall the kernel check collision with MSI window* when userspace
>>>calls VFIO_IOMMU_MAP_DMA?
>>>Joerg/Will No; Alex yes
>>>*for IOVA regions consumed downstream to the IOMMU: everyone says NO
>>
>> If we're starting off by having the SMMU drivers expose it as a fake
>> fixed region, I don't think we need to worry about this yet. We all seem
>> to agree that as long as we communicate the fixed regions to userspace,
>> it's then userspace's job to work around them. Let's come back to this
>> one once we actually get to the point of dynamically sizing and
>> allocating 'real' MSI remapping region(s).
>>
>> Ultimately, the kernel *will* police collisions either way, because an
>> underlying iommu_map() is going to fail if overlapping IOVAs are ever
>> actually used, so it's really just a question of whether to have a more
>> user-friendly failure mode.
> That's true on ARM but not on x86 where the APIC MSI region is not
> mapped I think.

Yes, but the APIC interrupt region is fixed, i.e. it falls under "IOVA
regions consumed downstream to the IOMMU" since the DMAR units are
physically incapable of remapping those addresses. I take "MSI window"
to mean specifically the thing we have to set aside and get a magic
remapping cookie for, which is also why it warrants its own special
internal type - we definitely *don't* want VFIO trying to set up a
remapping cookie on x86. We just don't let userspace know about the
difference, yet (if ever).

Robin.

>>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>>My current series does not expose them in iommu group sysfs.
>>>I understand we can expose the RMRR regions in the iomm group sysfs
>>>without necessarily supporting RMRR requiring device assignment.
>>>We can also add this support later.
>>
>> As you say, reporting them doesn't necessitate allowing device
>> assignment, and it's information which can already be easily grovelled
>> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
>> need to hide them, but the x86 folks can have the final word on that.
> agreed
> 
> Thanks
> 
> Eric
>>
>> Robin.
>>
>>> Thanks
>>>
>>> Eric
>>>
>>>

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Robin,

On 08/12/2016 14:14, Robin Murphy wrote:
> On 08/12/16 09:36, Auger Eric wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>
>>
>> While I am respinning this series into v4, here is a tentative summary
>> of technical topics for which no consensus was reached at this point.
>>
>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>>ranges. Not discussed at/after LPC.
>>x I currently report reserved regions. Alex expressed the need to
>>  report the full usable IOVA range instead (x86 min-max range
>>  minus MSI APIC window). I think this is meaningful for ARM
>>  too where arm-smmu might not support the full 64b range.
>>x Any objection we report the usable IOVA regions instead?
> 
> The issue with that is that we can't actually report "the usable
> regions" at the moment, as that involves pulling together disjoint
> properties of arbitrary hardware unrelated to the IOMMU. We'd be
> reporting "the not-definitely-unusable regions, which may have some
> unusable holes in them still". That seems like an ABI nightmare - I'd
> still much rather say "here are some, but not necessarily all, regions
> you definitely can't use", because saying "here are some regions which
> you might be able to use most of, probably" is what we're already doing
> today, via a single implicit region from 0 to ULONG_MAX ;)
> 
> The address space limits are definitely useful to know, but I think it
> would be better to expose them separately to avoid the ambiguity. At
> worst, I guess it would be reasonable to express the limits via an
> "out-of-range" reserved region type for 0 to $base and $top to
> ULONG-MAX. To *safely* expose usable regions, we'd have to start out
> with a very conservative assumption (e.g. only IOVAs matching physical
> RAM), and only expand them once we're sure we can detect every possible
> bit of problematic hardware in the system - that's just too limiting to
> be useful. And if we expose something knowingly inaccurate, we risk
> having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
> nobody wants that...
Makes sense to me. "out-of-range reserved region type for 0 to $base and
$top to ULONG-MAX" can be an alternative to fulfill the requirement.
> 
>> 2) Shall the kernel check collision with MSI window* when userspace
>>calls VFIO_IOMMU_MAP_DMA?
>>Joerg/Will No; Alex yes
>>*for IOVA regions consumed downstream to the IOMMU: everyone says NO
> 
> If we're starting off by having the SMMU drivers expose it as a fake
> fixed region, I don't think we need to worry about this yet. We all seem
> to agree that as long as we communicate the fixed regions to userspace,
> it's then userspace's job to work around them. Let's come back to this
> one once we actually get to the point of dynamically sizing and
> allocating 'real' MSI remapping region(s).
> 
> Ultimately, the kernel *will* police collisions either way, because an
> underlying iommu_map() is going to fail if overlapping IOVAs are ever
> actually used, so it's really just a question of whether to have a more
> user-friendly failure mode.
That's true on ARM but not on x86 where the APIC MSI region is not
mapped I think.
> 
>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>My current series does not expose them in iommu group sysfs.
>>I understand we can expose the RMRR regions in the iomm group sysfs
>>without necessarily supporting RMRR requiring device assignment.
>>We can also add this support later.
> 
> As you say, reporting them doesn't necessitate allowing device
> assignment, and it's information which can already be easily grovelled
> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
> need to hide them, but the x86 folks can have the final word on that.
agreed

Thanks

Eric
> 
> Robin.
> 
>> Thanks
>>
>> Eric
>>
>>
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>   the reserved regions
>>> - at the moment ARM MSI 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Robin,

On 08/12/2016 14:14, Robin Murphy wrote:
> On 08/12/16 09:36, Auger Eric wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>
>>
>> While I am respinning this series into v4, here is a tentative summary
>> of technical topics for which no consensus was reached at this point.
>>
>> 1) Shall we report the usable IOVA range instead of reserved IOVA
>>ranges. Not discussed at/after LPC.
>>x I currently report reserved regions. Alex expressed the need to
>>  report the full usable IOVA range instead (x86 min-max range
>>  minus MSI APIC window). I think this is meaningful for ARM
>>  too where arm-smmu might not support the full 64b range.
>>x Any objection we report the usable IOVA regions instead?
> 
> The issue with that is that we can't actually report "the usable
> regions" at the moment, as that involves pulling together disjoint
> properties of arbitrary hardware unrelated to the IOMMU. We'd be
> reporting "the not-definitely-unusable regions, which may have some
> unusable holes in them still". That seems like an ABI nightmare - I'd
> still much rather say "here are some, but not necessarily all, regions
> you definitely can't use", because saying "here are some regions which
> you might be able to use most of, probably" is what we're already doing
> today, via a single implicit region from 0 to ULONG_MAX ;)
> 
> The address space limits are definitely useful to know, but I think it
> would be better to expose them separately to avoid the ambiguity. At
> worst, I guess it would be reasonable to express the limits via an
> "out-of-range" reserved region type for 0 to $base and $top to
> ULONG-MAX. To *safely* expose usable regions, we'd have to start out
> with a very conservative assumption (e.g. only IOVAs matching physical
> RAM), and only expand them once we're sure we can detect every possible
> bit of problematic hardware in the system - that's just too limiting to
> be useful. And if we expose something knowingly inaccurate, we risk
> having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
> nobody wants that...
Makes sense to me. "out-of-range reserved region type for 0 to $base and
$top to ULONG-MAX" can be an alternative to fulfill the requirement.
> 
>> 2) Shall the kernel check collision with MSI window* when userspace
>>calls VFIO_IOMMU_MAP_DMA?
>>Joerg/Will No; Alex yes
>>*for IOVA regions consumed downstream to the IOMMU: everyone says NO
> 
> If we're starting off by having the SMMU drivers expose it as a fake
> fixed region, I don't think we need to worry about this yet. We all seem
> to agree that as long as we communicate the fixed regions to userspace,
> it's then userspace's job to work around them. Let's come back to this
> one once we actually get to the point of dynamically sizing and
> allocating 'real' MSI remapping region(s).
> 
> Ultimately, the kernel *will* police collisions either way, because an
> underlying iommu_map() is going to fail if overlapping IOVAs are ever
> actually used, so it's really just a question of whether to have a more
> user-friendly failure mode.
That's true on ARM but not on x86 where the APIC MSI region is not
mapped I think.
> 
>> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>>My current series does not expose them in iommu group sysfs.
>>I understand we can expose the RMRR regions in the iomm group sysfs
>>without necessarily supporting RMRR requiring device assignment.
>>We can also add this support later.
> 
> As you say, reporting them doesn't necessitate allowing device
> assignment, and it's information which can already be easily grovelled
> out of dmesg (for intel-iommu at least) - there doesn't seem to be any
> need to hide them, but the x86 folks can have the final word on that.
agreed

Thanks

Eric
> 
> Robin.
> 
>> Thanks
>>
>> Eric
>>
>>
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>   the reserved regions
>>> - at the moment ARM MSI 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 09:36, Auger Eric wrote:
> Hi,
> 
> On 15/11/2016 14:09, Eric Auger wrote:
>> Following LPC discussions, we now report reserved regions through
>> iommu-group sysfs reserved_regions attribute file.
> 
> 
> While I am respinning this series into v4, here is a tentative summary
> of technical topics for which no consensus was reached at this point.
> 
> 1) Shall we report the usable IOVA range instead of reserved IOVA
>ranges. Not discussed at/after LPC.
>x I currently report reserved regions. Alex expressed the need to
>  report the full usable IOVA range instead (x86 min-max range
>  minus MSI APIC window). I think this is meaningful for ARM
>  too where arm-smmu might not support the full 64b range.
>x Any objection we report the usable IOVA regions instead?

The issue with that is that we can't actually report "the usable
regions" at the moment, as that involves pulling together disjoint
properties of arbitrary hardware unrelated to the IOMMU. We'd be
reporting "the not-definitely-unusable regions, which may have some
unusable holes in them still". That seems like an ABI nightmare - I'd
still much rather say "here are some, but not necessarily all, regions
you definitely can't use", because saying "here are some regions which
you might be able to use most of, probably" is what we're already doing
today, via a single implicit region from 0 to ULONG_MAX ;)

The address space limits are definitely useful to know, but I think it
would be better to expose them separately to avoid the ambiguity. At
worst, I guess it would be reasonable to express the limits via an
"out-of-range" reserved region type for 0 to $base and $top to
ULONG-MAX. To *safely* expose usable regions, we'd have to start out
with a very conservative assumption (e.g. only IOVAs matching physical
RAM), and only expand them once we're sure we can detect every possible
bit of problematic hardware in the system - that's just too limiting to
be useful. And if we expose something knowingly inaccurate, we risk
having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
nobody wants that...

> 2) Shall the kernel check collision with MSI window* when userspace
>calls VFIO_IOMMU_MAP_DMA?
>Joerg/Will No; Alex yes
>*for IOVA regions consumed downstream to the IOMMU: everyone says NO

If we're starting off by having the SMMU drivers expose it as a fake
fixed region, I don't think we need to worry about this yet. We all seem
to agree that as long as we communicate the fixed regions to userspace,
it's then userspace's job to work around them. Let's come back to this
one once we actually get to the point of dynamically sizing and
allocating 'real' MSI remapping region(s).

Ultimately, the kernel *will* police collisions either way, because an
underlying iommu_map() is going to fail if overlapping IOVAs are ever
actually used, so it's really just a question of whether to have a more
user-friendly failure mode.

> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>My current series does not expose them in iommu group sysfs.
>I understand we can expose the RMRR regions in the iomm group sysfs
>without necessarily supporting RMRR requiring device assignment.
>We can also add this support later.

As you say, reporting them doesn't necessitate allowing device
assignment, and it's information which can already be easily grovelled
out of dmesg (for intel-iommu at least) - there doesn't seem to be any
need to hide them, but the x86 folks can have the final word on that.

Robin.

> Thanks
> 
> Eric
> 
> 
>>
>> Reserved regions are populated through the IOMMU get_resv_region callback
>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>> arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>> 1MB large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 08/12/16 09:36, Auger Eric wrote:
> Hi,
> 
> On 15/11/2016 14:09, Eric Auger wrote:
>> Following LPC discussions, we now report reserved regions through
>> iommu-group sysfs reserved_regions attribute file.
> 
> 
> While I am respinning this series into v4, here is a tentative summary
> of technical topics for which no consensus was reached at this point.
> 
> 1) Shall we report the usable IOVA range instead of reserved IOVA
>ranges. Not discussed at/after LPC.
>x I currently report reserved regions. Alex expressed the need to
>  report the full usable IOVA range instead (x86 min-max range
>  minus MSI APIC window). I think this is meaningful for ARM
>  too where arm-smmu might not support the full 64b range.
>x Any objection we report the usable IOVA regions instead?

The issue with that is that we can't actually report "the usable
regions" at the moment, as that involves pulling together disjoint
properties of arbitrary hardware unrelated to the IOMMU. We'd be
reporting "the not-definitely-unusable regions, which may have some
unusable holes in them still". That seems like an ABI nightmare - I'd
still much rather say "here are some, but not necessarily all, regions
you definitely can't use", because saying "here are some regions which
you might be able to use most of, probably" is what we're already doing
today, via a single implicit region from 0 to ULONG_MAX ;)

The address space limits are definitely useful to know, but I think it
would be better to expose them separately to avoid the ambiguity. At
worst, I guess it would be reasonable to express the limits via an
"out-of-range" reserved region type for 0 to $base and $top to
ULONG-MAX. To *safely* expose usable regions, we'd have to start out
with a very conservative assumption (e.g. only IOVAs matching physical
RAM), and only expand them once we're sure we can detect every possible
bit of problematic hardware in the system - that's just too limiting to
be useful. And if we expose something knowingly inaccurate, we risk
having another "bogoMIPS in /proc/cpuinfo" ABI burden on our hands, and
nobody wants that...

> 2) Shall the kernel check collision with MSI window* when userspace
>calls VFIO_IOMMU_MAP_DMA?
>Joerg/Will No; Alex yes
>*for IOVA regions consumed downstream to the IOMMU: everyone says NO

If we're starting off by having the SMMU drivers expose it as a fake
fixed region, I don't think we need to worry about this yet. We all seem
to agree that as long as we communicate the fixed regions to userspace,
it's then userspace's job to work around them. Let's come back to this
one once we actually get to the point of dynamically sizing and
allocating 'real' MSI remapping region(s).

Ultimately, the kernel *will* police collisions either way, because an
underlying iommu_map() is going to fail if overlapping IOVAs are ever
actually used, so it's really just a question of whether to have a more
user-friendly failure mode.

> 3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
>My current series does not expose them in iommu group sysfs.
>I understand we can expose the RMRR regions in the iomm group sysfs
>without necessarily supporting RMRR requiring device assignment.
>We can also add this support later.

As you say, reporting them doesn't necessitate allowing device
assignment, and it's information which can already be easily grovelled
out of dmesg (for intel-iommu at least) - there doesn't seem to be any
need to hide them, but the x86 folks can have the final word on that.

Robin.

> Thanks
> 
> Eric
> 
> 
>>
>> Reserved regions are populated through the IOMMU get_resv_region callback
>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>> arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>> 1MB large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi,

On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.


While I am respinning this series into v4, here is a tentative summary
of technical topics for which no consensus was reached at this point.

1) Shall we report the usable IOVA range instead of reserved IOVA
   ranges. Not discussed at/after LPC.
   x I currently report reserved regions. Alex expressed the need to
 report the full usable IOVA range instead (x86 min-max range
 minus MSI APIC window). I think this is meaningful for ARM
 too where arm-smmu might not support the full 64b range.
   x Any objection we report the usable IOVA regions instead?

2) Shall the kernel check collision with MSI window* when userspace
   calls VFIO_IOMMU_MAP_DMA?
   Joerg/Will No; Alex yes
   *for IOVA regions consumed downstream to the IOMMU: everyone says NO

3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
   My current series does not expose them in iommu group sysfs.
   I understand we can expose the RMRR regions in the iomm group sysfs
   without necessarily supporting RMRR requiring device assignment.
   We can also add this support later.

Thanks

Eric


> 
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi,

On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.


While I am respinning this series into v4, here is a tentative summary
of technical topics for which no consensus was reached at this point.

1) Shall we report the usable IOVA range instead of reserved IOVA
   ranges. Not discussed at/after LPC.
   x I currently report reserved regions. Alex expressed the need to
 report the full usable IOVA range instead (x86 min-max range
 minus MSI APIC window). I think this is meaningful for ARM
 too where arm-smmu might not support the full 64b range.
   x Any objection we report the usable IOVA regions instead?

2) Shall the kernel check collision with MSI window* when userspace
   calls VFIO_IOMMU_MAP_DMA?
   Joerg/Will No; Alex yes
   *for IOVA regions consumed downstream to the IOMMU: everyone says NO

3) RMRR reporting in the iommu group sysfs? Joerg: yes; Don: no
   My current series does not expose them in iommu group sysfs.
   I understand we can expose the RMRR regions in the iomm group sysfs
   without necessarily supporting RMRR requiring device assignment.
   We can also add this support later.

Thanks

Eric


> 
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Shanker,

On 07/12/2016 19:52, Shanker Donthineni wrote:
> Hi Eric,
> 
> Is there any reason why you are not supporting SMMUv3 driver? Qualcomm
> hardware doesn't not support SMMUv2 hardware, please add support for
> SMMUv3 in next patch set. I've ported ' RFC,v3,09/10] iommu/arm-smmu:
> Implement reserved region get/put callbacks' to SMMUv3 driver and tested
> device-pass-through feature on Qualcomm server platform without any issue.
> 
> Tested-by: Shanker Donthineni 
Thanks!

No reason behind not supporting smmuv3 except I don't have any HW to test.

I will add this support in next version.

Thanks

Eric

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Shanker,

On 07/12/2016 19:52, Shanker Donthineni wrote:
> Hi Eric,
> 
> Is there any reason why you are not supporting SMMUv3 driver? Qualcomm
> hardware doesn't not support SMMUv2 hardware, please add support for
> SMMUv3 in next patch set. I've ported ' RFC,v3,09/10] iommu/arm-smmu:
> Implement reserved region get/put callbacks' to SMMUv3 driver and tested
> device-pass-through feature on Qualcomm server platform without any issue.
> 
> Tested-by: Shanker Donthineni 
Thanks!

No reason behind not supporting smmuv3 except I don't have any HW to test.

I will add this support in next version.

Thanks

Eric

RE: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Bharat Bhushan]

Hi Eric,

I have tested this series on NXP platform.

Thanks
-Bharat

> -Original Message-
> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
> Sent: Tuesday, November 15, 2016 6:39 PM
> To: eric.au...@redhat.com; eric.auger@gmail.com;
> christoffer.d...@linaro.org; marc.zyng...@arm.com;
> robin.mur...@arm.com; alex.william...@redhat.com;
> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
> pranav.sawargaon...@gmail.com
> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
> IOVA reserved regions
> 
> Following LPC discussions, we now report reserved regions through iommu-
> group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region
> callback (former get_dm_regions), now implemented by amd-iommu, intel-
> iommu and arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
> large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++
> ---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 
> --
> 1.9.1
> 
> ___
> iommu mailing list
> io...@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Bharat Bhushan]

Hi Eric,

I have tested this series on NXP platform.

Thanks
-Bharat

> -Original Message-
> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
> Sent: Tuesday, November 15, 2016 6:39 PM
> To: eric.au...@redhat.com; eric.auger@gmail.com;
> christoffer.d...@linaro.org; marc.zyng...@arm.com;
> robin.mur...@arm.com; alex.william...@redhat.com;
> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
> pranav.sawargaon...@gmail.com
> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
> IOVA reserved regions
> 
> Following LPC discussions, we now report reserved regions through iommu-
> group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region
> callback (former get_dm_regions), now implemented by amd-iommu, intel-
> iommu and arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
> large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++
> ---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 
> --
> 1.9.1
> 
> ___
> iommu mailing list
> io...@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Shanker Donthineni]

Hi Eric,

Is there any reason why you are not supporting SMMUv3 driver? Qualcomm 
hardware doesn't not support SMMUv2 hardware, please add support for 
SMMUv3 in next patch set. I've ported ' RFC,v3,09/10] iommu/arm-smmu: 
Implement reserved region get/put callbacks' to SMMUv3 driver and tested 
device-pass-through feature on Qualcomm server platform without any issue.


Tested-by: Shanker Donthineni 

--
Shanker Donthineni
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm 
Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux 
Foundation Collaborative Project.

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Shanker Donthineni]

Hi Eric,

Is there any reason why you are not supporting SMMUv3 driver? Qualcomm 
hardware doesn't not support SMMUv2 hardware, please add support for 
SMMUv3 in next patch set. I've ported ' RFC,v3,09/10] iommu/arm-smmu: 
Implement reserved region get/put callbacks' to SMMUv3 driver and tested 
device-pass-through feature on Qualcomm server platform without any issue.


Tested-by: Shanker Donthineni 

--
Shanker Donthineni
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm 
Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux 
Foundation Collaborative Project.

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 30/11/16 14:08, Auger Eric wrote:
> Hi Will,
> 
> On 30/11/2016 11:37, Will Deacon wrote:
>> On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window (arbitrarily located at 0x800 and
 1MB large) and the PCI host bridge windows.

 The series integrates a not officially posted patch from Robin:
 "iommu/dma: Allow MSI-only cookies".

 This series currently does not address IRQ safety assessment.
>>>
>>> I will respin this series taking into account Joerg's comment. Does
>>> anyone have additional comments or want to put forward some conceptual
>>> issues with the current direction and with this implementation?
>>>
>>> As for the IRQ safety assessment, in a first step I would propose to
>>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>>> assignment as unsafe. Any objection?
>>
>> Well, yeah, because it's perfectly safe with GICv3.
> 
> Well except if you have an MSI controller in-between the device and the
> sMMU (typically embedded in the host bridge). Detecting this situation
> is not straightforward; hence my proposal.

That's not the GICv3 (ITS) case, though, and either way it's irrelevant
to the "safety" aspect in question; the fact that writes to the ITS
carry sideband signals which disambiguate and isolate MSIs has nothing
to do with whether that write undergoes address translation along the way.

It's also not legacy INTx, and the fact that we have to pretend the SMMU
provides MSI isolation in order to make things work with INTx is an
entirely separate piece of brokenness I've raised several times before.
I more than anyone would love to remove IOMMU_CAP_INTR_REMAP from the
SMMU drivers yesterday, but doing so breaks the existing use-case on ARM
unless we actually fix that aspect of VFIO first (I did look into it
once, but it seemed way beyond my comprehension at the time).

Robin.

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 30/11/16 14:08, Auger Eric wrote:
> Hi Will,
> 
> On 30/11/2016 11:37, Will Deacon wrote:
>> On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window (arbitrarily located at 0x800 and
 1MB large) and the PCI host bridge windows.

 The series integrates a not officially posted patch from Robin:
 "iommu/dma: Allow MSI-only cookies".

 This series currently does not address IRQ safety assessment.
>>>
>>> I will respin this series taking into account Joerg's comment. Does
>>> anyone have additional comments or want to put forward some conceptual
>>> issues with the current direction and with this implementation?
>>>
>>> As for the IRQ safety assessment, in a first step I would propose to
>>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>>> assignment as unsafe. Any objection?
>>
>> Well, yeah, because it's perfectly safe with GICv3.
> 
> Well except if you have an MSI controller in-between the device and the
> sMMU (typically embedded in the host bridge). Detecting this situation
> is not straightforward; hence my proposal.

That's not the GICv3 (ITS) case, though, and either way it's irrelevant
to the "safety" aspect in question; the fact that writes to the ITS
carry sideband signals which disambiguate and isolate MSIs has nothing
to do with whether that write undergoes address translation along the way.

It's also not legacy INTx, and the fact that we have to pretend the SMMU
provides MSI isolation in order to make things work with INTx is an
entirely separate piece of brokenness I've raised several times before.
I more than anyone would love to remove IOMMU_CAP_INTR_REMAP from the
SMMU drivers yesterday, but doing so breaks the existing use-case on ARM
unless we actually fix that aspect of VFIO first (I did look into it
once, but it seemed way beyond my comprehension at the time).

Robin.

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Will,

On 30/11/2016 11:37, Will Deacon wrote:
> On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>
>> I will respin this series taking into account Joerg's comment. Does
>> anyone have additional comments or want to put forward some conceptual
>> issues with the current direction and with this implementation?
>>
>> As for the IRQ safety assessment, in a first step I would propose to
>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>> assignment as unsafe. Any objection?
> 
> Well, yeah, because it's perfectly safe with GICv3.

Well except if you have an MSI controller in-between the device and the
sMMU (typically embedded in the host bridge). Detecting this situation
is not straightforward; hence my proposal.

Thanks

Eric
> 
> Will
> 
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Will,

On 30/11/2016 11:37, Will Deacon wrote:
> On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>
>> I will respin this series taking into account Joerg's comment. Does
>> anyone have additional comments or want to put forward some conceptual
>> issues with the current direction and with this implementation?
>>
>> As for the IRQ safety assessment, in a first step I would propose to
>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>> assignment as unsafe. Any objection?
> 
> Well, yeah, because it's perfectly safe with GICv3.

Well except if you have an MSI controller in-between the device and the
sMMU (typically embedded in the host bridge). Detecting this situation
is not straightforward; hence my proposal.

Thanks

Eric
> 
> Will
> 
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 30/11/16 10:52, Ganapatrao Kulkarni wrote:
> On Wed, Nov 30, 2016 at 3:44 PM, Auger Eric  wrote:
>> Hi Ganapat,
>>
>> On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
>>> Hi Eric,
>>>
>>> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
>>> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
>>> is this patch part of some other series?
>>
>> Actually this is a very old patch from Alex aimed at working around lack
>> of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513
>>
> 
> i have tried this patchset on thunderx-83xx for vfio and it works for me!
> i was wondering is this patch required? i guess not.

If your system and devices actually support and properly advertise ACS
then there's nothing to override. Conversely, if you're happy assigning
everything behind a single RC to the same guest then ACS doesn't really
matter. It's only the in-between case - when the host still wants to
keep control of one or more devices, but they all get grouped together
due to lack of ACS - that warrants working around.

Robin.

> 
> please cc me when you respin this patchset.
> 
> thanks
> Ganapat
> 
>> Thanks
>>
>> Eric
>>>
>>> thanks
>>> Ganapat
>>>
>>> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
 Hi,

 On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.
>
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
>
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
>
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
>
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
>
> This series currently does not address IRQ safety assessment.

 I will respin this series taking into account Joerg's comment. Does
 anyone have additional comments or want to put forward some conceptual
 issues with the current direction and with this implementation?

 As for the IRQ safety assessment, in a first step I would propose to
 remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
 assignment as unsafe. Any objection?

 Thanks

 Eric


> Best Regards
>
> Eric
>
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
>
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
>
>
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
>
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
>

 ___
 linux-arm-kernel mailing list
 linux-arm-ker...@lists.infradead.org
 http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Robin Murphy]

On 30/11/16 10:52, Ganapatrao Kulkarni wrote:
> On Wed, Nov 30, 2016 at 3:44 PM, Auger Eric  wrote:
>> Hi Ganapat,
>>
>> On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
>>> Hi Eric,
>>>
>>> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
>>> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
>>> is this patch part of some other series?
>>
>> Actually this is a very old patch from Alex aimed at working around lack
>> of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513
>>
> 
> i have tried this patchset on thunderx-83xx for vfio and it works for me!
> i was wondering is this patch required? i guess not.

If your system and devices actually support and properly advertise ACS
then there's nothing to override. Conversely, if you're happy assigning
everything behind a single RC to the same guest then ACS doesn't really
matter. It's only the in-between case - when the host still wants to
keep control of one or more devices, but they all get grouped together
due to lack of ACS - that warrants working around.

Robin.

> 
> please cc me when you respin this patchset.
> 
> thanks
> Ganapat
> 
>> Thanks
>>
>> Eric
>>>
>>> thanks
>>> Ganapat
>>>
>>> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
 Hi,

 On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.
>
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
>
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
>
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
>
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
>
> This series currently does not address IRQ safety assessment.

 I will respin this series taking into account Joerg's comment. Does
 anyone have additional comments or want to put forward some conceptual
 issues with the current direction and with this implementation?

 As for the IRQ safety assessment, in a first step I would propose to
 remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
 assignment as unsafe. Any objection?

 Thanks

 Eric


> Best Regards
>
> Eric
>
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
>
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
>
>
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
>
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
>

 ___
 linux-arm-kernel mailing list
 linux-arm-ker...@lists.infradead.org
 http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Ganapatrao Kulkarni]

On Wed, Nov 30, 2016 at 3:44 PM, Auger Eric  wrote:
> Hi Ganapat,
>
> On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
>> Hi Eric,
>>
>> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
>> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
>> is this patch part of some other series?
>
> Actually this is a very old patch from Alex aimed at working around lack
> of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513
>

i have tried this patchset on thunderx-83xx for vfio and it works for me!
i was wondering is this patch required? i guess not.

please cc me when you respin this patchset.

thanks
Ganapat

> Thanks
>
> Eric
>>
>> thanks
>> Ganapat
>>
>> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
>>> Hi,
>>>
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window (arbitrarily located at 0x800 and
 1MB large) and the PCI host bridge windows.

 The series integrates a not officially posted patch from Robin:
 "iommu/dma: Allow MSI-only cookies".

 This series currently does not address IRQ safety assessment.
>>>
>>> I will respin this series taking into account Joerg's comment. Does
>>> anyone have additional comments or want to put forward some conceptual
>>> issues with the current direction and with this implementation?
>>>
>>> As for the IRQ safety assessment, in a first step I would propose to
>>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>>> assignment as unsafe. Any objection?
>>>
>>> Thanks
>>>
>>> Eric
>>>
>>>
 Best Regards

 Eric

 Git: complete series available at
 https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3

 History:
 RFC v2 -> v3:
 - switch to an iommu-group sysfs API
 - use new dummy allocator provided by Robin
 - dummy allocator initialized by vfio-iommu-type1 after enumerating
   the reserved regions
 - at the moment ARM MSI base address/size is left unchanged compared
   to v2
 - we currently report reserved regions and not usable IOVA regions as
   requested by Alex

 RFC v1 -> v2:
 - fix intel_add_reserved_regions
 - add mutex lock/unlock in vfio_iommu_type1


 Eric Auger (10):
   iommu/dma: Allow MSI-only cookies
   iommu: Rename iommu_dm_regions into iommu_resv_regions
   iommu: Add new reserved IOMMU attributes
   iommu: iommu_alloc_resv_region
   iommu: Do not map reserved regions
   iommu: iommu_get_group_resv_regions
   iommu: Implement reserved_regions iommu-group sysfs file
   iommu/vt-d: Implement reserved region get/put callbacks
   iommu/arm-smmu: Implement reserved region get/put callbacks
   vfio/type1: Get MSI cookie

  drivers/iommu/amd_iommu.c   |  20 +++---
  drivers/iommu/arm-smmu.c|  52 +++
  drivers/iommu/dma-iommu.c   | 116 ++---
  drivers/iommu/intel-iommu.c |  50 ++
  drivers/iommu/iommu.c   | 141 
 
  drivers/vfio/vfio_iommu_type1.c |  26 
  include/linux/dma-iommu.h   |   7 ++
  include/linux/iommu.h   |  49 ++
  8 files changed, 391 insertions(+), 70 deletions(-)

>>>
>>> ___
>>> linux-arm-kernel mailing list
>>> linux-arm-ker...@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Ganapatrao Kulkarni]

On Wed, Nov 30, 2016 at 3:44 PM, Auger Eric  wrote:
> Hi Ganapat,
>
> On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
>> Hi Eric,
>>
>> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
>> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
>> is this patch part of some other series?
>
> Actually this is a very old patch from Alex aimed at working around lack
> of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513
>

i have tried this patchset on thunderx-83xx for vfio and it works for me!
i was wondering is this patch required? i guess not.

please cc me when you respin this patchset.

thanks
Ganapat

> Thanks
>
> Eric
>>
>> thanks
>> Ganapat
>>
>> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
>>> Hi,
>>>
>>> On 15/11/2016 14:09, Eric Auger wrote:
 Following LPC discussions, we now report reserved regions through
 iommu-group sysfs reserved_regions attribute file.

 Reserved regions are populated through the IOMMU get_resv_region callback
 (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
 arm-smmu.

 The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
 IOMMU_RESV_NOMAP reserved region.

 arm-smmu reports the MSI window (arbitrarily located at 0x800 and
 1MB large) and the PCI host bridge windows.

 The series integrates a not officially posted patch from Robin:
 "iommu/dma: Allow MSI-only cookies".

 This series currently does not address IRQ safety assessment.
>>>
>>> I will respin this series taking into account Joerg's comment. Does
>>> anyone have additional comments or want to put forward some conceptual
>>> issues with the current direction and with this implementation?
>>>
>>> As for the IRQ safety assessment, in a first step I would propose to
>>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>>> assignment as unsafe. Any objection?
>>>
>>> Thanks
>>>
>>> Eric
>>>
>>>
 Best Regards

 Eric

 Git: complete series available at
 https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3

 History:
 RFC v2 -> v3:
 - switch to an iommu-group sysfs API
 - use new dummy allocator provided by Robin
 - dummy allocator initialized by vfio-iommu-type1 after enumerating
   the reserved regions
 - at the moment ARM MSI base address/size is left unchanged compared
   to v2
 - we currently report reserved regions and not usable IOVA regions as
   requested by Alex

 RFC v1 -> v2:
 - fix intel_add_reserved_regions
 - add mutex lock/unlock in vfio_iommu_type1


 Eric Auger (10):
   iommu/dma: Allow MSI-only cookies
   iommu: Rename iommu_dm_regions into iommu_resv_regions
   iommu: Add new reserved IOMMU attributes
   iommu: iommu_alloc_resv_region
   iommu: Do not map reserved regions
   iommu: iommu_get_group_resv_regions
   iommu: Implement reserved_regions iommu-group sysfs file
   iommu/vt-d: Implement reserved region get/put callbacks
   iommu/arm-smmu: Implement reserved region get/put callbacks
   vfio/type1: Get MSI cookie

  drivers/iommu/amd_iommu.c   |  20 +++---
  drivers/iommu/arm-smmu.c|  52 +++
  drivers/iommu/dma-iommu.c   | 116 ++---
  drivers/iommu/intel-iommu.c |  50 ++
  drivers/iommu/iommu.c   | 141 
 
  drivers/vfio/vfio_iommu_type1.c |  26 
  include/linux/dma-iommu.h   |   7 ++
  include/linux/iommu.h   |  49 ++
  8 files changed, 391 insertions(+), 70 deletions(-)

>>>
>>> ___
>>> linux-arm-kernel mailing list
>>> linux-arm-ker...@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Will Deacon]

On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
> On 15/11/2016 14:09, Eric Auger wrote:
> > Following LPC discussions, we now report reserved regions through
> > iommu-group sysfs reserved_regions attribute file.
> > 
> > Reserved regions are populated through the IOMMU get_resv_region callback
> > (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> > arm-smmu.
> > 
> > The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> > IOMMU_RESV_NOMAP reserved region.
> > 
> > arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> > 1MB large) and the PCI host bridge windows.
> > 
> > The series integrates a not officially posted patch from Robin:
> > "iommu/dma: Allow MSI-only cookies".
> > 
> > This series currently does not address IRQ safety assessment.
> 
> I will respin this series taking into account Joerg's comment. Does
> anyone have additional comments or want to put forward some conceptual
> issues with the current direction and with this implementation?
> 
> As for the IRQ safety assessment, in a first step I would propose to
> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
> assignment as unsafe. Any objection?

Well, yeah, because it's perfectly safe with GICv3.

Will

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Will Deacon]

On Wed, Nov 30, 2016 at 10:49:33AM +0100, Auger Eric wrote:
> On 15/11/2016 14:09, Eric Auger wrote:
> > Following LPC discussions, we now report reserved regions through
> > iommu-group sysfs reserved_regions attribute file.
> > 
> > Reserved regions are populated through the IOMMU get_resv_region callback
> > (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> > arm-smmu.
> > 
> > The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> > IOMMU_RESV_NOMAP reserved region.
> > 
> > arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> > 1MB large) and the PCI host bridge windows.
> > 
> > The series integrates a not officially posted patch from Robin:
> > "iommu/dma: Allow MSI-only cookies".
> > 
> > This series currently does not address IRQ safety assessment.
> 
> I will respin this series taking into account Joerg's comment. Does
> anyone have additional comments or want to put forward some conceptual
> issues with the current direction and with this implementation?
> 
> As for the IRQ safety assessment, in a first step I would propose to
> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
> assignment as unsafe. Any objection?

Well, yeah, because it's perfectly safe with GICv3.

Will

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Ganapat,

On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
> Hi Eric,
> 
> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
> is this patch part of some other series?

Actually this is a very old patch from Alex aimed at working around lack
of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513

Thanks

Eric
> 
> thanks
> Ganapat
> 
> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>
>> I will respin this series taking into account Joerg's comment. Does
>> anyone have additional comments or want to put forward some conceptual
>> issues with the current direction and with this implementation?
>>
>> As for the IRQ safety assessment, in a first step I would propose to
>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>> assignment as unsafe. Any objection?
>>
>> Thanks
>>
>> Eric
>>
>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>   the reserved regions
>>> - at the moment ARM MSI base address/size is left unchanged compared
>>>   to v2
>>> - we currently report reserved regions and not usable IOVA regions as
>>>   requested by Alex
>>>
>>> RFC v1 -> v2:
>>> - fix intel_add_reserved_regions
>>> - add mutex lock/unlock in vfio_iommu_type1
>>>
>>>
>>> Eric Auger (10):
>>>   iommu/dma: Allow MSI-only cookies
>>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>>   iommu: Add new reserved IOMMU attributes
>>>   iommu: iommu_alloc_resv_region
>>>   iommu: Do not map reserved regions
>>>   iommu: iommu_get_group_resv_regions
>>>   iommu: Implement reserved_regions iommu-group sysfs file
>>>   iommu/vt-d: Implement reserved region get/put callbacks
>>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>>   vfio/type1: Get MSI cookie
>>>
>>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>>  drivers/iommu/arm-smmu.c|  52 +++
>>>  drivers/iommu/dma-iommu.c   | 116 ++---
>>>  drivers/iommu/intel-iommu.c |  50 ++
>>>  drivers/iommu/iommu.c   | 141 
>>> 
>>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>>  include/linux/dma-iommu.h   |   7 ++
>>>  include/linux/iommu.h   |  49 ++
>>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>>
>>
>> ___
>> linux-arm-kernel mailing list
>> linux-arm-ker...@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Ganapat,

On 30/11/2016 11:04, Ganapatrao Kulkarni wrote:
> Hi Eric,
> 
> in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
> there is 11th patch "pci: Enable overrides for missing ACS capabilities"
> is this patch part of some other series?

Actually this is a very old patch from Alex aimed at working around lack
of PCIe ACS support: https://lkml.org/lkml/2013/5/30/513

Thanks

Eric
> 
> thanks
> Ganapat
> 
> On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
>> Hi,
>>
>> On 15/11/2016 14:09, Eric Auger wrote:
>>> Following LPC discussions, we now report reserved regions through
>>> iommu-group sysfs reserved_regions attribute file.
>>>
>>> Reserved regions are populated through the IOMMU get_resv_region callback
>>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>>> arm-smmu.
>>>
>>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>>> IOMMU_RESV_NOMAP reserved region.
>>>
>>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>>> 1MB large) and the PCI host bridge windows.
>>>
>>> The series integrates a not officially posted patch from Robin:
>>> "iommu/dma: Allow MSI-only cookies".
>>>
>>> This series currently does not address IRQ safety assessment.
>>
>> I will respin this series taking into account Joerg's comment. Does
>> anyone have additional comments or want to put forward some conceptual
>> issues with the current direction and with this implementation?
>>
>> As for the IRQ safety assessment, in a first step I would propose to
>> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
>> assignment as unsafe. Any objection?
>>
>> Thanks
>>
>> Eric
>>
>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> Git: complete series available at
>>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>>
>>> History:
>>> RFC v2 -> v3:
>>> - switch to an iommu-group sysfs API
>>> - use new dummy allocator provided by Robin
>>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>>   the reserved regions
>>> - at the moment ARM MSI base address/size is left unchanged compared
>>>   to v2
>>> - we currently report reserved regions and not usable IOVA regions as
>>>   requested by Alex
>>>
>>> RFC v1 -> v2:
>>> - fix intel_add_reserved_regions
>>> - add mutex lock/unlock in vfio_iommu_type1
>>>
>>>
>>> Eric Auger (10):
>>>   iommu/dma: Allow MSI-only cookies
>>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>>   iommu: Add new reserved IOMMU attributes
>>>   iommu: iommu_alloc_resv_region
>>>   iommu: Do not map reserved regions
>>>   iommu: iommu_get_group_resv_regions
>>>   iommu: Implement reserved_regions iommu-group sysfs file
>>>   iommu/vt-d: Implement reserved region get/put callbacks
>>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>>   vfio/type1: Get MSI cookie
>>>
>>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>>  drivers/iommu/arm-smmu.c|  52 +++
>>>  drivers/iommu/dma-iommu.c   | 116 ++---
>>>  drivers/iommu/intel-iommu.c |  50 ++
>>>  drivers/iommu/iommu.c   | 141 
>>> 
>>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>>  include/linux/dma-iommu.h   |   7 ++
>>>  include/linux/iommu.h   |  49 ++
>>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>>
>>
>> ___
>> linux-arm-kernel mailing list
>> linux-arm-ker...@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Ganapatrao Kulkarni]

Hi Eric,

in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
there is 11th patch "pci: Enable overrides for missing ACS capabilities"
is this patch part of some other series?

thanks
Ganapat

On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
> Hi,
>
> On 15/11/2016 14:09, Eric Auger wrote:
>> Following LPC discussions, we now report reserved regions through
>> iommu-group sysfs reserved_regions attribute file.
>>
>> Reserved regions are populated through the IOMMU get_resv_region callback
>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>> arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>> 1MB large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>
> I will respin this series taking into account Joerg's comment. Does
> anyone have additional comments or want to put forward some conceptual
> issues with the current direction and with this implementation?
>
> As for the IRQ safety assessment, in a first step I would propose to
> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
> assignment as unsafe. Any objection?
>
> Thanks
>
> Eric
>
>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: iommu_alloc_resv_region
>>   iommu: Do not map reserved regions
>>   iommu: iommu_get_group_resv_regions
>>   iommu: Implement reserved_regions iommu-group sysfs file
>>   iommu/vt-d: Implement reserved region get/put callbacks
>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>   vfio/type1: Get MSI cookie
>>
>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>  drivers/iommu/arm-smmu.c|  52 +++
>>  drivers/iommu/dma-iommu.c   | 116 ++---
>>  drivers/iommu/intel-iommu.c |  50 ++
>>  drivers/iommu/iommu.c   | 141 
>> 
>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>  include/linux/dma-iommu.h   |   7 ++
>>  include/linux/iommu.h   |  49 ++
>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>
>
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Ganapatrao Kulkarni]

Hi Eric,

in you repo "https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3;
there is 11th patch "pci: Enable overrides for missing ACS capabilities"
is this patch part of some other series?

thanks
Ganapat

On Wed, Nov 30, 2016 at 3:19 PM, Auger Eric  wrote:
> Hi,
>
> On 15/11/2016 14:09, Eric Auger wrote:
>> Following LPC discussions, we now report reserved regions through
>> iommu-group sysfs reserved_regions attribute file.
>>
>> Reserved regions are populated through the IOMMU get_resv_region callback
>> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
>> arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
>> 1MB large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>
> I will respin this series taking into account Joerg's comment. Does
> anyone have additional comments or want to put forward some conceptual
> issues with the current direction and with this implementation?
>
> As for the IRQ safety assessment, in a first step I would propose to
> remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
> assignment as unsafe. Any objection?
>
> Thanks
>
> Eric
>
>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: iommu_alloc_resv_region
>>   iommu: Do not map reserved regions
>>   iommu: iommu_get_group_resv_regions
>>   iommu: Implement reserved_regions iommu-group sysfs file
>>   iommu/vt-d: Implement reserved region get/put callbacks
>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>   vfio/type1: Get MSI cookie
>>
>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>  drivers/iommu/arm-smmu.c|  52 +++
>>  drivers/iommu/dma-iommu.c   | 116 ++---
>>  drivers/iommu/intel-iommu.c |  50 ++
>>  drivers/iommu/iommu.c   | 141 
>> 
>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>  include/linux/dma-iommu.h   |   7 ++
>>  include/linux/iommu.h   |  49 ++
>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>
>
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi,

On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.

I will respin this series taking into account Joerg's comment. Does
anyone have additional comments or want to put forward some conceptual
issues with the current direction and with this implementation?

As for the IRQ safety assessment, in a first step I would propose to
remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
assignment as unsafe. Any objection?

Thanks

Eric


> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi,

On 15/11/2016 14:09, Eric Auger wrote:
> Following LPC discussions, we now report reserved regions through
> iommu-group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region callback
> (former get_dm_regions), now implemented by amd-iommu, intel-iommu and
> arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and
> 1MB large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.

I will respin this series taking into account Joerg's comment. Does
anyone have additional comments or want to put forward some conceptual
issues with the current direction and with this implementation?

As for the IRQ safety assessment, in a first step I would propose to
remove the IOMMU_CAP_INTR_REMAP from arm-smmus and consider the
assignment as unsafe. Any objection?

Thanks

Eric


> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141 
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 

RE: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Bharat Bhushan]

Hi Eric,

Have you sent out QEMU side patches based on this new approach? In case I 
missed please point me the patches? 

Thanks
-Bharat

> -Original Message-
> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
> Sent: Tuesday, November 15, 2016 6:39 PM
> To: eric.au...@redhat.com; eric.auger@gmail.com;
> christoffer.d...@linaro.org; marc.zyng...@arm.com;
> robin.mur...@arm.com; alex.william...@redhat.com;
> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
> pranav.sawargaon...@gmail.com
> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
> IOVA reserved regions
> 
> Following LPC discussions, we now report reserved regions through iommu-
> group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region
> callback (former get_dm_regions), now implemented by amd-iommu, intel-
> iommu and arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
> large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++
> ---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 
> --
> 1.9.1
> 
> ___
> iommu mailing list
> io...@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Bharat Bhushan]

Hi Eric,

Have you sent out QEMU side patches based on this new approach? In case I 
missed please point me the patches? 

Thanks
-Bharat

> -Original Message-
> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
> Sent: Tuesday, November 15, 2016 6:39 PM
> To: eric.au...@redhat.com; eric.auger@gmail.com;
> christoffer.d...@linaro.org; marc.zyng...@arm.com;
> robin.mur...@arm.com; alex.william...@redhat.com;
> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
> pranav.sawargaon...@gmail.com
> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
> IOVA reserved regions
> 
> Following LPC discussions, we now report reserved regions through iommu-
> group sysfs reserved_regions attribute file.
> 
> Reserved regions are populated through the IOMMU get_resv_region
> callback (former get_dm_regions), now implemented by amd-iommu, intel-
> iommu and arm-smmu.
> 
> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
> IOMMU_RESV_NOMAP reserved region.
> 
> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
> large) and the PCI host bridge windows.
> 
> The series integrates a not officially posted patch from Robin:
> "iommu/dma: Allow MSI-only cookies".
> 
> This series currently does not address IRQ safety assessment.
> 
> Best Regards
> 
> Eric
> 
> Git: complete series available at
> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
> 
> History:
> RFC v2 -> v3:
> - switch to an iommu-group sysfs API
> - use new dummy allocator provided by Robin
> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>   the reserved regions
> - at the moment ARM MSI base address/size is left unchanged compared
>   to v2
> - we currently report reserved regions and not usable IOVA regions as
>   requested by Alex
> 
> RFC v1 -> v2:
> - fix intel_add_reserved_regions
> - add mutex lock/unlock in vfio_iommu_type1
> 
> 
> Eric Auger (10):
>   iommu/dma: Allow MSI-only cookies
>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>   iommu: Add new reserved IOMMU attributes
>   iommu: iommu_alloc_resv_region
>   iommu: Do not map reserved regions
>   iommu: iommu_get_group_resv_regions
>   iommu: Implement reserved_regions iommu-group sysfs file
>   iommu/vt-d: Implement reserved region get/put callbacks
>   iommu/arm-smmu: Implement reserved region get/put callbacks
>   vfio/type1: Get MSI cookie
> 
>  drivers/iommu/amd_iommu.c   |  20 +++---
>  drivers/iommu/arm-smmu.c|  52 +++
>  drivers/iommu/dma-iommu.c   | 116 ++
> ---
>  drivers/iommu/intel-iommu.c |  50 ++
>  drivers/iommu/iommu.c   | 141
> 
>  drivers/vfio/vfio_iommu_type1.c |  26 
>  include/linux/dma-iommu.h   |   7 ++
>  include/linux/iommu.h   |  49 ++
>  8 files changed, 391 insertions(+), 70 deletions(-)
> 
> --
> 1.9.1
> 
> ___
> iommu mailing list
> io...@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Bharat,

On 18/11/2016 06:34, Bharat Bhushan wrote:
> Hi Eric,
> 
> Have you sent out QEMU side patches based on this new approach? In case I 
> missed please point me the patches? 
Upstream QEMU works fine for PCIe/MSI passthrough on ARM since mach virt
address space does not collide with fixed MSI region.

Thanks

Eric
> 
> Thanks
> -Bharat
> 
>> -Original Message-
>> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
>> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
>> Sent: Tuesday, November 15, 2016 6:39 PM
>> To: eric.au...@redhat.com; eric.auger@gmail.com;
>> christoffer.d...@linaro.org; marc.zyng...@arm.com;
>> robin.mur...@arm.com; alex.william...@redhat.com;
>> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
>> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
>> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
>> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
>> pranav.sawargaon...@gmail.com
>> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
>> IOVA reserved regions
>>
>> Following LPC discussions, we now report reserved regions through iommu-
>> group sysfs reserved_regions attribute file.
>>
>> Reserved regions are populated through the IOMMU get_resv_region
>> callback (former get_dm_regions), now implemented by amd-iommu, intel-
>> iommu and arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
>> large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: iommu_alloc_resv_region
>>   iommu: Do not map reserved regions
>>   iommu: iommu_get_group_resv_regions
>>   iommu: Implement reserved_regions iommu-group sysfs file
>>   iommu/vt-d: Implement reserved region get/put callbacks
>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>   vfio/type1: Get MSI cookie
>>
>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>  drivers/iommu/arm-smmu.c|  52 +++
>>  drivers/iommu/dma-iommu.c   | 116 ++
>> ---
>>  drivers/iommu/intel-iommu.c |  50 ++
>>  drivers/iommu/iommu.c   | 141
>> 
>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>  include/linux/dma-iommu.h   |   7 ++
>>  include/linux/iommu.h   |  49 ++
>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>
>> --
>> 1.9.1
>>
>> ___
>> iommu mailing list
>> io...@lists.linux-foundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
> 
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and IOVA reserved regions

[Auger Eric]

Hi Bharat,

On 18/11/2016 06:34, Bharat Bhushan wrote:
> Hi Eric,
> 
> Have you sent out QEMU side patches based on this new approach? In case I 
> missed please point me the patches? 
Upstream QEMU works fine for PCIe/MSI passthrough on ARM since mach virt
address space does not collide with fixed MSI region.

Thanks

Eric
> 
> Thanks
> -Bharat
> 
>> -Original Message-
>> From: iommu-boun...@lists.linux-foundation.org [mailto:iommu-
>> boun...@lists.linux-foundation.org] On Behalf Of Eric Auger
>> Sent: Tuesday, November 15, 2016 6:39 PM
>> To: eric.au...@redhat.com; eric.auger@gmail.com;
>> christoffer.d...@linaro.org; marc.zyng...@arm.com;
>> robin.mur...@arm.com; alex.william...@redhat.com;
>> will.dea...@arm.com; j...@8bytes.org; t...@linutronix.de;
>> ja...@lakedaemon.net; linux-arm-ker...@lists.infradead.org
>> Cc: drjo...@redhat.com; k...@vger.kernel.org; punit.agra...@arm.com;
>> linux-kernel@vger.kernel.org; io...@lists.linux-foundation.org;
>> pranav.sawargaon...@gmail.com
>> Subject: [RFC v3 00/10] KVM PCIe/MSI passthrough on ARM/ARM64 and
>> IOVA reserved regions
>>
>> Following LPC discussions, we now report reserved regions through iommu-
>> group sysfs reserved_regions attribute file.
>>
>> Reserved regions are populated through the IOMMU get_resv_region
>> callback (former get_dm_regions), now implemented by amd-iommu, intel-
>> iommu and arm-smmu.
>>
>> The intel-iommu reports the [FEE0_h - FEF0_000h] MSI window as an
>> IOMMU_RESV_NOMAP reserved region.
>>
>> arm-smmu reports the MSI window (arbitrarily located at 0x800 and 1MB
>> large) and the PCI host bridge windows.
>>
>> The series integrates a not officially posted patch from Robin:
>> "iommu/dma: Allow MSI-only cookies".
>>
>> This series currently does not address IRQ safety assessment.
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://github.com/eauger/linux/tree/v4.9-rc5-reserved-rfc-v3
>>
>> History:
>> RFC v2 -> v3:
>> - switch to an iommu-group sysfs API
>> - use new dummy allocator provided by Robin
>> - dummy allocator initialized by vfio-iommu-type1 after enumerating
>>   the reserved regions
>> - at the moment ARM MSI base address/size is left unchanged compared
>>   to v2
>> - we currently report reserved regions and not usable IOVA regions as
>>   requested by Alex
>>
>> RFC v1 -> v2:
>> - fix intel_add_reserved_regions
>> - add mutex lock/unlock in vfio_iommu_type1
>>
>>
>> Eric Auger (10):
>>   iommu/dma: Allow MSI-only cookies
>>   iommu: Rename iommu_dm_regions into iommu_resv_regions
>>   iommu: Add new reserved IOMMU attributes
>>   iommu: iommu_alloc_resv_region
>>   iommu: Do not map reserved regions
>>   iommu: iommu_get_group_resv_regions
>>   iommu: Implement reserved_regions iommu-group sysfs file
>>   iommu/vt-d: Implement reserved region get/put callbacks
>>   iommu/arm-smmu: Implement reserved region get/put callbacks
>>   vfio/type1: Get MSI cookie
>>
>>  drivers/iommu/amd_iommu.c   |  20 +++---
>>  drivers/iommu/arm-smmu.c|  52 +++
>>  drivers/iommu/dma-iommu.c   | 116 ++
>> ---
>>  drivers/iommu/intel-iommu.c |  50 ++
>>  drivers/iommu/iommu.c   | 141
>> 
>>  drivers/vfio/vfio_iommu_type1.c |  26 
>>  include/linux/dma-iommu.h   |   7 ++
>>  include/linux/iommu.h   |  49 ++
>>  8 files changed, 391 insertions(+), 70 deletions(-)
>>
>> --
>> 1.9.1
>>
>> ___
>> iommu mailing list
>> io...@lists.linux-foundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/iommu
> 
> ___
> linux-arm-kernel mailing list
> linux-arm-ker...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>