Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 26/02/15 13:56, Jan Beulich wrote:
> --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
> return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
> }
>
> -static inline struct page_info* alloc_amd_iommu_pgtable(void)
> +static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
> {
> struct page_info *pg;
> void *vaddr;
>
> -pg = alloc_domheap_page(NULL, 0);
> +pg = alloc_domheap_page(d, MEMF_no_owner);
Same comment as with the VT-d side of things. This should be based on
the proximity information of the IOMMU, not of the owning domain.
~Andrew
> if ( pg == NULL )
> return 0;
> vaddr = __map_domain_page(pg);
>
>
>
>
>
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
>>> On 05.03.15 at 18:30, wrote:
> On 26/02/15 13:56, Jan Beulich wrote:
>> --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>> +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>> @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
>> return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
>> }
>>
>> -static inline struct page_info* alloc_amd_iommu_pgtable(void)
>> +static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
>> {
>> struct page_info *pg;
>> void *vaddr;
>>
>> -pg = alloc_domheap_page(NULL, 0);
>> +pg = alloc_domheap_page(d, MEMF_no_owner);
>
> Same comment as with the VT-d side of things. This should be based on
> the proximity information of the IOMMU, not of the owning domain.
I think I buy this argument on the VT-d side (under the assumption
that there's going to be at least one IOMMU per node), but I'm not
sure here: The most modern AMD box I have has just a single
IOMMU for 4 nodes it reports.
Jan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 06/03/2015 07:50, Jan Beulich wrote:
On 05.03.15 at 18:30, wrote:
>> On 26/02/15 13:56, Jan Beulich wrote:
>>> --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>>> +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>>> @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
>>> return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
>>> }
>>>
>>> -static inline struct page_info* alloc_amd_iommu_pgtable(void)
>>> +static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
>>> {
>>> struct page_info *pg;
>>> void *vaddr;
>>>
>>> -pg = alloc_domheap_page(NULL, 0);
>>> +pg = alloc_domheap_page(d, MEMF_no_owner);
>> Same comment as with the VT-d side of things. This should be based on
>> the proximity information of the IOMMU, not of the owning domain.
> I think I buy this argument on the VT-d side (under the assumption
> that there's going to be at least one IOMMU per node), but I'm not
> sure here: The most modern AMD box I have has just a single
> IOMMU for 4 nodes it reports.
It is not possible for an IOMMU to cover multiple NUMA nodes worth of
IO, because of the position it has to sit relative to the IO root ports
and QPI/HT links.
In AMD systems, the IOMMUs lives in the northbridges, meaning one per
numa node (as it is the northbridges which contain the hypertransport links)
The BIOS/firmware will only report IOMMUs from northbridges which have
IO connected to their IO hypertransport link (most systems in the wild
have all IO hanging off one or two Numa nodes). On the other hand, I
have an AMD system with 8 IOMMUs in use.
In Intel systems, there is one IOMMU for each socket (to cover the
on-chip root ports and GPU if applicable) and one IOMMU in the IOH/PCH
(depending on generation) to cover the legacy IO.
In all cases, the IOMMUs are local to a single NUMA node, and would
benefit from having the control pages and pagetables allocated in local RAM.
~Andrew
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 3/6/2015 6:15 AM, Andrew Cooper wrote:
On 06/03/2015 07:50, Jan Beulich wrote:
On 05.03.15 at 18:30, wrote:
On 26/02/15 13:56, Jan Beulich wrote:
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
}
-static inline struct page_info* alloc_amd_iommu_pgtable(void)
+static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
{
struct page_info *pg;
void *vaddr;
-pg = alloc_domheap_page(NULL, 0);
+pg = alloc_domheap_page(d, MEMF_no_owner);
Same comment as with the VT-d side of things. This should be based on
the proximity information of the IOMMU, not of the owning domain.
I think I buy this argument on the VT-d side (under the assumption
that there's going to be at least one IOMMU per node), but I'm not
sure here: The most modern AMD box I have has just a single
IOMMU for 4 nodes it reports.
It is not possible for an IOMMU to cover multiple NUMA nodes worth of
IO, because of the position it has to sit relative to the IO root ports
and QPI/HT links.
In AMD systems, the IOMMUs lives in the northbridges, meaning one per
numa node (as it is the northbridges which contain the hypertransport links)
The BIOS/firmware will only report IOMMUs from northbridges which have
IO connected to their IO hypertransport link (most systems in the wild
have all IO hanging off one or two Numa nodes). On the other hand, I
have an AMD system with 8 IOMMUs in use.
Actually, a single IOMMU could handle multiple nodes. For example, in
scenario of a multi-chip-module (MCM) setup, there could be at least 2-4
nodes sharing one IOMMU depending on how the platform vendor configuring
the system. In the server platforms, IOMMU is in AMD northbridge
chipsets (e.g. SR56xx). This website has an example of such system
configuration
(http://www.qdpma.com/systemarchitecture/SystemArchitecture_Opteron.html).
For AMD IOMMU, the IVRS table specifies the PCI bus/device ranges to be
handled by each IOMMU. This is probably should be considered here.
In Intel systems, there is one IOMMU for each socket (to cover the
on-chip root ports and GPU if applicable) and one IOMMU in the IOH/PCH
(depending on generation) to cover the legacy IO.
In all cases, the IOMMUs are local to a single NUMA node, and would
benefit from having the control pages and pagetables allocated in local RAM.
As state above, this is not the case for AMD IOMMU.
Thanks,
Suravee
~Andrew
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 09/03/15 15:42, Suravee Suthikulanit wrote:
> On 3/6/2015 6:15 AM, Andrew Cooper wrote:
>> On 06/03/2015 07:50, Jan Beulich wrote:
>> On 05.03.15 at 18:30, wrote:
On 26/02/15 13:56, Jan Beulich wrote:
> --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
> @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
> return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >>
> PAGE_SHIFT;
> }
>
> -static inline struct page_info* alloc_amd_iommu_pgtable(void)
> +static inline struct page_info *alloc_amd_iommu_pgtable(struct
> domain *d)
> {
> struct page_info *pg;
> void *vaddr;
>
> -pg = alloc_domheap_page(NULL, 0);
> +pg = alloc_domheap_page(d, MEMF_no_owner);
Same comment as with the VT-d side of things. This should be based on
the proximity information of the IOMMU, not of the owning domain.
>>> I think I buy this argument on the VT-d side (under the assumption
>>> that there's going to be at least one IOMMU per node), but I'm not
>>> sure here: The most modern AMD box I have has just a single
>>> IOMMU for 4 nodes it reports.
>>
>> It is not possible for an IOMMU to cover multiple NUMA nodes worth of
>> IO, because of the position it has to sit relative to the IO root ports
>> and QPI/HT links.
>>
>> In AMD systems, the IOMMUs lives in the northbridges, meaning one per
>> numa node (as it is the northbridges which contain the hypertransport
>> links)
>>
>> The BIOS/firmware will only report IOMMUs from northbridges which have
>> IO connected to their IO hypertransport link (most systems in the wild
>> have all IO hanging off one or two Numa nodes). On the other hand, I
>> have an AMD system with 8 IOMMUs in use.
>
>
> Actually, a single IOMMU could handle multiple nodes. For example, in
> scenario of a multi-chip-module (MCM) setup, there could be at least
> 2-4 nodes sharing one IOMMU depending on how the platform vendor
> configuring the system. In the server platforms, IOMMU is in AMD
> northbridge chipsets (e.g. SR56xx). This website has an example of
> such system configuration
> (http://www.qdpma.com/systemarchitecture/SystemArchitecture_Opteron.html).
Ok - I was basing my example on the last layout I had the manual for,
which I believe was Bulldozer.
However, my point still stands that there is an IOMMU between any IO and
RAM. An individual IOMMU will always benefit from having its
iopagetables on the local numa node, rather than the numa node(s) which
the domain owning the device is running on.
>
> For AMD IOMMU, the IVRS table specifies the PCI bus/device ranges to
> be handled by each IOMMU. This is probably should be considered here.
Presumably a PCI transaction must never get onto the HT bus without
having already undergone translation, or there can be no guarantee that
it would be routed via the IOMMU? Or are you saying that there are
cases where a transaction will enter the HT bus, route sideways to an
IOMMU, undergo translation, then route back onto the HT bus to the
target RAM/processor?
~Andrew
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 3/9/2015 12:26 PM, Andrew Cooper wrote:
On 09/03/15 15:42, Suravee Suthikulanit wrote:
On 3/6/2015 6:15 AM, Andrew Cooper wrote:
On 06/03/2015 07:50, Jan Beulich wrote:
On 05.03.15 at 18:30, wrote:
On 26/02/15 13:56, Jan Beulich wrote:
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >>
PAGE_SHIFT;
}
-static inline struct page_info* alloc_amd_iommu_pgtable(void)
+static inline struct page_info *alloc_amd_iommu_pgtable(struct
domain *d)
{
struct page_info *pg;
void *vaddr;
-pg = alloc_domheap_page(NULL, 0);
+pg = alloc_domheap_page(d, MEMF_no_owner);
Same comment as with the VT-d side of things. This should be based on
the proximity information of the IOMMU, not of the owning domain.
I think I buy this argument on the VT-d side (under the assumption
that there's going to be at least one IOMMU per node), but I'm not
sure here: The most modern AMD box I have has just a single
IOMMU for 4 nodes it reports.
It is not possible for an IOMMU to cover multiple NUMA nodes worth of
IO, because of the position it has to sit relative to the IO root ports
and QPI/HT links.
In AMD systems, the IOMMUs lives in the northbridges, meaning one per
numa node (as it is the northbridges which contain the hypertransport
links)
The BIOS/firmware will only report IOMMUs from northbridges which have
IO connected to their IO hypertransport link (most systems in the wild
have all IO hanging off one or two Numa nodes). On the other hand, I
have an AMD system with 8 IOMMUs in use.
Actually, a single IOMMU could handle multiple nodes. For example, in
scenario of a multi-chip-module (MCM) setup, there could be at least
2-4 nodes sharing one IOMMU depending on how the platform vendor
configuring the system. In the server platforms, IOMMU is in AMD
northbridge chipsets (e.g. SR56xx). This website has an example of
such system configuration
(http://www.qdpma.com/systemarchitecture/SystemArchitecture_Opteron.html).
Ok - I was basing my example on the last layout I had the manual for,
which I believe was Bulldozer.
However, my point still stands that there is an IOMMU between any IO and
RAM. An individual IOMMU will always benefit from having its
iopagetables on the local numa node, rather than the numa node(s) which
the domain owning the device is running on.
I agree that having the IO page tables on the NUMA node that is closest
to the IOMMU would be beneficial. However, I am not sure at the moment
that this information could be easily determined. I think ACPI _PXM for
devices should be able to provide this information, but this is optional
and often not available.
For AMD IOMMU, the IVRS table specifies the PCI bus/device ranges to
be handled by each IOMMU. This is probably should be considered here.
Presumably a PCI transaction must never get onto the HT bus without
having already undergone translation, or there can be no guarantee that
it would be routed via the IOMMU? Or are you saying that there are
cases where a transaction will enter the HT bus, route sideways to an
IOMMU, undergo translation, then route back onto the HT bus to the
target RAM/processor?
~Andrew
IOMMU sits between PCI devices (downstream) and HT (uptream), all DMA
transactions from downstream must go through IOMMU. On the other hand,
the I/O page translation is handled by IOMMU, and it is a separate
traffic than the downstream device DMA transactions.
Suravee
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
>>> On 09.03.15 at 20:02, wrote: > I agree that having the IO page tables on the NUMA node that is closest > to the IOMMU would be beneficial. And I already withdrew this patch and the corresponding VT-d one. > However, I am not sure at the moment > that this information could be easily determined. I think ACPI _PXM for > devices should be able to provide this information, but this is optional > and often not available. And even if it was available, it would be too late at least for Dom0's allocations (as it requires Dom0's interpreter to dig out this detail). The best we could do in that case would be to try to replace the existing tables. Or assume Dom0 is being placed suitably by the dom0_nodes= option. Or add yet another option. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from
On 03/10/2015 03:35 AM, Jan Beulich wrote: On 09.03.15 at 20:02, wrote: I agree that having the IO page tables on the NUMA node that is closest to the IOMMU would be beneficial. And I already withdrew this patch and the corresponding VT-d one. However, I am not sure at the moment that this information could be easily determined. I think ACPI _PXM for devices should be able to provide this information, but this is optional and often not available. And even if it was available, it would be too late at least for Dom0's allocations (as it requires Dom0's interpreter to dig out this detail). The best we could do in that case would be to try to replace the existing tables. Or assume Dom0 is being placed suitably by the dom0_nodes= option. Or add yet another option. There is a nodeID register on each northbridge (D1F0x60). You would have to figure out how to map it to _PXMs though. -boris ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
