Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-20 Thread G.R. 
Thank you for all your help Jan && Roger.
I think we can settle on this thread now.
After disabling the debug build, the gfx-passthrough back to work now.

On Mon, Feb 20, 2017 at 12:38 PM, G.R.  wrote:
>>Feb 10, 2017 02:00，"Roger Pau Monné"  wrote:
>>
>>On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
>>> >>> On 09.02.17 at 15:46,  wrote:
>>> > BTW -- I think that fix should not be conflicting with your debug change,
>>> > right?
>>>
>>> Yes - ideally you'd keep that one in place along with adding Roger's
>>> patch.
>>
>>Please use the patch below, the previous one was missing a break, which made 
>>it
>>completely useless, sorry.
>>
>>Roger.
>
> Thanks for the updated patch and sorry for the delayed response.
> I would like to confirm that the updated patch appears to work. The
> flooding fault is gone now.
> I wasn't able to get ipxe launched in domU with gfx-passthrough=1 though.
> Hopefully this is not a real issue as I reported the following before:
>>For unknown reason, debug version of hypervisor will cause domU hang if 
>>gfx-passthrough=1 is present (traditional device model).
>
> Rui

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-19 Thread G.R. 
>Feb 10, 2017 02:00，"Roger Pau Monné"  wrote:
>
>On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
>> >>> On 09.02.17 at 15:46,  wrote:
>> > BTW -- I think that fix should not be conflicting with your debug change,
>> > right?
>>
>> Yes - ideally you'd keep that one in place along with adding Roger's
>> patch.
>
>Please use the patch below, the previous one was missing a break, which made it
>completely useless, sorry.
>
>Roger.

Thanks for the updated patch and sorry for the delayed response.
I would like to confirm that the updated patch appears to work. The
flooding fault is gone now.
I wasn't able to get ipxe launched in domU with gfx-passthrough=1 though.
Hopefully this is not a real issue as I reported the following before:
>For unknown reason, debug version of hypervisor will cause domU hang if 
>gfx-passthrough=1 is present (traditional device model).

Rui

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-09 Thread Roger Pau Monné 
On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
> >>> On 09.02.17 at 15:46,  wrote:
> > BTW -- I think that fix should not be conflicting with your debug change, 
> > right?
> 
> Yes - ideally you'd keep that one in place along with adding Roger's
> patch.

Please use the patch below, the previous one was missing a break, which made it
completely useless, sorry.

Roger.

---8<---
From 27d234b34b95f9b5900dd0760a0f16161459bcdd Mon Sep 17 00:00:00 2001
From: Roger Pau Monne 
Date: Thu, 9 Feb 2017 17:59:32 +
Subject: [PATCH] x86/iommu: add IOMMU entries for p2m_mmio_direct pages
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

There's nothing wrong with allowing the domain to perform DMA transfers to
MMIO areas that it already can access from the CPU, and this allows us to
remove the hack in set_identity_p2m_entry for PVH Dom0.

Signed-off-by: Roger Pau Monné 
---
 xen/arch/x86/mm/p2m-ept.c |  5 +++--
 xen/arch/x86/mm/p2m-pt.c  | 17 ++---
 xen/arch/x86/mm/p2m.c |  9 -
 xen/include/asm-x86/p2m.h |  7 ++-
 4 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 04878f5..f47f323 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -672,7 +672,7 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 uint8_t ipat = 0;
 bool_t need_modify_vtd_table = 1;
 bool_t vtd_pte_present = 0;
-unsigned int iommu_flags = p2m_get_iommu_flags(p2mt);
+unsigned int iommu_flags = p2m_get_iommu_flags(p2mt, mfn);
 bool_t needs_sync = 1;
 ept_entry_t old_entry = { .epte = 0 };
 ept_entry_t new_entry = { .epte = 0 };
@@ -798,7 +798,8 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 
 /* Safe to read-then-write because we hold the p2m lock */
 if ( ept_entry->mfn == new_entry.mfn &&
- p2m_get_iommu_flags(ept_entry->sa_p2mt) == iommu_flags )
+ p2m_get_iommu_flags(ept_entry->sa_p2mt, _mfn(ept_entry->mfn)) ==
+ iommu_flags )
 need_modify_vtd_table = 0;
 
 ept_p2m_type_to_flags(p2m, &new_entry, p2mt, p2ma);
diff --git a/xen/arch/x86/mm/p2m-pt.c b/xen/arch/x86/mm/p2m-pt.c
index 3b025d5..a23d0bd 100644
--- a/xen/arch/x86/mm/p2m-pt.c
+++ b/xen/arch/x86/mm/p2m-pt.c
@@ -499,7 +499,7 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 l2_pgentry_t l2e_content;
 l3_pgentry_t l3e_content;
 int rc;
-unsigned int iommu_pte_flags = p2m_get_iommu_flags(p2mt);
+unsigned int iommu_pte_flags = p2m_get_iommu_flags(p2mt, mfn);
 /*
  * old_mfn and iommu_old_flags control possible flush/update needs on the
  * IOMMU: We need to flush when MFN or flags (i.e. permissions) change.
@@ -565,9 +565,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long 
gfn, mfn_t mfn,
 {
 if ( flags & _PAGE_PSE )
 {
-iommu_old_flags =
-p2m_get_iommu_flags(p2m_flags_to_type(flags));
 old_mfn = l1e_get_pfn(*p2m_entry);
+iommu_old_flags =
+p2m_get_iommu_flags(p2m_flags_to_type(flags),
+_mfn(old_mfn));
 }
 else
 {
@@ -609,9 +610,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long 
gfn, mfn_t mfn,
 p2m_entry = p2m_find_entry(table, &gfn_remainder, gfn,
0, L1_PAGETABLE_ENTRIES);
 ASSERT(p2m_entry);
-iommu_old_flags =
-p2m_get_iommu_flags(p2m_flags_to_type(l1e_get_flags(*p2m_entry)));
 old_mfn = l1e_get_pfn(*p2m_entry);
+iommu_old_flags =
+p2m_get_iommu_flags(p2m_flags_to_type(l1e_get_flags(*p2m_entry)),
+_mfn(old_mfn));
 
 if ( mfn_valid(mfn) || p2m_allows_invalid_mfn(p2mt) )
 entry_content = p2m_l1e_from_pfn(mfn_x(mfn),
@@ -637,9 +639,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long 
gfn, mfn_t mfn,
 {
 if ( flags & _PAGE_PSE )
 {
-iommu_old_flags =
-p2m_get_iommu_flags(p2m_flags_to_type(flags));
 old_mfn = l1e_get_pfn(*p2m_entry);
+iommu_old_flags =
+p2m_get_iommu_flags(p2m_flags_to_type(flags),
+_mfn(old_mfn));
 }
 else
 {
diff --git a/xen/arch/x86/mm/p2m.c b/xen/arch/x86/mm/p2m.c
index 6548e9f..bd8ce35 100644
--- a/xen/arch/x86/mm/p2m.c
+++ b/xen/arch/x86/mm/p2m.c
@@ -1053,16 +1053,7 @@ int set_identity_p2m_entry(struct domain *d, unsigned 
long gfn,
 ret = p2m_set_entry(p2m, gfn, _mfn(gfn), PAGE_ORDER_4K,
 p2m_mmio_direct, p2ma);
 else if ( mfn_x(mfn) == gfn && p2mt == p2m_mmio_direct && a == p2ma )
-{

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-09 Thread Jan Beulich 
>>> On 09.02.17 at 15:46,  wrote:
> BTW -- I think that fix should not be conflicting with your debug change, 
> right?

Yes - ideally you'd keep that one in place along with adding Roger's
patch.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-09 Thread G.R. 
On Wed, Feb 8, 2017 at 11:59 PM, Jan Beulich  wrote:
 On 08.02.17 at 15:56,  wrote:
>> On Wed, Feb 8, 2017 at 10:29 PM, G.R.  
>> wrote:
>>> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich  wrote:
>>> On 07.02.17 at 16:44,  wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
> On 05.02.17 at 06:51,  wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>> to reduce the overall amount of output (and help readability).
>
> Please see attached...

 So can you please give
 https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
 a try?

>>> Hmm, it does not help.
>>> But I'll need to double check if I was misleading you.
>>> I used attempt dom0pvh=1 but it was too unstable and I was only able
>>> to disable it through hacking grub.cfg through sshfs remotely.
>>> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
>>> back when I was generating the log yesterday.
>>>
>>> Going to do it once again now.
>>
>> It appears that dom0pvh or not does not affect the debug output
>> without Roger's patch.
>> Anyway, attaching the output for you to double check.
>
> Well, if this indeed was with his patch in place, then I'm puzzled.
> I'd have to further extend the debugging patch then, but this may
> take a few days to get to.

Please hold-off and let me double check for you. I'm also confused by
my current situation right now.
I think should be running without Roger's fix with dom0pvh=0.
But I happen to see a lot of fault message right now, from dom0.
Maybe I forgot to reboot last night after reverting back.
I'll build with Roger's fix again and do the experiment once more.
BTW -- I think that fix should not be conflicting with your debug change, right?

Thanks,
Rui

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-08 Thread Jan Beulich 
>>> On 08.02.17 at 15:56,  wrote:
> On Wed, Feb 8, 2017 at 10:29 PM, G.R.  
> wrote:
>> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich  wrote:
>> On 07.02.17 at 16:44,  wrote:
 On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
 On 05.02.17 at 06:51,  wrote:
>> I finally get some spare time to collect the debug info.
>
> As I continue to be puzzled, best I could come up with is an
> extension to the debug patch. Please use the attached one
> in place of the earlier one, ideally on top of
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>  
> to reduce the overall amount of output (and help readability).

 Please see attached...
>>>
>>> So can you please give
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html 
>>> a try?
>>>
>> Hmm, it does not help.
>> But I'll need to double check if I was misleading you.
>> I used attempt dom0pvh=1 but it was too unstable and I was only able
>> to disable it through hacking grub.cfg through sshfs remotely.
>> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
>> back when I was generating the log yesterday.
>>
>> Going to do it once again now.
> 
> It appears that dom0pvh or not does not affect the debug output
> without Roger's patch.
> Anyway, attaching the output for you to double check.

Well, if this indeed was with his patch in place, then I'm puzzled.
I'd have to further extend the debugging patch then, but this may
take a few days to get to.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-08 Thread G.R. 
On Wed, Feb 8, 2017 at 10:29 PM, G.R.  wrote:
> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich  wrote:
> On 07.02.17 at 16:44,  wrote:
>>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
>>> On 05.02.17 at 06:51,  wrote:
> I finally get some spare time to collect the debug info.

 As I continue to be puzzled, best I could come up with is an
 extension to the debug patch. Please use the attached one
 in place of the earlier one, ideally on top of
 https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
 to reduce the overall amount of output (and help readability).
>>>
>>> Please see attached...
>>
>> So can you please give
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
>> a try?
>>
> Hmm, it does not help.
> But I'll need to double check if I was misleading you.
> I used attempt dom0pvh=1 but it was too unstable and I was only able
> to disable it through hacking grub.cfg through sshfs remotely.
> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
> back when I was generating the log yesterday.
>
> Going to do it once again now.

It appears that dom0pvh or not does not affect the debug output
without Roger's patch.
Anyway, attaching the output for you to double check.

However, dom0pvh does make different with Roger's patch.
With dom0pvh=1 + Roger's patch, the same fault message previously
observed for domU is now also showing up for dom0.


rmrr_dbg_dom0pvh_off.xz
Description: application/xz
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-08 Thread G.R. 
On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich  wrote:
 On 07.02.17 at 16:44,  wrote:
>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
>> On 05.02.17 at 06:51,  wrote:
 I finally get some spare time to collect the debug info.
>>>
>>> As I continue to be puzzled, best I could come up with is an
>>> extension to the debug patch. Please use the attached one
>>> in place of the earlier one, ideally on top of
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>>> to reduce the overall amount of output (and help readability).
>>
>> Please see attached...
>
> So can you please give
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
> a try?
>
Hmm, it does not help.
But I'll need to double check if I was misleading you.
I used attempt dom0pvh=1 but it was too unstable and I was only able
to disable it through hacking grub.cfg through sshfs remotely.
I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
back when I was generating the log yesterday.

Going to do it once again now.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-08 Thread Jan Beulich 
>>> On 07.02.17 at 16:44,  wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
> On 05.02.17 at 06:51,  wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

So can you please give
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
a try?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-07 Thread Jan Beulich 
>>> On 07.02.17 at 16:44,  wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
> On 05.02.17 at 06:51,  wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

Okay, that's because p2m_get_iommu_flags() returns zero for
p2m_mmio_direct, which iirc we already have a patch floating
around for (needed for PVHv2). Roger - what's the status of
that? Would it make sense to split this out of your series, to get
in rather sooner?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-07 Thread Jan Beulich 
>>> On 07.02.17 at 16:44,  wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
> On 05.02.17 at 06:51,  wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

Thanks, this confirms my suspicion that EPT tables get updated, but
IOMMU ones don't. I'll have to go over the code yet another time to
see if I can spot what's wrong, or whether I need to further extend
the debugging patch.

Are you using a PVH Dom0 here, btw? If so, is the problem
connected to that? The fact that things work there may be
attributable to the mappings already being there even before the
identity mapping calls.

I further note that there's an apparently legitimate IOMMU fault
before the first guest gets started. This may indicate an issue
with the Dom0 kernel.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-07 Thread G.R. 
On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich  wrote:
 On 05.02.17 at 06:51,  wrote:
>> I finally get some spare time to collect the debug info.
>
> As I continue to be puzzled, best I could come up with is an
> extension to the debug patch. Please use the attached one
> in place of the earlier one, ideally on top of
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
> to reduce the overall amount of output (and help readability).

Please see attached...


dmsg2.xz
Description: application/xz
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-07 Thread Jan Beulich 
>>> On 07.02.17 at 03:12,  wrote:
> On Mon, Feb 6, 2017 at 7:43 PM, Jan Beulich  wrote:
> On 05.02.17 at 06:51,  wrote:
>>> Please find the full log in the attachment.
>>
>> Sadly that one is only a partial log again. I'd really need to see the
>> boot messages too, in particular to (hopefully) be able to judge
>> whether your system uses shared or separate EPT and VT-d tables.
>>
> In the dom0.xz attachment (the second one on Feb 5th), the xl dmesg
> info from boot stage is retained.
> If that's not good enough, please instruct on how to generate the desired 
> log.

That's good enough; sorry for not having checked there.

> Quote some log snippets here, please find the full log in the old 
> attachment:
> It appears that the system uses separated EPT && VT-d tables.
> Is this good or bad?

Neither - it is merely relevant. In particular (and the new debug
patch may help understand this) it may mean that while EPT
tables get updated, IOMMU page tables may not be (even if the
code suggests this is being taken care of).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-06 Thread G.R. 
On Mon, Feb 6, 2017 at 7:43 PM, Jan Beulich  wrote:
 On 05.02.17 at 06:51,  wrote:
>> Please find the full log in the attachment.
>
> Sadly that one is only a partial log again. I'd really need to see the
> boot messages too, in particular to (hopefully) be able to judge
> whether your system uses shared or separate EPT and VT-d tables.
>
In the dom0.xz attachment (the second one on Feb 5th), the xl dmesg
info from boot stage is retained.
If that's not good enough, please instruct on how to generate the desired log.

Quote some log snippets here, please find the full log in the old attachment:
It appears that the system uses separated EPT && VT-d tables.
Is this good or bad?

(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled. <== EPT && VT-d
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed

> Jan
>

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-06 Thread Jan Beulich 
>>> On 05.02.17 at 06:51,  wrote:
> I finally get some spare time to collect the debug info.

As I continue to be puzzled, best I could come up with is an
extension to the debug patch. Please use the attached one
in place of the earlier one, ideally on top of
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
to reduce the overall amount of output (and help readability).

Jan

--- unstable.orig/xen/arch/x86/mm/p2m.c 2016-11-09 15:17:11.0 +0100
+++ unstable/xen/arch/x86/mm/p2m.c  2017-02-06 13:32:14.0 +0100
@@ -1048,10 +1048,37 @@ int set_identity_p2m_entry(struct domain
 gfn_lock(p2m, gfn, 0);
 
 mfn = p2m->get_entry(p2m, gfn, &p2mt, &a, 0, NULL, NULL);
+if(!(gfn & 0xff)) {//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(++cnt > thr) {
+  thr |= cnt;
+  printk("d%d: GFN %"PRI_gfn" -> %"PRI_mfn":%d:%d\n", d->domain_id, gfn, mfn, 
p2mt, a);
+ }
+}
 
 if ( p2mt == p2m_invalid || p2mt == p2m_mmio_dm )
+{//temp
 ret = p2m_set_entry(p2m, gfn, _mfn(gfn), PAGE_ORDER_4K,
 p2m_mmio_direct, p2ma);
+ if(ret || !(gfn & 0xff)) {//temp
+  static domid_t last;
+  static unsigned long cnt, thr;
+  if(d->domain_id > last) {
+   thr = cnt = 0;
+   last = d->domain_id;
+  }
+  if(ret || ++cnt > thr) {
+   thr |= cnt;
+   mfn = p2m->get_entry(p2m, gfn, &p2mt, &a, 0, NULL, NULL);
+   printk("d%d: GFN %"PRI_gfn" -> %"PRI_mfn":%d:%d (%d)\n", d->domain_id, gfn, 
mfn, p2mt, a, ret);
+  }
+ }
+}
 else if ( mfn_x(mfn) == gfn && p2mt == p2m_mmio_direct && a == p2ma )
 {
 ret = 0;
--- unstable.orig/xen/drivers/passthrough/vtd/iommu.c   2017-01-19 
17:27:05.0 +0100
+++ unstable/xen/drivers/passthrough/vtd/iommu.c2017-02-06 
13:21:42.0 +0100
@@ -897,8 +897,23 @@ static int iommu_page_fault_do_one(struc
kind, fault_reason, reason);
 
 if ( iommu_verbose && fault_type == DMA_REMAP )
+{//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ const struct pci_dev*pdev;
+ pcidevs_lock();
+ pdev = pci_get_real_pdev(seg, PCI_BUS(source_id), PCI_DEVFN2(source_id));
+ if(pdev && pdev->domain && pdev->domain->domain_id > last) {
+  thr = cnt = 0;
+  last = pdev->domain->domain_id;
+ }
+ pcidevs_unlock();
+ if(++cnt > thr) {
+  thr |= cnt;
 print_vtd_entries(iommu, PCI_BUS(source_id), PCI_DEVFN2(source_id),
   addr >> PAGE_SHIFT);
+ }
+}
 
 return 0;
 }
@@ -1890,6 +1905,7 @@ static void iommu_set_pgd(struct domain
 
 static int rmrr_identity_mapping(struct domain *d, bool_t map,
  const struct acpi_rmrr_unit *rmrr,
+u16 bdf,//temp
  u32 flag)
 {
 unsigned long base_pfn = rmrr->base_address >> PAGE_SHIFT_4K;
@@ -1914,6 +1930,7 @@ static int rmrr_identity_mapping(struct
 if ( map )
 {
 ++mrmrr->count;
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] count %u\n", d->domain_id, base_pfn, 
end_pfn, mrmrr->count);//temp
 return 0;
 }
 
@@ -1928,6 +1945,7 @@ static int rmrr_identity_mapping(struct
 }
 
 list_del(&mrmrr->list);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] zapped (%d)\n", d->domain_id, 
mrmrr->base >> PAGE_SHIFT_4K, end_pfn, ret);//temp
 xfree(mrmrr);
 return ret;
 }
@@ -1941,11 +1959,32 @@ static int rmrr_identity_mapping(struct
 int err = set_identity_p2m_entry(d, base_pfn, p2m_access_rw, flag);
 
 if ( err )
+{//temp
+ printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] map error %d @ %lx\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, err, base_pfn);
 return err;
+} else {
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(!(base_pfn & 0xff) && ++cnt > thr) {
+  const struct pci_dev*pdev = pci_get_pdev(rmrr->segment, PCI_BUS(bdf), 
PCI_DEVFN2(bdf));
+  const struct acpi_drhd_unit*drhd = pdev ? acpi_find_matched_drhd_unit(pdev) 
: NULL;
+  thr |= cnt;
+  printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] mapped %lx\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, base_pfn);
+  if(drhd)
+   print_vtd_entries(drhd->iommu, PCI_BUS(bdf), PCI_DEVFN2(bdf), base_pfn);
+  if(hap_enabled(d))
+   ept_walk_table(d, base_pfn);
+ }
+}
 base_pfn++;
 }
 
 mrmrr = xmalloc(struct mapped_rmrr);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] alloc -> %p\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, mrmrr);//temp
 if ( !mrmrr )
 return -ENOMEM;
 mrmrr->base = rmrr->base_address;
@@ -1987,7 +2026,7 @@ static int intel_iommu_add_device(u8 dev
  * Since RMRRs are always reserved in the e820 map for the hardware
  * domain, there shouldn't be a conflict.
  */
-ret = rmrr_identity_mapping(p

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-06 Thread Jan Beulich 
>>> On 05.02.17 at 06:51,  wrote:
> Please find the full log in the attachment.

Sadly that one is only a partial log again. I'd really need to see the
boot messages too, in particular to (hopefully) be able to judge
whether your system uses shared or separate EPT and VT-d tables.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-06 Thread G.R. 
On Mon, Feb 6, 2017 at 3:40 PM, Jan Beulich  wrote:
 On 05.02.17 at 08:18,  wrote:
>> But we didn't see a map error in debug log either.
>
> I'll have to look into this more closely.

Let me know when you need more info / debug log.:-)

BTW, if this helps my hardware setup is based on i7-3770 + ASRock H77M-iTX.
I'm not sure if this chip is officially VT-D supported but I've been
using this for ~3 years with SATA && IGD passthrough.
So I assume this is not an HW issue. (at least for now)

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-05 Thread Jan Beulich 
>>> On 05.02.17 at 08:18,  wrote:
> On Sun, Feb 5, 2017 at 1:51 PM, G.R.  wrote:
>> On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich  wrote:
>> On 17.01.17 at 16:08,  wrote:
 But fortunately commenting out that line could still reproduce the IOMMU
 fault.
 I was lucky to capture the full log before it fills up my 100MB ring buffer
 (in less than 2 seconds).
>>>
>>> So here's a first take at a debugging patch. I've tried to limit existing
>>> output, so that you'd have better chance of again capturing all
>>> interesting messages.
>>>
>>
>> Hi Jan,
>> I finally get some spare time to collect the debug info.
>> Please find the full log in the attachment.
>>
>> The mapping appears to be working:
>> (XEN) d8: RMRR [cf800,dfa00] mapped cf800
>> (XEN) d8: RMRR [cf800,dfa00] mapped cf900
>> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
>> (XEN) d8: RMRR [cf800,dfa00] mapped cff00
>> (XEN) d8: RMRR [cf800,dfa00] mapped d0700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d1700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d3700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d7700
>> (XEN) d8: RMRR [cf800,dfa00] mapped df700
>> (XEN) d8: RMRR [cf800,dfa00] alloc -> 83013156ffb0
>>
>> But I'm not sure if the vtd_entries look correct: (Is the 'not
>> present' line okay?)

Rather not, hence ...

>> Still see the 'Fault overflow' line in the very first fault.
>> The fault is about write-access not permitted.

... this.

>> Is the map read-only here?

No, as you've noticed above, it's simply not present.

>> Or are we looking at the correct PTE?

Comparing the two dumps, I think we do.

> Attraching an xl dmesg log for dom0, which shows a more reasonable vtd_entry.
> Does it mean that the mapping wasn't properly setup in the domU case?

Looks like so.

> But we didn't see a map error in debug log either.

I'll have to look into this more closely.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-04 Thread G.R. 
On Sun, Feb 5, 2017 at 1:51 PM, G.R.  wrote:
> On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich  wrote:
> On 17.01.17 at 16:08,  wrote:
>>> But fortunately commenting out that line could still reproduce the IOMMU
>>> fault.
>>> I was lucky to capture the full log before it fills up my 100MB ring buffer
>>> (in less than 2 seconds).
>>
>> So here's a first take at a debugging patch. I've tried to limit existing
>> output, so that you'd have better chance of again capturing all
>> interesting messages.
>>
>
> Hi Jan,
> I finally get some spare time to collect the debug info.
> Please find the full log in the attachment.
>
> The mapping appears to be working:
> (XEN) d8: RMRR [cf800,dfa00] mapped cf800
> (XEN) d8: RMRR [cf800,dfa00] mapped cf900
> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
> (XEN) d8: RMRR [cf800,dfa00] mapped cff00
> (XEN) d8: RMRR [cf800,dfa00] mapped d0700
> (XEN) d8: RMRR [cf800,dfa00] mapped d1700
> (XEN) d8: RMRR [cf800,dfa00] mapped d3700
> (XEN) d8: RMRR [cf800,dfa00] mapped d7700
> (XEN) d8: RMRR [cf800,dfa00] mapped df700
> (XEN) d8: RMRR [cf800,dfa00] alloc -> 83013156ffb0
>
> But I'm not sure if the vtd_entries look correct: (Is the 'not
> present' line okay?)
> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
> (XEN) print_vtd_entries: iommu 8304152ec600 dev :00:02.0 gmfn cfb00
> (XEN) root_entry = 820040056000
> (XEN) root_entry[0] = 201fc6001
> (XEN) context = 82004002
> (XEN) context[10] = 1_13956c001
> (XEN) l3 = 820040022000
> (XEN) l3_index = 3
> (XEN) l3[3] = 1394ec003
> (XEN) l2 = 820040023000
> (XEN) l2_index = 7d
> (XEN) l2[7d] = 0
> (XEN) l2[7d] not present
>
> Still see the 'Fault overflow' line in the very first fault.
> The fault is about write-access not permitted.
> Is the map read-only here? Or are we looking at the correct PTE?
> (XEN) [VT-D]iommu.c:924: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:926: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> cfa0, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) print_vtd_entries: iommu 8304152ec600 dev :00:02.0 gmfn cfa00
> (XEN) root_entry = 8304152e9000
> (XEN) root_entry[0] = 201fc6001
> (XEN) context = 830201fc6000
> (XEN) context[10] = 1_13956c001
> (XEN) l3 = 83013956c000
> (XEN) l3_index = 3
> (XEN) l3[3] = 1394ec003
> (XEN) l2 = 8301394ec000
> (XEN) l2_index = 7d
> (XEN) l2[7d] = 0
> (XEN) l2[7d] not present
>

Attraching an xl dmesg log for dom0, which shows a more reasonable vtd_entry.
Does it mean that the mapping wasn't properly setup in the domU case?
But we didn't see a map error in debug log either.

(XEN) d0: RMRR [cf800,dfa00] mapped cfb00
(XEN) print_vtd_entries: iommu 8304152ec600 dev :00:02.0 gmfn cfb00
(XEN) root_entry = 8304152e9000
(XEN) root_entry[0] = 2030ca001
(XEN) context = 8302030ca000
(XEN) context[10] = 1_2032d1001
(XEN) l3 = 8302032d1000
(XEN) l3_index = 3
(XEN) l3[3] = 2030c7003
(XEN) l2 = 8302030c7000
(XEN) l2_index = 7d
(XEN) l2[7d] = 2030c5003
(XEN) l1 = 8302030c5000
(XEN) l1_index = 100
(XEN) l1[100] = cfb3


dom0.xz
Description: application/xz
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-02-04 Thread G.R. 
On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich  wrote:
 On 17.01.17 at 16:08,  wrote:
>> But fortunately commenting out that line could still reproduce the IOMMU
>> fault.
>> I was lucky to capture the full log before it fills up my 100MB ring buffer
>> (in less than 2 seconds).
>
> So here's a first take at a debugging patch. I've tried to limit existing
> output, so that you'd have better chance of again capturing all
> interesting messages.
>

Hi Jan,
I finally get some spare time to collect the debug info.
Please find the full log in the attachment.

The mapping appears to be working:
(XEN) d8: RMRR [cf800,dfa00] mapped cf800
(XEN) d8: RMRR [cf800,dfa00] mapped cf900
(XEN) d8: RMRR [cf800,dfa00] mapped cfb00
(XEN) d8: RMRR [cf800,dfa00] mapped cff00
(XEN) d8: RMRR [cf800,dfa00] mapped d0700
(XEN) d8: RMRR [cf800,dfa00] mapped d1700
(XEN) d8: RMRR [cf800,dfa00] mapped d3700
(XEN) d8: RMRR [cf800,dfa00] mapped d7700
(XEN) d8: RMRR [cf800,dfa00] mapped df700
(XEN) d8: RMRR [cf800,dfa00] alloc -> 83013156ffb0

But I'm not sure if the vtd_entries look correct: (Is the 'not
present' line okay?)
(XEN) d8: RMRR [cf800,dfa00] mapped cfb00
(XEN) print_vtd_entries: iommu 8304152ec600 dev :00:02.0 gmfn cfb00
(XEN) root_entry = 820040056000
(XEN) root_entry[0] = 201fc6001
(XEN) context = 82004002
(XEN) context[10] = 1_13956c001
(XEN) l3 = 820040022000
(XEN) l3_index = 3
(XEN) l3[3] = 1394ec003
(XEN) l2 = 820040023000
(XEN) l2_index = 7d
(XEN) l2[7d] = 0
(XEN) l2[7d] not present

Still see the 'Fault overflow' line in the very first fault.
The fault is about write-access not permitted.
Is the map read-only here? Or are we looking at the correct PTE?
(XEN) [VT-D]iommu.c:924: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:926: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
cfa0, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu 8304152ec600 dev :00:02.0 gmfn cfa00
(XEN) root_entry = 8304152e9000
(XEN) root_entry[0] = 201fc6001
(XEN) context = 830201fc6000
(XEN) context[10] = 1_13956c001
(XEN) l3 = 83013956c000
(XEN) l3_index = 3
(XEN) l3[3] = 1394ec003
(XEN) l2 = 8301394ec000
(XEN) l2_index = 7d
(XEN) l2[7d] = 0
(XEN) l2[7d] not present

> Jan
>


dmesg.xz
Description: application/xz
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-19 Thread Jan Beulich 
>>> On 17.01.17 at 16:08,  wrote:
> But fortunately commenting out that line could still reproduce the IOMMU
> fault.
> I was lucky to capture the full log before it fills up my 100MB ring buffer
> (in less than 2 seconds).

So here's a first take at a debugging patch. I've tried to limit existing
output, so that you'd have better chance of again capturing all
interesting messages.

Jan

--- unstable.orig/xen/drivers/passthrough/vtd/iommu.c   2017-01-03 
10:55:55.0 +0100
+++ unstable/xen/drivers/passthrough/vtd/iommu.c2017-01-19 
17:26:48.0 +0100
@@ -897,8 +897,23 @@ static int iommu_page_fault_do_one(struc
kind, fault_reason, reason);
 
 if ( iommu_verbose && fault_type == DMA_REMAP )
+{//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ const struct pci_dev*pdev;
+ pcidevs_lock();
+ pdev = pci_get_real_pdev(seg, PCI_BUS(source_id), PCI_DEVFN2(source_id));
+ if(pdev && pdev->domain && pdev->domain->domain_id > last) {
+  thr = cnt = 0;
+  last = pdev->domain->domain_id;
+ }
+ pcidevs_unlock();
+ if(++cnt > thr) {
+  thr |= cnt;
 print_vtd_entries(iommu, PCI_BUS(source_id), PCI_DEVFN2(source_id),
   addr >> PAGE_SHIFT);
+ }
+}
 
 return 0;
 }
@@ -1890,6 +1905,7 @@ static void iommu_set_pgd(struct domain
 
 static int rmrr_identity_mapping(struct domain *d, bool_t map,
  const struct acpi_rmrr_unit *rmrr,
+u16 bdf,//temp
  u32 flag)
 {
 unsigned long base_pfn = rmrr->base_address >> PAGE_SHIFT_4K;
@@ -1914,6 +1930,7 @@ static int rmrr_identity_mapping(struct
 if ( map )
 {
 ++mrmrr->count;
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] count %u\n", d->domain_id, base_pfn, 
end_pfn, mrmrr->count);//temp
 return 0;
 }
 
@@ -1928,6 +1945,7 @@ static int rmrr_identity_mapping(struct
 }
 
 list_del(&mrmrr->list);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] zapped (%d)\n", d->domain_id, 
mrmrr->base >> PAGE_SHIFT_4K, end_pfn, ret);//temp
 xfree(mrmrr);
 return ret;
 }
@@ -1941,11 +1959,30 @@ static int rmrr_identity_mapping(struct
 int err = set_identity_p2m_entry(d, base_pfn, p2m_access_rw, flag);
 
 if ( err )
+{//temp
+ printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] map error %d @ %lx\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, err, base_pfn);
 return err;
+} else {
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(!(base_pfn & 0xff) && ++cnt > thr) {
+  const struct pci_dev*pdev = pci_get_pdev(rmrr->segment, PCI_BUS(bdf), 
PCI_DEVFN2(bdf));
+  const struct acpi_drhd_unit*drhd = pdev ? acpi_find_matched_drhd_unit(pdev) 
: NULL;
+  thr |= cnt;
+  printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] mapped %lx\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, base_pfn);
+  if(drhd)
+   print_vtd_entries(drhd->iommu, PCI_BUS(bdf), PCI_DEVFN2(bdf), base_pfn);
+ }
+}
 base_pfn++;
 }
 
 mrmrr = xmalloc(struct mapped_rmrr);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] alloc -> %p\n", d->domain_id, 
rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, mrmrr);//temp
 if ( !mrmrr )
 return -ENOMEM;
 mrmrr->base = rmrr->base_address;
@@ -1987,7 +2024,7 @@ static int intel_iommu_add_device(u8 dev
  * Since RMRRs are always reserved in the e820 map for the hardware
  * domain, there shouldn't be a conflict.
  */
-ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, 0);
+ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, bdf, 0);
 if ( ret )
 dprintk(XENLOG_ERR VTDPREFIX, "d%d: RMRR mapping failed\n",
 pdev->domain->domain_id);
@@ -2032,7 +2069,7 @@ static int intel_iommu_remove_device(u8
  * Any flag is nothing to clear these mappings but here
  * its always safe and strict to set 0.
  */
-rmrr_identity_mapping(pdev->domain, 0, rmrr, 0);
+rmrr_identity_mapping(pdev->domain, 0, rmrr, bdf, 0);
 }
 
 return domain_context_unmap(pdev->domain, devfn, pdev);
@@ -2199,7 +2236,7 @@ static void __hwdom_init setup_hwdom_rmr
  * domain, there shouldn't be a conflict. So its always safe and
  * strict to set 0.
  */
-ret = rmrr_identity_mapping(d, 1, rmrr, 0);
+ret = rmrr_identity_mapping(d, 1, rmrr, bdf, 0);
 if ( ret )
 dprintk(XENLOG_ERR VTDPREFIX,
  "IOMMU: mapping reserved region failed\n");
@@ -2356,7 +2393,7 @@ static int reassign_device_ownership(
  * Any RMRR flag is always ignored when remove a device,
  * but its always safe and strict to set 0.
  */
-ret = rmrr_identity_mapping(source, 0, rmrr, 0);
+

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-19 Thread G.R. 
On Wed, Jan 18, 2017 at 12:34 AM, Jan Beulich  wrote:

> >>> On 17.01.17 at 16:08,  wrote:
> > I was lucky to capture the full log before it fills up my 100MB ring
> buffer
> > (in less than 2 seconds).
> > Please find the log in the attachment.
>
> Sadly nothing helpful in there; I'm a little puzzled though that the
> first thing we see is
>
> (XEN) [VT-D]iommu.c:909: iommu_fault_status: Fault Overflow
>
> which suggests there were (unlogged) faults already before.
>
> My primary suspicion right now is that you problem is due to the
> relatively large RMRR, as the first logged fault occurs on the first
> 2Mb boundary after the start of the RMRR. I'll therefore have to
> find time to create a debugging patch for you.
>
That's unfortunate! But anyway we stepped one small step ahead.
Waiting for your patch. I'll be offline for 3 days, will check back after
that.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-17 Thread Jan Beulich 
>>> On 17.01.17 at 16:08,  wrote:
> I was lucky to capture the full log before it fills up my 100MB ring buffer
> (in less than 2 seconds).
> Please find the log in the attachment.

Sadly nothing helpful in there; I'm a little puzzled though that the
first thing we see is

(XEN) [VT-D]iommu.c:909: iommu_fault_status: Fault Overflow

which suggests there were (unlogged) faults already before.

My primary suspicion right now is that you problem is due to the
relatively large RMRR, as the first logged fault occurs on the first
2Mb boundary after the start of the RMRR. I'll therefore have to
find time to create a debugging patch for you.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-17 Thread G.R. 
On Tue, Jan 17, 2017 at 8:54 PM, Jan Beulich  wrote:

> >>> On 17.01.17 at 11:49,  wrote:
> > I was trying to figure out if I followed your instruction properly.
> > My first attempt only resulted in a binary with similar size with my
> > previous one.
> > Probably something went wrong.
> > I put my source under /nas/src/xen, and I have a /nas/src/xen/.config
> file
> > for the
> > python layout knob according to the wiki.
> > My first attempt put th CONFIG_DEBUG=y line in the same file.
> > But now I suspect if I should use /nas/src/xen/xen/.config (note the
> double
> > 'xen').
>
> Yes indeed, that's the one. And you shouldn't add a new line, but
> instead edit the existing one (with CONFIG_DEBUG commented out).
>

Hi Jan, I think debug build works this time.
For unknown reason, debug version of hypervisor will cause domU hang if
gfx-passthrough=1 is present (traditional device model).
But fortunately commenting out that line could still reproduce the IOMMU
fault.
I was lucky to capture the full log before it fills up my 100MB ring buffer
(in less than 2 seconds).
Please find the log in the attachment.

Thanks,
G.R.


dmsg.log.bz2
Description: BZip2 compressed data
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-17 Thread Jan Beulich 
>>> On 17.01.17 at 11:49,  wrote:
> On Tue, Jan 17, 2017 at 12:11 AM, Jan Beulich  wrote:
> 
>> >>> On 16.01.17 at 16:15,  wrote:
>> > On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich  wrote:
>> >
>> >> For building a debug hypervisor, all you need to do is set
>> >> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> >> knobs to avoid log flooding - after all you've asked for the
>> >> verbosity via "iommu=verbose,debug".
>> >>
>> > I assume I do not need to redo the ./configure here.
>> > And I assume the xen/.config here refers to the root of the repos instead
>> > of the xen.git/xen subdirectory?
>>
>> I don't understand - I'd normally assume the two to be the same
>> (with just different context made visible).
>>
> I was trying to figure out if I followed your instruction properly.
> My first attempt only resulted in a binary with similar size with my
> previous one.
> Probably something went wrong.
> I put my source under /nas/src/xen, and I have a /nas/src/xen/.config file
> for the
> python layout knob according to the wiki.
> My first attempt put th CONFIG_DEBUG=y line in the same file.
> But now I suspect if I should use /nas/src/xen/xen/.config (note the double
> 'xen').

Yes indeed, that's the one. And you shouldn't add a new line, but
instead edit the existing one (with CONFIG_DEBUG commented out).

>> I couldn't find obvious debug knob in the gcc command-line, even though
>> the
>> > build is with -O1.
>>
>> Nor do I understand this remark.
>>
> I checked the GCC command-lines during build process with the .config
> change, with the expectation of something like -DDEBUG -g etc.
> But actually I saw none of them, only saw -O1.

The primary thing would be for -DNDEBUG to no longer be there (it
should be there for the non-debug build).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-17 Thread G.R. 
On Tue, Jan 17, 2017 at 12:11 AM, Jan Beulich  wrote:

> >>> On 16.01.17 at 16:15,  wrote:
> > On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich  wrote:
> >
> >> For building a debug hypervisor, all you need to do is set
> >> CONFIG_DEBUG=y in xen/.config. I don't think there are any
> >> knobs to avoid log flooding - after all you've asked for the
> >> verbosity via "iommu=verbose,debug".
> >>
> > I assume I do not need to redo the ./configure here.
> > And I assume the xen/.config here refers to the root of the repos instead
> > of the xen.git/xen subdirectory?
>
> I don't understand - I'd normally assume the two to be the same
> (with just different context made visible).
>
I was trying to figure out if I followed your instruction properly.
My first attempt only resulted in a binary with similar size with my
previous one.
Probably something went wrong.
I put my source under /nas/src/xen, and I have a /nas/src/xen/.config file
for the
python layout knob according to the wiki.
My first attempt put th CONFIG_DEBUG=y line in the same file.
But now I suspect if I should use /nas/src/xen/xen/.config (note the double
'xen').

> I couldn't find obvious debug knob in the gcc command-line, even though
> the
> > build is with -O1.
>
> Nor do I understand this remark.
>
I checked the GCC command-lines during build process with the .config
change, with the expectation of something like -DDEBUG -g etc.
But actually I saw none of them, only saw -O1.

G.R.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread Jan Beulich 
>>> On 16.01.17 at 16:21,  wrote:
> BTW, before I generate more verbose && complete debug log, just want to
> update that I also see the following in dom0 (without attempting any
> pass-through to the IGD device)
> But this time the log is not flooding at all. Not sure if this is relevant
> to what I see from the domU with pci pass-through.
> 
> (XEN) Bogus DMIBAR 0xfed18001 on :00:00.0
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 73, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 73, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 73, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 73, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 73, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

This address is clearly neither RAM nor covered by any RMRR, so
the question is why there is I/O being generated to it. Is it perhaps
named by one of the device's BARs?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread Jan Beulich 
>>> On 16.01.17 at 16:15,  wrote:
> On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich  wrote:
> 
>> >>> On 16.01.17 at 14:43,  wrote:
>> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:
>> >> >>> On 16.01.17 at 10:25,  wrote:
>> > The fault log itself is really flooding. With a small 4MB ring buffer, I
>> > wasn't able to capture how it begins.
>>
>> If you can't set up a serial console, grow the ring buffer.
>>
> Larger ring buffer seems to be the only option to me.
> Seems that 'serial console' needs to be something physical.

Indeed.

>> > That RMRR setup has changed dramatically (from being basically
>> >> non-existent in the older versions), especially for USB devices (I
>> >> don't think I can conclude what type of device :02:00.0 is).
>> >> There are messages logged with various failures in that process,
>> >> but some would be issued by debug hypervisors only. A good
>> >> first step (before possibly doing actual code instrumentation)
>> >> would therefore be to retry with a debug hypervisor, and post
>> >> the full log (huge amounts of trailing IOMMU fault messages may
>> >> of course be stripped as long as they're sufficiently similar, to
>> >> keep the overall log size manageable).
>> >>
>> > I can give it a try when I get some spare time.
>> > Could you show me the flow to build a debug hypervisor and the most
>> > relevant debug knobs to avoid log flooding?
>>
>> For building a debug hypervisor, all you need to do is set
>> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> knobs to avoid log flooding - after all you've asked for the
>> verbosity via "iommu=verbose,debug".
>>
> I assume I do not need to redo the ./configure here.
> And I assume the xen/.config here refers to the root of the repos instead
> of the xen.git/xen subdirectory?

I don't understand - I'd normally assume the two to be the same
(with just different context made visible).

> I couldn't find obvious debug knob in the gcc command-line, even though the
> build is with -O1.

Nor do I understand this remark.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread G.R. 
BTW, before I generate more verbose && complete debug log, just want to
update that I also see the following in dom0 (without attempting any
pass-through to the IGD device)
But this time the log is not flooding at all. Not sure if this is relevant
to what I see from the domU with pci pass-through.

(XEN) Bogus DMIBAR 0xfed18001 on :00:00.0
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
73, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
73, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
73, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
73, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
73, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set


On Mon, Jan 16, 2017 at 11:15 PM, G.R. 
wrote:

>
>
> On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich  wrote:
>
>> >>> On 16.01.17 at 14:43,  wrote:
>> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:
>> >> >>> On 16.01.17 at 10:25,  wrote:
>> > The fault log itself is really flooding. With a small 4MB ring buffer, I
>> > wasn't able to capture how it begins.
>>
>> If you can't set up a serial console, grow the ring buffer.
>>
> Larger ring buffer seems to be the only option to me.
> Seems that 'serial console' needs to be something physical.
>
>
>> > That RMRR setup has changed dramatically (from being basically
>> >> non-existent in the older versions), especially for USB devices (I
>> >> don't think I can conclude what type of device :02:00.0 is).
>> >> There are messages logged with various failures in that process,
>> >> but some would be issued by debug hypervisors only. A good
>> >> first step (before possibly doing actual code instrumentation)
>> >> would therefore be to retry with a debug hypervisor, and post
>> >> the full log (huge amounts of trailing IOMMU fault messages may
>> >> of course be stripped as long as they're sufficiently similar, to
>> >> keep the overall log size manageable).
>> >>
>> > I can give it a try when I get some spare time.
>> > Could you show me the flow to build a debug hypervisor and the most
>> > relevant debug knobs to avoid log flooding?
>>
>> For building a debug hypervisor, all you need to do is set
>> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> knobs to avoid log flooding - after all you've asked for the
>> verbosity via "iommu=verbose,debug".
>>
> I assume I do not need to redo the ./configure here.
> And I assume the xen/.config here refers to the root of the repos instead
> of the xen.git/xen subdirectory?
> I couldn't find obvious debug knob in the gcc command-line, even though
> the build is with -O1.
>
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread G.R. 
On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich  wrote:

> >>> On 16.01.17 at 14:43,  wrote:
> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:
> >> >>> On 16.01.17 at 10:25,  wrote:
> > The fault log itself is really flooding. With a small 4MB ring buffer, I
> > wasn't able to capture how it begins.
>
> If you can't set up a serial console, grow the ring buffer.
>
Larger ring buffer seems to be the only option to me.
Seems that 'serial console' needs to be something physical.


> > That RMRR setup has changed dramatically (from being basically
> >> non-existent in the older versions), especially for USB devices (I
> >> don't think I can conclude what type of device :02:00.0 is).
> >> There are messages logged with various failures in that process,
> >> but some would be issued by debug hypervisors only. A good
> >> first step (before possibly doing actual code instrumentation)
> >> would therefore be to retry with a debug hypervisor, and post
> >> the full log (huge amounts of trailing IOMMU fault messages may
> >> of course be stripped as long as they're sufficiently similar, to
> >> keep the overall log size manageable).
> >>
> > I can give it a try when I get some spare time.
> > Could you show me the flow to build a debug hypervisor and the most
> > relevant debug knobs to avoid log flooding?
>
> For building a debug hypervisor, all you need to do is set
> CONFIG_DEBUG=y in xen/.config. I don't think there are any
> knobs to avoid log flooding - after all you've asked for the
> verbosity via "iommu=verbose,debug".
>
I assume I do not need to redo the ./configure here.
And I assume the xen/.config here refers to the root of the repos instead
of the xen.git/xen subdirectory?
I couldn't find obvious debug knob in the gcc command-line, even though the
build is with -O1.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread George Dunlap 
On Mon, Jan 16, 2017 at 1:43 PM, G.R.  wrote:
> On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:
>>
>> >>> On 16.01.17 at 10:25,  wrote:
>> > Here are some relevant logs, please help comment what's going on here
>> > and
>> > what's the next step of diagnose.
>> > It appears that the fault address 0xcfxx falls within the host RMRR
>> > region.
>>
>> Might be a problem in the RMRR setup itself, when the guest gets
>> the device assigned. But I'm not sure, as you've provided only
>> fragments of the log, instead of the full one (allowing to see in
>> which order the messages got logged). In any event the addresses
>> are, as you say, properly within the device's RMRR range.
>
> Thanks for your quick reply, Jan.
> I meant to provide full log through third party service like pastebin but my
> network at work just get it blocked.
> Here it is: http://pastebin.com/RHVzhR6H

FWIW you can also attach the log as a text file.

 -George

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread Jan Beulich 
>>> On 16.01.17 at 14:43,  wrote:
> On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:
>> >>> On 16.01.17 at 10:25,  wrote:
>> > Here are some relevant logs, please help comment what's going on here and
>> > what's the next step of diagnose.
>> > It appears that the fault address 0xcfxx falls within the host RMRR
>> > region.
>>
>> Might be a problem in the RMRR setup itself, when the guest gets
>> the device assigned. But I'm not sure, as you've provided only
>> fragments of the log, instead of the full one (allowing to see in
>> which order the messages got logged). In any event the addresses
>> are, as you say, properly within the device's RMRR range.
>>
> Thanks for your quick reply, Jan.
> I meant to provide full log through third party service like pastebin but
> my network at work just get it blocked.
> Here it is: http://pastebin.com/RHVzhR6H 
> Note that the log here is before the fault issue shows up.

Well, it stopping before the first fault occurs makes it only
marginally bette than what you've provided so far.

> The fault log itself is really flooding. With a small 4MB ring buffer, I
> wasn't able to capture how it begins.

If you can't set up a serial console, grow the ring buffer.

> That RMRR setup has changed dramatically (from being basically
>> non-existent in the older versions), especially for USB devices (I
>> don't think I can conclude what type of device :02:00.0 is).
>> There are messages logged with various failures in that process,
>> but some would be issued by debug hypervisors only. A good
>> first step (before possibly doing actual code instrumentation)
>> would therefore be to retry with a debug hypervisor, and post
>> the full log (huge amounts of trailing IOMMU fault messages may
>> of course be stripped as long as they're sufficiently similar, to
>> keep the overall log size manageable).
>>
> I can give it a try when I get some spare time.
> Could you show me the flow to build a debug hypervisor and the most
> relevant debug knobs to avoid log flooding?

For building a debug hypervisor, all you need to do is set
CONFIG_DEBUG=y in xen/.config. I don't think there are any
knobs to avoid log flooding - after all you've asked for the
verbosity via "iommu=verbose,debug".

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread G.R. 
On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich  wrote:

> >>> On 16.01.17 at 10:25,  wrote:
> > Here are some relevant logs, please help comment what's going on here and
> > what's the next step of diagnose.
> > It appears that the fault address 0xcfxx falls within the host RMRR
> > region.
>
> Might be a problem in the RMRR setup itself, when the guest gets
> the device assigned. But I'm not sure, as you've provided only
> fragments of the log, instead of the full one (allowing to see in
> which order the messages got logged). In any event the addresses
> are, as you say, properly within the device's RMRR range.
>
Thanks for your quick reply, Jan.
I meant to provide full log through third party service like pastebin but
my network at work just get it blocked.
Here it is: http://pastebin.com/RHVzhR6H
Note that the log here is before the fault issue shows up.
As I already mentioned, there are two domUs in the log and the suffering
one is dom2.

The fault log itself is really flooding. With a small 4MB ring buffer, I
wasn't able to capture how it begins.
>From what I can tell, some one is scanning through the region in a fixed
pace. (in general, with some ping-pong occasionally)
The content from print_vtd_entries if fairly stable. This is what I get
from 'sort|uniq -c' post-processing, after removing line with fault address:
   7219 (XEN) context[10] = 1_2215f6001
   7219 (XEN) context = 830251bcb000
   5259 (XEN) l2[7d] = 0
   5259 (XEN) l2[7d] not present
   1961 (XEN) l2[7e] = 0
   1961 (XEN) l2[7e] not present
   7219 (XEN) l2 = 830221476000
   5258 (XEN) l2_index = 7d
   1961 (XEN) l2_index = 7e
   7219 (XEN) l3[3] = 221476003
   7219 (XEN) l3 = 8302215f6000
   7219 (XEN) l3_index = 3
   7219 (XEN) root_entry[0] = 251bcb001
   7219 (XEN) root_entry = 8304152e9000
   7219 (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

The fault address pattern could be found here: http://pastebin.com/rWWH3QUG
(Note that I dropped redundant columns to fit the size limitation...)

And here is a list of my host PCI devices:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd
Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset
Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High
Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI
Express Root Port 1 (rev c4)
00:1c.3 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI
Express Root Port 4 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller
(rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset
Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus
Controller (rev 04)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

That RMRR setup has changed dramatically (from being basically
> non-existent in the older versions), especially for USB devices (I
> don't think I can conclude what type of device :02:00.0 is).
> There are messages logged with various failures in that process,
> but some would be issued by debug hypervisors only. A good
> first step (before possibly doing actual code instrumentation)
> would therefore be to retry with a debug hypervisor, and post
> the full log (huge amounts of trailing IOMMU fault messages may
> of course be stripped as long as they're sufficiently similar, to
> keep the overall log size manageable).
>
I can give it a try when I get some spare time.
Could you show me the flow to build a debug hypervisor and the most
relevant debug knobs to avoid log flooding?


>
> > However, the hvmloader is setting up memory region starting from address
> > 0xe000.
> > Is the hvmloader memory map relevant here?
>
> No, it shouldn't be.
>
> > Unfortunately the iommu.c does not provide detailed log on the mapping
> > except a simple 'd2:PCI: map :00:02.0'
>
> If we made it so, it would become unreasonably verbose.
>
> Jan
>
>
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] IOMMU fault with IGD passthrough setup on XEN 4.8.0

2017-01-16 Thread Jan Beulich 
>>> On 16.01.17 at 10:25,  wrote:
> Here are some relevant logs, please help comment what's going on here and
> what's the next step of diagnose.
> It appears that the fault address 0xcfxx falls within the host RMRR
> region.

Might be a problem in the RMRR setup itself, when the guest gets
the device assigned. But I'm not sure, as you've provided only
fragments of the log, instead of the full one (allowing to see in
which order the messages got logged). In any event the addresses
are, as you say, properly within the device's RMRR range.

That RMRR setup has changed dramatically (from being basically
non-existent in the older versions), especially for USB devices (I
don't think I can conclude what type of device :02:00.0 is).
There are messages logged with various failures in that process,
but some would be issued by debug hypervisors only. A good
first step (before possibly doing actual code instrumentation)
would therefore be to retry with a debug hypervisor, and post
the full log (huge amounts of trailing IOMMU fault messages may
of course be stripped as long as they're sufficiently similar, to
keep the overall log size manageable).

> However, the hvmloader is setting up memory region starting from address
> 0xe000.
> Is the hvmloader memory map relevant here?

No, it shouldn't be.

> Unfortunately the iommu.c does not provide detailed log on the mapping
> except a simple 'd2:PCI: map :00:02.0'

If we made it so, it would become unreasonably verbose.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel