Re: [Xen-devel] [BUG] win2008 guest cannot get ip through sriov

[Xu, Quan]

On May 27, 2016 10:06 PM, Jan Beulich  wrote:
> >>> On 27.05.16 at 15:34,  wrote:
> > On Fri, May 27, 2016 at 06:16:30AM -0600, Jan Beulich wrote:
> >> >>> On 27.05.16 at 12:39,  wrote:
> >> > Is this a regression? Does it work on previous versions of Xen?
> >>
> >> I think this is what was already reported by other Intel people, see
> >> e.g. Quan's most recent reply:
> >> http://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01896.
> >> html It is not clear where the problem is, and not seeing the issue
> >> myself makes it hard to analyze. In any event this quite likely is a
> >> regression.
> >>
> >
> > My reading of that email thread and all relevant links (including the
> > KVM bug report) is that there is a regression vf driver, but not in Xen.
> 
> Just from reading that I would tend to agree. But the report here is about
> Win2K8.
> 


Jan, 
Do you know which commit is a regression one? I try to find out the regression 
commit.  That may be helpful to find out the root cause.

Btw, some feedback from QA team, rhel 6.4 VM  doesn't work, but rhel 7.2 VM 
does.

> Jan
> 
> > There isn't enough evidence to rule out a bug in Xen, but over all
> > I'm inclined to think this bug is not in Xen given all the things presented.
> >
> > Wei.
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

On June 01, 2016 6:24 PM, Jan Beulich  wrote:
> >>> On 31.05.16 at 15:57,  wrote:
> > --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
> > +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
> > @@ -295,12 +297,22 @@ static void __hwdom_init
> amd_iommu_hwdom_init(struct domain *d)
> >   * a pfn_valid() check would seem desirable here.
> >   */
> >  if ( mfn_valid(pfn) )
> > -amd_iommu_map_page(d, pfn, pfn,
> > -   IOMMUF_readable|IOMMUF_writable);
> > +{
> > +int ret;
> > +
> > +ret = amd_iommu_map_page(d, pfn, pfn,
> > +
> > + IOMMUF_readable|IOMMUF_writable);
> 
> Same here as for the earlier patch regarding assignment vs initializer.
> 

I'll fix it in next v7.

> But overall the entire change to this function seems to rather belong into
> patch 2.

Indeed.

 As would a respective change to vtd_set_hwdom_mapping(), which
> I'm not sure which patch you've put that in.
> 

Sorry,  I missed it. I indeed it need to fix it as similar as above.
I wonder whether I could add a __must_check annotation to iommu_map_page() or 
not, as which may be inconsistent with iommu_unmap_page().

these modifications should belong to patch 2.

> > --- a/xen/include/xen/iommu.h
> > +++ b/xen/include/xen/iommu.h
> > @@ -166,8 +166,8 @@ struct iommu_ops {  #endif /* HAS_PCI */
> >
> >  void (*teardown)(struct domain *d);
> > -int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
> > -unsigned int flags);
> > +int __must_check (*map_page)(struct domain *d, unsigned long gfn,
> > + unsigned long mfn, unsigned int
> > + flags);
> 
> With this and with the rule set forth in the context of the discussion of v5,
> iommu_map_page() (as well as any other caller of this hook that do not
> themselves _consume_ the error [e.g. hwdom init ones]) should become or
> already be __must_check, which afaict isn't the case.

But does this rule also apply to these 'void' annotation functions?  .e.g, in 
the call tree of hwdom init ones / domain crash ones, we are no need to bubble 
up
error code, leaving these void annotation as is.

> The same then, btw.,
> applies to patch 3, and hence I have to withdraw the R-b that you've got
> there.
> 

I find these callers are grant_table/mm, and we limit __must_check annotation 
to IOMMU functions for this patch set..
So I think I can remain R-b as is for patch 3. 

btw, your R-b is a very expensive tag to me, and I really don't want to drop 
it. :):)..

Quan



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On June 01, 2016 6:05 PM, Jan Beulich  wrote:
> >>> On 31.05.16 at 15:57,  wrote:
> > --- a/xen/arch/x86/mm/p2m-pt.c
> > +++ b/xen/arch/x86/mm/p2m-pt.c
> > @@ -673,6 +673,8 @@ p2m_pt_set_entry(struct p2m_domain *p2m,
> unsigned long gfn, mfn_t mfn,
> >  if ( iommu_enabled && need_iommu(p2m->domain) &&
> >   (iommu_old_flags != iommu_pte_flags || old_mfn != mfn_x(mfn)) )
> >  {
> > +int ret;
> > +
> >  if ( iommu_use_hap_pt(p2m->domain) )
> >  {
> >  if ( iommu_old_flags )
> > @@ -680,11 +682,27 @@ p2m_pt_set_entry(struct p2m_domain *p2m,
> unsigned long gfn, mfn_t mfn,
> >  }
> >  else if ( iommu_pte_flags )
> >  for ( i = 0; i < (1UL << page_order); i++ )
> > -iommu_map_page(p2m->domain, gfn + i, mfn_x(mfn) + i,
> > -   iommu_pte_flags);
> > +{
> > +ret = iommu_map_page(p2m->domain, gfn + i, mfn_x(mfn) + i,
> > + iommu_pte_flags);
> > +if ( unlikely(ret) )
> > +{
> > +while ( i-- )
> > +iommu_unmap_page(p2m->domain, gfn + i);
> > +
> > +if ( !rc )
> > +rc = ret;
> > +
> > +break;
> > +}
> > +}
> 
> So why do you not use the same code structure here as you do on the EPT
> side? 

I try to modify these code in a slight way, but if you point out some extra 
issue, I am pleased to fix it.
Furthermore, I was not sure whether I made an arbitrary decision here to add 
the condition 'rc == 0' or not,
Even I was aware of that the 'rc' is zero when the code run here..

> I.e. do away with using "ret" altogether, moving it all into ...
> 
> >  else
> >  for ( i = 0; i < (1UL << page_order); i++ )
> > -iommu_unmap_page(p2m->domain, gfn + i);
> > +{
> > +ret = iommu_unmap_page(p2m->domain, gfn + i);
> > +if ( !rc )
> > +rc = ret;
> > +}
> 
> ... this extremely narrow scope?
> 

What about this fix:

--- a/xen/arch/x86/mm/p2m-pt.c
+++ b/xen/arch/x86/mm/p2m-pt.c
@@ -670,7 +670,7 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
  && (gfn + (1UL << page_order) - 1 > p2m->max_mapped_pfn) )
 p2m->max_mapped_pfn = gfn + (1UL << page_order) - 1;

-if ( iommu_enabled && need_iommu(p2m->domain) &&
+if ( rc == 0 && iommu_enabled && need_iommu(p2m->domain) &&
  (iommu_old_flags != iommu_pte_flags || old_mfn != mfn_x(mfn)) )
 {
 if ( iommu_use_hap_pt(p2m->domain) )
@@ -678,13 +678,33 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long 
gfn, mfn_t mfn,
 if ( iommu_old_flags )
 amd_iommu_flush_pages(p2m->domain, gfn, page_order);
 }
-else if ( iommu_pte_flags )
-for ( i = 0; i < (1UL << page_order); i++ )
-iommu_map_page(p2m->domain, gfn + i, mfn_x(mfn) + i,
-   iommu_pte_flags);
 else
-for ( i = 0; i < (1UL << page_order); i++ )
-iommu_unmap_page(p2m->domain, gfn + i);
+{
+if ( iommu_pte_flags )
+for ( i = 0; i < (1UL << page_order); i++ )
+{
+rc = iommu_map_page(p2m->domain, gfn + i, mfn_x(mfn) + i,
+iommu_pte_flags);
+if ( unlikely(rc) )
+{
+while ( i-- )
+iommu_unmap_page(p2m->domain, gfn + i);
+
+break;
+}
+}
+else
+{
+int ret;
+
+for ( i = 0; i < (1UL << page_order); i++ )
+{
+ret = iommu_unmap_page(p2m->domain, gfn + i);
+if ( !rc )
+rc = ret;
+}
+}
+}
 }

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones)

[Xu, Quan]

On June 01, 2016 6:39 PM, Jan Beulich  wrote:
> >>> On 31.05.16 at 15:57,  wrote:
> >  static int device_power_down(void)
> >  {
> > -console_suspend();
> > +if ( console_suspend() )
> > +return SAVED_NONE;
> >
> > -time_suspend();
> > +if ( time_suspend() )
> > +return SAVED_CONSOLE;
> >
> > -i8259A_suspend();
> > +if ( i8259A_suspend() )
> > +return SAVED_TIME;
> >
> > +/* ioapic_suspend cannot fail */
> >  ioapic_suspend();
> >
> > -iommu_suspend();
> > +if ( iommu_suspend() )
> > +return SAVED_IOAPIC;
> >
> > -lapic_suspend();
> > +if ( lapic_suspend() )
> > +return SAVED_IOMMU;
> >
> > -return 0;
> > +return SAVED_NONE;
> 
> SAVED_ALL

Agreed. 
I was disturbed by the below 'if ( error > 0 )'.

> 
> > @@ -169,6 +203,10 @@ static int enter_state(u32 state)
> >  {
> >  printk(XENLOG_ERR "Some devices failed to power down.");
> >  system_state = SYS_STATE_resume;
> > +
> > +if ( error > 0 )
> > +device_power_up(error);
> 
> if ( error != SAVED_NONE )
> 
> (Or really you could just call this without any if().)

I prefer to drop this if().

> 
> > @@ -2389,16 +2393,25 @@ static int intel_iommu_group_id(u16 seg, u8
> > bus, u8 devfn)  }
> >
> >  static u32 iommu_state[MAX_IOMMUS][MAX_IOMMU_REGS];
> > -static void vtd_suspend(void)
> > +
> > +static int __must_check vtd_suspend(void)
> >  {
> >  struct acpi_drhd_unit *drhd;
> >  struct iommu *iommu;
> >  u32i;
> > +int rc = 0;
> 
> Pointless initializer.
> 

Indeed, if "return 0" to make obvious that no error path comes at the end of 
this function.

> >  if ( !iommu_enabled )
> > -return;
> > +return 0;
> >
> > -iommu_flush_all();
> > +rc = iommu_flush_all();
> > +if ( unlikely(rc) )
> > +{
> > +printk(XENLOG_WARNING VTDPREFIX
> > +   " suspend: IOMMU flush all failed: %d\n", rc);
> > +
> > +return rc;
> > +}
> >
> >  for_each_drhd_unit ( drhd )
> >  {
> > @@ -2427,6 +2440,8 @@ static void vtd_suspend(void)
> >  if ( !iommu_intremap && iommu_qinval )
> >  disable_qinval(iommu);
> >  }
> > +
> > +return rc;
> >  }
> 
> Perhaps better "return 0" to make obvious that no error path comes here.
> 

Agreed.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [Patch v6 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On June 01, 2016 11:37 PM, Jan Beulich  wrote:
> >>> On 31.05.16 at 15:57,  wrote:
> > @@ -1542,14 +1600,36 @@ int domain_context_unmap_one(
> >  return -EINVAL;
> >  }
> >
> > -if ( iommu_flush_context_device(iommu, iommu_domid,
> > -(((u16)bus) << 8) | devfn,
> > -DMA_CCMD_MASK_NOBIT, 0) )
> > -iommu_flush_write_buffer(iommu);
> > -else
> > +rc = iommu_flush_context_device(iommu, iommu_domid,
> > +PCI_BDF2(bus, devfn),
> > +DMA_CCMD_MASK_NOBIT, 0);
> > +
> > +/*
> > + * The current logic for rc returns:
> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > + *   - zero  on success.
> > + *   - negative  on failure. Continue to flush IOMMU IOTLB on a
> > + *   best effort basis.
> > + *
> > + * Moreover, IOMMU flush handlers flush_context_qi or flush_iotlb_qi
> > + * (or flush_context_reg and flush_iotlb_reg, deep functions in the
> > + * call trees of iommu_flush_context_device and iommu_flush_iotlb_dsi)
> > + * are with the same logic to bubble up positive return value.
> > + */
> 
> This is the 3rd instance of that comment. I'd prefer the latter ones to simply
> refer to the first one, but I'll obviously leave it to the maintainers to 
> decide.
> 

Kevin / Feng .. What's your opinion?

> With those cosmetic issues taken care of
> Reviewed-by: Jen Beulich 
> 

-Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v11 2/3] vt-d: synchronize for Device-TLB flush one by one

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Today we do Device-TLB flush synchronization after issuing flush
requests for all ATS devices belonging to a VM. Doing so however
imposes a limitation, i.e. that we can not figure out which flush
request is blocked in the flush queue list, based on VT-d spec.

To prepare correct Device-TLB flush timeout handling in next patch,
we change the behavior to synchronize for every Device-TLB flush
request. So the Device-TLB flush interface is changed a little bit,
by checking timeout within the function instead of outside of function.

Accordingly we also do a similar change for flush interfaces of
IOTLB/IEC/Context, i.e. moving synchronization into the function.
Since there is no user of a non-synced interface, we just rename
existing ones with _sync suffix.

v11:
   1. Add __must_check annotation to invalidate_sync().
   2. Remove the redundant invalidate_sync() in flush_iotlb_qi().
   3. Refactor invalidate_sync() to indicate whether we need to flush
  device IOTLB or not (I'll change it back in next patch, as I'll
  add a specific function - dev_invalidate_sync() for device IOTLB
  invalidation).

Signed-off-by: Quan Xu <quan...@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h  |  5 +--
 xen/drivers/passthrough/vtd/qinval.c  | 65 ---
 xen/drivers/passthrough/vtd/x86/ats.c |  8 ++---
 3 files changed, 45 insertions(+), 33 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 6772839..45357f2 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -59,8 +59,9 @@ int ats_device(const struct pci_dev *, const struct 
acpi_drhd_unit *);
 int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
-int qinval_device_iotlb(struct iommu *iommu,
-u32 max_invs_pend, u16 sid, u16 size, u64 addr);
+int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
+  u32 max_invs_pend,
+  u16 sid, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 1a37565..9116588 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -30,6 +30,9 @@
 
 #define IOMMU_QI_TIMEOUT MILLISECS(1)
 
+static int __must_check invalidate_sync(struct iommu *iommu,
+bool_t flush_dev_iotlb);
+
 static void print_qi_regs(struct iommu *iommu)
 {
 u64 val;
@@ -69,8 +72,10 @@ static void qinval_update_qtail(struct iommu *iommu, 
unsigned int index)
 dmar_writeq(iommu->reg, DMAR_IQT_REG, (val << QINVAL_INDEX_SHIFT));
 }
 
-static void queue_invalidate_context(struct iommu *iommu,
-u16 did, u16 source_id, u8 function_mask, u8 granu)
+static int __must_check queue_invalidate_context_sync(struct iommu *iommu,
+  u16 did, u16 source_id,
+  u8 function_mask,
+  u8 granu)
 {
 unsigned long flags;
 unsigned int index;
@@ -97,10 +102,14 @@ static void queue_invalidate_context(struct iommu *iommu,
 spin_unlock_irqrestore(>register_lock, flags);
 
 unmap_vtd_domain_page(qinval_entries);
+
+return invalidate_sync(iommu, 0);
 }
 
-static void queue_invalidate_iotlb(struct iommu *iommu,
-u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
+static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
+u8 granu, u8 dr, u8 dw,
+u16 did, u8 am, u8 ih,
+u64 addr)
 {
 unsigned long flags;
 unsigned int index;
@@ -130,6 +139,8 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
 unmap_vtd_domain_page(qinval_entries);
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
+
+return invalidate_sync(iommu, 0);
 }
 
 static int __must_check queue_invalidate_wait(struct iommu *iommu,
@@ -200,8 +211,9 @@ static int __must_check invalidate_sync(struct iommu *iommu,
 return 0;
 }
 
-int qinval_device_iotlb(struct iommu *iommu,
-u32 max_invs_pend, u16 sid, u16 size, u64 addr)
+int qinval_device_iotlb_sync(struct iommu *iommu,
+ u32 max_invs_pend,
+ u16 sid, u16 size, u64 addr)
 {
 unsigned long flags;
 unsigned int index;
@@ -230,15 +242,17 @@ int qinval_device_iotlb(struct iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
 
-return 0;
+r

[Xen-devel] [Patch v11 1/3] IOMMU: add a timeout parameter for device IOTLB invalidation

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of
the device IOTLB invalidation in milliseconds. By default, the
timeout is 1ms, which can be boot-time changed.

Add a __must_check annotation. The followup patch titled
'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck.
That is the other callers of this routine (two or three levels up)
ignore the return code. This patch does not address this but the
other does.

v11: Change the timeout parameter from 'vtd_qi_timeout' to
'iommu_dev_iotlb_timeout', which is not only for VT-d device
IOTLB invalidation, but also for other IOMMU implementations.

Signed-off-by: Quan Xu <quan...@intel.com>
---
 docs/misc/xen-command-line.markdown  |  9 +
 xen/drivers/passthrough/iommu.c  |  3 +++
 xen/drivers/passthrough/vtd/qinval.c | 34 +++---
 xen/include/xen/iommu.h  |  2 ++
 4 files changed, 37 insertions(+), 11 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown 
b/docs/misc/xen-command-line.markdown
index b4bae11..34a0f9c 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -996,6 +996,15 @@ debug hypervisor only).
 
 >> Enable IOMMU debugging code (implies `verbose`).
 
+### iommu\_dev\_iotlb\_timeout
+> `= `
+
+> Default: `1`
+
+Specify the timeout of the device IOTLB invalidation in milliseconds.
+By default, the timeout is 1 ms. When you see error 'Queue invalidate
+wait descriptor timed out', try increasing this value.
+
 ### iommu\_inclusive\_mapping (VT-d)
 > `= `
 
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 098b601..e12de69 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -24,6 +24,9 @@
 static void parse_iommu_param(char *s);
 static void iommu_dump_p2m_table(unsigned char key);
 
+unsigned int __read_mostly iommu_dev_iotlb_timeout = 1;
+integer_param("iommu_dev_iotlb_timeout", iommu_dev_iotlb_timeout);
+
 /*
  * The 'iommu' parameter enables the IOMMU.  Optional comma separated
  * value may contain:
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index aa7841a..1a37565 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -28,6 +28,8 @@
 #include "vtd.h"
 #include "extern.h"
 
+#define IOMMU_QI_TIMEOUT MILLISECS(1)
+
 static void print_qi_regs(struct iommu *iommu)
 {
 u64 val;
@@ -130,10 +132,10 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
 spin_unlock_irqrestore(>register_lock, flags);
 }
 
-static int queue_invalidate_wait(struct iommu *iommu,
-u8 iflag, u8 sw, u8 fn)
+static int __must_check queue_invalidate_wait(struct iommu *iommu,
+  u8 iflag, u8 sw, u8 fn,
+  bool_t flush_dev_iotlb)
 {
-s_time_t start_time;
 volatile u32 poll_slot = QINVAL_STAT_INIT;
 unsigned int index;
 unsigned long flags;
@@ -163,14 +165,21 @@ static int queue_invalidate_wait(struct iommu *iommu,
 /* Now we don't support interrupt method */
 if ( sw )
 {
+s_time_t timeout;
+
 /* In case all wait descriptor writes to same addr with same data */
-start_time = NOW();
+timeout = flush_dev_iotlb ?
+  (NOW() + iommu_dev_iotlb_timeout * MILLISECS(1)) :
+  (NOW() + IOMMU_QI_TIMEOUT);
+
 while ( poll_slot != QINVAL_STAT_DONE )
 {
-if ( NOW() > (start_time + DMAR_OPERATION_TIMEOUT) )
+if ( NOW() > timeout )
 {
 print_qi_regs(iommu);
-panic("queue invalidate wait descriptor was not executed");
+printk(XENLOG_WARNING VTDPREFIX
+   " Queue invalidate wait descriptor timed out\n");
+return -ETIMEDOUT;
 }
 cpu_relax();
 }
@@ -180,12 +189,14 @@ static int queue_invalidate_wait(struct iommu *iommu,
 return -EOPNOTSUPP;
 }
 
-static int invalidate_sync(struct iommu *iommu)
+static int __must_check invalidate_sync(struct iommu *iommu,
+bool_t flush_dev_iotlb)
 {
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
 if ( qi_ctrl->qinval_maddr )
-return queue_invalidate_wait(iommu, 0, 1, 1);
+return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
+
 return 0;
 }
 
@@ -254,7 +265,7 @@ static int __iommu_flush_iec(struct iommu *iommu, u8 granu, 
u8 im, u16 iidx)
 int ret;
 
 queue_invalidate_iec(iommu, granu, im, iidx);
-ret = invalidate_sync(iommu);
+ret = invalidate_sync(iommu, 0);
 /*
  * reading vt-d architecture register will ensure
  * draining happens in implementation independent

[Xen-devel] [Patch v11 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

If Device-TLB flush timed out, we hide the target ATS device
immediately and crash the domain owning this ATS device. If
impacted domain is hardware domain, just throw out a warning.

By hiding the device, we make sure it can't be assigned to any
domain any longer (see device_assigned).

v11:
   1. Pass down struct pci_dev *, instead of SBDF inside struct
  pci_ats_dev.
   2. change invalidate_sync() back as I add a specific function
  - dev_invalidate_sync() for device IOTLB.

Signed-off-by: Quan Xu <quan...@intel.com>
---
 xen/drivers/passthrough/pci.c |  6 +--
 xen/drivers/passthrough/vtd/extern.h  |  5 +-
 xen/drivers/passthrough/vtd/qinval.c  | 86 +--
 xen/drivers/passthrough/vtd/x86/ats.c |  7 +--
 xen/include/xen/pci.h |  1 +
 5 files changed, 82 insertions(+), 23 deletions(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 98936f55c..843dc88 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -419,7 +419,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev 
*pdev)
 xfree(pdev);
 }
 
-static void _pci_hide_device(struct pci_dev *pdev)
+void pci_hide_existing_device(struct pci_dev *pdev)
 {
 if ( pdev->domain )
 return;
@@ -436,7 +436,7 @@ int __init pci_hide_device(int bus, int devfn)
 pdev = alloc_pdev(get_pseg(0), bus, devfn);
 if ( pdev )
 {
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 rc = 0;
 }
 pcidevs_unlock();
@@ -466,7 +466,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
 }
 
 __set_bit(PCI_BDF2(bus, devfn), pseg->ro_map);
-_pci_hide_device(pdev);
+pci_hide_existing_device(pdev);
 
 return 0;
 }
diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index 45357f2..94ca97a 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -21,6 +21,7 @@
 #define _VTD_EXTERN_H_
 
 #include "dmar.h"
+#include "../ats.h"
 #include 
 
 #define VTDPREFIX "[VT-D]"
@@ -60,8 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
  u64 addr, unsigned int size_order, u64 type);
 
 int __must_check qinval_device_iotlb_sync(struct iommu *iommu,
-  u32 max_invs_pend,
-  u16 sid, u16 size, u64 addr);
+  struct pci_ats_dev *ats_dev,
+  u16 did, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c 
b/xen/drivers/passthrough/vtd/qinval.c
index 9116588..bea3e86 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -30,8 +30,7 @@
 
 #define IOMMU_QI_TIMEOUT MILLISECS(1)
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb);
+static int __must_check invalidate_sync(struct iommu *iommu);
 
 static void print_qi_regs(struct iommu *iommu)
 {
@@ -103,7 +102,7 @@ static int __must_check 
queue_invalidate_context_sync(struct iommu *iommu,
 
 unmap_vtd_domain_page(qinval_entries);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_iotlb_sync(struct iommu *iommu,
@@ -140,7 +139,7 @@ static int __must_check queue_invalidate_iotlb_sync(struct 
iommu *iommu,
 qinval_update_qtail(iommu, index);
 spin_unlock_irqrestore(>register_lock, flags);
 
-return invalidate_sync(iommu, 0);
+return invalidate_sync(iommu);
 }
 
 static int __must_check queue_invalidate_wait(struct iommu *iommu,
@@ -200,20 +199,81 @@ static int __must_check queue_invalidate_wait(struct 
iommu *iommu,
 return -EOPNOTSUPP;
 }
 
-static int __must_check invalidate_sync(struct iommu *iommu,
-bool_t flush_dev_iotlb)
+static int __must_check invalidate_sync(struct iommu *iommu)
 {
 struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
 
 if ( qi_ctrl->qinval_maddr )
-return queue_invalidate_wait(iommu, 0, 1, 1, flush_dev_iotlb);
+return queue_invalidate_wait(iommu, 0, 1, 1, 0);
 
 return 0;
 }
 
+static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
+ struct pci_ats_dev *ats_dev)
+{
+struct domain *d = NULL;
+struct pci_dev *pdev;
+
+if ( test_bit(did, iommu->domid_bitmap) )
+d = rcu_lock_domain_by_id(iommu->domid_map[did]);
+
+/*
+ * In case the domain has been freed or the IOMMU domid bitmap is
+ * not valid, the device no longer belongs to this domain.
+ */
+if ( d == NULL )
+return;
+
+pcidevs_lock();
+

[Xen-devel] [Patch v11 0/3] VT-d Device-TLB flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patches fix current timeout concern and also allow limited ATS support:

1. add a timeout parameter for device IOTLB invalidation

 The parameter 'iommu_dev_iotlb_timeout' specifies the timeout of the device
 IOTLB invalidation in milliseconds. By default, the timeout is 1ms, which can
 be boot-time changed.

2. Synchronize for Device-TLB flush one by one

 Today we do Device-TLB flush synchronization after issuing flush
 requests for all ATS devices belonging to a VM. Doing so however
 imposes a limitation, i.e. that we can not figure out which flush
 request is blocked in the flush queue list, based on VT-d spec.

 To prepare correct Device-TLB flush timeout handling in next patch,
 we change the behavior to synchronize for every Device-TLB flush
 request. So the Device-TLB flush interface is changed a little bit,
 by checking timeout within the function instead of outside of function.

 Accordingly we also do a similar change for flush interfaces of
 IOTLB/IEC/Context, i.e. moving synchronization into the function.
 Since there is no user of a non-synced interface, we just rename
 existing ones with _sync suffix.

3. Fix vt-d Device-TLB flush timeout issue

 If Device-TLB flush timed out, we hide the target ATS device
 immediately and crash the domain owning this ATS device. If
 impacted domain is hardware domain, just throw out a warning.

 By hiding the device, we make sure it can't be assigned to any
 domain any longer (see device_assigned).


**NOTE**
   This patch set should base on one prereq patch sets:
a). Check VT-d Device-TLB flush error.


Not covered in this series:

a) Eliminate the panic() in IOMMU_WAIT_OP, used only in VT-d register 
read/write.
   Further discussion is required on whether and how to improve it.
b) Handle IOTLB/Context/IEC flush timeout.


Quan Xu (3):
  IOMMU: add a timeout parameter for device IOTLB invalidation
  vt-d: synchronize for Device-TLB flush one by one
  vt-d: fix vt-d Device-TLB flush timeout issue

 docs/misc/xen-command-line.markdown   |   9 ++
 xen/drivers/passthrough/iommu.c   |   3 +
 xen/drivers/passthrough/pci.c |   6 +-
 xen/drivers/passthrough/vtd/extern.h  |   6 +-
 xen/drivers/passthrough/vtd/qinval.c  | 157 ++
 xen/drivers/passthrough/vtd/x86/ats.c |   7 +-
 xen/include/xen/iommu.h   |   2 +
 xen/include/xen/pci.h |   1 +
 8 files changed, 144 insertions(+), 47 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 03/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
---
 xen/drivers/passthrough/arm/smmu.c|  2 +-
 xen/drivers/passthrough/vtd/iommu.c   | 15 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  2 +-
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 54a03a6..1ce4ddf 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2774,7 +2774,7 @@ static int arm_smmu_map_page(struct domain *d, unsigned 
long gfn,
return guest_physmap_add_entry(d, gfn, mfn, 0, t);
 }
 
-static int arm_smmu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check arm_smmu_unmap_page(struct domain *d, unsigned long 
gfn)
 {
/*
 * This function should only be used by gnttab code when the domain
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index db83949..4844193 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -609,7 +609,7 @@ static void intel_iommu_iotlb_flush_all(struct domain *d)
 }
 
 /* clear one page's page table */
-static void dma_pte_clear_one(struct domain *domain, u64 addr)
+static int __must_check dma_pte_clear_one(struct domain *domain, u64 addr)
 {
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
@@ -621,7 +621,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 if ( pg_maddr == 0 )
 {
 spin_unlock(>arch.mapping_lock);
-return;
+return 0;
 }
 
 page = (struct dma_pte *)map_vtd_domain_page(pg_maddr);
@@ -631,7 +631,7 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 {
 spin_unlock(>arch.mapping_lock);
 unmap_vtd_domain_page(page);
-return;
+return 0;
 }
 
 dma_clear_pte(*pte);
@@ -642,6 +642,8 @@ static void dma_pte_clear_one(struct domain *domain, u64 
addr)
 __intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
 
 unmap_vtd_domain_page(page);
+
+return 0;
 }
 
 static void iommu_free_pagetable(u64 pt_maddr, int level)
@@ -1739,15 +1741,14 @@ static int intel_iommu_map_page(
 return 0;
 }
 
-static int intel_iommu_unmap_page(struct domain *d, unsigned long gfn)
+static int __must_check intel_iommu_unmap_page(struct domain *d,
+   unsigned long gfn)
 {
 /* Do nothing if hardware domain and iommu supports pass thru. */
 if ( iommu_passthrough && is_hardware_domain(d) )
 return 0;
 
-dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
-
-return 0;
+return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
 void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 9c51172..57b6cc1 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -53,7 +53,7 @@ int amd_iommu_update_ivrs_mapping_acpi(void);
 /* mapping functions */
 int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
unsigned int flags);
-int amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
+int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
u64 phys_addr, unsigned long size,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 19ba976..73a7f1e 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -168,7 +168,7 @@ struct iommu_ops {
 void (*teardown)(struct domain *d);
 int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
 unsigned int flags);
-int (*unmap_page)(struct domain *d, unsigned long gfn);
+int __must_check (*unmap_page)(struct domain *d, unsigned long gfn);
 void (*free_page_table)(struct page_info *);
 #ifdef CONFIG_X86
 void (*update_ire_from_apic)(unsigned int apic, unsigned int reg, unsigned 
int value);
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 01/11] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Treat IOMMU mapping and unmapping failures as a fatal to the DomU
If IOMMU mapping and unmapping failed, crash the DomU and propagate
the error up to the call trees.

No spamming of the log can occur. For DomU, we avoid logging any
message for already dying domains. For Dom0, that'll still be more
verbose than we'd really like, but it at least wouldn't outright
flood the console.

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Kevin Tian <kevin.t...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>

v6:
  1. Enhance commit message, changing "No spamming can occur"
 to "No spamming of the log can occur".
  2. Enhance log messages.
---
 xen/drivers/passthrough/iommu.c | 30 --
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 9d104d2..673e126 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -240,21 +240,47 @@ int iommu_map_page(struct domain *d, unsigned long gfn, 
unsigned long mfn,
unsigned int flags)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->map_page(d, gfn, mfn, flags);
+rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU mapping gfn %#lx to mfn %#lx failed: %d\n",
+   d->domain_id, gfn, mfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 int iommu_unmap_page(struct domain *d, unsigned long gfn)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+int rc;
 
 if ( !iommu_enabled || !hd->platform_ops )
 return 0;
 
-return hd->platform_ops->unmap_page(d, gfn);
+rc = hd->platform_ops->unmap_page(d, gfn);
+if ( unlikely(rc) )
+{
+if ( !d->is_shutting_down && printk_ratelimit() )
+printk(XENLOG_ERR
+   "d%d: IOMMU unmapping gfn %#lx failed: %d\n",
+   d->domain_id, gfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
 }
 
 static void iommu_free_pagetables(unsigned long unused)
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 04/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>

CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Jan Beulich <jbeul...@suse.com>

v6:
  1. Add __must_check annotation to amd_iommu_map_page().
  2. Return the first error instead of the last one.
---
 xen/drivers/passthrough/amd/pci_amd_iommu.c   | 16 ++--
 xen/drivers/passthrough/arm/smmu.c|  4 ++--
 xen/drivers/passthrough/vtd/iommu.c   |  7 ---
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  4 ++--
 xen/include/xen/iommu.h   |  4 ++--
 5 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 70b7475..17c1f6d 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -285,6 +285,8 @@ static void __hwdom_init amd_iommu_hwdom_init(struct domain 
*d)
 
 if ( !iommu_passthrough && !need_iommu(d) )
 {
+int rc = 0;
+
 /* Set up 1:1 page table for dom0 */
 for ( i = 0; i < max_pdx; i++ )
 {
@@ -295,12 +297,22 @@ static void __hwdom_init amd_iommu_hwdom_init(struct 
domain *d)
  * a pfn_valid() check would seem desirable here.
  */
 if ( mfn_valid(pfn) )
-amd_iommu_map_page(d, pfn, pfn, 
-   IOMMUF_readable|IOMMUF_writable);
+{
+int ret;
+
+ret = amd_iommu_map_page(d, pfn, pfn,
+ IOMMUF_readable|IOMMUF_writable);
+if ( !rc )
+rc = ret;
+}
 
 if ( !(i & 0xf) )
 process_pending_softirqs();
 }
+
+if ( rc )
+AMD_IOMMU_DEBUG("d%d: IOMMU mapping failed: %d\n",
+d->domain_id, rc);
 }
 
 for_each_amd_iommu ( iommu )
diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index 1ce4ddf..ee5c89d 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2745,8 +2745,8 @@ static void arm_smmu_iommu_domain_teardown(struct domain 
*d)
xfree(xen_domain);
 }
 
-static int arm_smmu_map_page(struct domain *d, unsigned long gfn,
-unsigned long mfn, unsigned int flags)
+static int __must_check arm_smmu_map_page(struct domain *d, unsigned long gfn,
+  unsigned long mfn, unsigned int 
flags)
 {
p2m_type_t t;
 
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 4844193..e900019 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1687,9 +1687,10 @@ static void iommu_domain_teardown(struct domain *d)
 spin_unlock(>arch.mapping_lock);
 }
 
-static int intel_iommu_map_page(
-struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags)
+static int __must_check intel_iommu_map_page(struct domain *d,
+ unsigned long gfn,
+ unsigned long mfn,
+ unsigned int flags)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct dma_pte *page = NULL, *pte = NULL, old, new = { 0 };
diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h 
b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
index 57b6cc1..ac9f036 100644
--- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
+++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
@@ -51,8 +51,8 @@ int amd_iommu_init(void);
 int amd_iommu_update_ivrs_mapping_acpi(void);
 
 /* mapping functions */
-int amd_iommu_map_page(struct domain *d, unsigned long gfn, unsigned long mfn,
-   unsigned int flags);
+int __must_check amd_iommu_map_page(struct domain *d, unsigned long gfn,
+unsigned long mfn, unsigned int flags);
 int __must_check amd_iommu_unmap_page(struct domain *d, unsigned long gfn);
 u64 amd_iommu_get_next_table_from_pte(u32 *entry);
 int amd_iommu_reserve_domain_unity_map(struct domain *domain,
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 73a7f1e..14041a1 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -166,8 +166,8 @@ struct iommu_ops {
 #endif /* HAS_PCI */
 
 void (*teardown)(struct domain *d);
-int (*map_page)(struct domain *d, unsigned long gfn, unsigned long mfn,
-unsigned int flags);
+int __must_check (*map_page)(struct domain *d, unsigned long gf

[Xen-devel] [Patch v6 02/11] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

When IOMMU mapping is failed, we issue a best effort rollback, stopping
IOMMU mapping, unmapping the previous IOMMU maps and then reporting the
error up to the call trees. When rollback is not feasible (in early
initialization phase or trade-off of complexity) for the hardware domain,
we do things on a best effort basis, only throwing out an error message.

IOMMU unmapping should perhaps continue despite an error, in an attempt
to do best effort cleanup.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>

v6:
  1. Limit __must_check annotation to IOMMU functions for this patch set.
  2. Replace 'ret' with a more specific name 'iommu_ret' in __get_page_type().
  3. Return the first error instead of the last one.
---
 xen/arch/x86/mm.c   | 13 -
 xen/arch/x86/mm/p2m-ept.c   | 33 ++---
 xen/arch/x86/mm/p2m-pt.c| 24 +---
 xen/arch/x86/mm/p2m.c   | 16 ++--
 xen/drivers/passthrough/iommu.c | 14 +-
 5 files changed, 82 insertions(+), 18 deletions(-)

diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 8d10a3e..ae7c8ab 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
int preemptible)
 {
 unsigned long nx, x, y = page->u.inuse.type_info;
-int rc = 0;
+int rc = 0, iommu_ret = 0;
 
 ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
 
@@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
 {
 if ( (x & PGT_type_mask) == PGT_writable_page )
-iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
+iommu_ret = iommu_unmap_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)));
 else if ( type == PGT_writable_page )
-iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
-   page_to_mfn(page),
-   IOMMUF_readable|IOMMUF_writable);
+iommu_ret = iommu_map_page(d, mfn_to_gmfn(d, 
page_to_mfn(page)),
+   page_to_mfn(page),
+   IOMMUF_readable|IOMMUF_writable);
 }
 }
 
@@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info *page, 
unsigned long type,
 if ( (x & PGT_partial) && !(nx & PGT_partial) )
 put_page(page);
 
+if ( !rc )
+rc = iommu_ret;
+
 return rc;
 }
 
diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 1ed5b47..ebb4343 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -667,6 +667,7 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 unsigned long gfn_remainder = gfn;
 unsigned int i, target = order / EPT_TABLE_ORDER;
 int ret, rc = 0;
+bool_t entry_written = 0;
 bool_t direct_mmio = (p2mt == p2m_mmio_direct);
 uint8_t ipat = 0;
 bool_t need_modify_vtd_table = 1;
@@ -812,10 +813,15 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, 
mfn_t mfn,
 rc = atomic_write_ept_entry(ept_entry, new_entry, target);
 if ( unlikely(rc) )
 old_entry.epte = 0;
-else if ( p2mt != p2m_invalid &&
-  (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
-/* Track the highest gfn for which we have ever had a valid mapping */
-p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+else
+{
+entry_written = 1;
+
+if ( p2mt != p2m_invalid &&
+ (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
+/* Track the highest gfn for which we have ever had a valid 
mapping */
+p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
+}
 
 out:
 if ( needs_sync )
@@ -831,10 +837,23 @@ out:
 {
 if ( iommu_flags )
 for ( i = 0; i < (1 << order); i++ )
-iommu_map_page(d, gfn + i, mfn_x(mfn) + i, iommu_flags);
+{
+rc = iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
iommu_flags);
+if ( unlikely(rc) )
+{
+while ( i-- )
+iommu_unmap_page(p2m->domain, gfn + i);
+
+break;
+}
+}
 else
 for ( i = 0; i < (1 << order); i++ )
-   

[Xen-devel] [Patch v6 08/11] IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v6: Add __must_check annotation to intel_iommu_iotlb_flush_all().
---
 xen/drivers/passthrough/arm/smmu.c  | 13 -
 xen/drivers/passthrough/iommu.c |  8 ++--
 xen/drivers/passthrough/vtd/iommu.c | 32 
 xen/include/xen/iommu.h |  5 +++--
 4 files changed, 33 insertions(+), 25 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c 
b/xen/drivers/passthrough/arm/smmu.c
index ee5c89d..1d21568 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -2540,7 +2540,7 @@ static int force_stage = 2;
  */
 static u32 platform_features = ARM_SMMU_FEAT_COHERENT_WALK;
 
-static void arm_smmu_iotlb_flush_all(struct domain *d)
+static int __must_check arm_smmu_iotlb_flush_all(struct domain *d)
 {
struct arm_smmu_xen_domain *smmu_domain = dom_iommu(d)->arch.priv;
struct iommu_domain *cfg;
@@ -2557,13 +2557,16 @@ static void arm_smmu_iotlb_flush_all(struct domain *d)
arm_smmu_tlb_inv_context(cfg->priv);
}
spin_unlock(_domain->lock);
+
+   return 0;
 }
 
-static void arm_smmu_iotlb_flush(struct domain *d, unsigned long gfn,
- unsigned int page_count)
+static int __must_check arm_smmu_iotlb_flush(struct domain *d,
+ unsigned long gfn,
+ unsigned int page_count)
 {
-/* ARM SMMU v1 doesn't have flush by VMA and VMID */
-arm_smmu_iotlb_flush_all(d);
+   /* ARM SMMU v1 doesn't have flush by VMA and VMID */
+   return arm_smmu_iotlb_flush_all(d);
 }
 
 static struct iommu_domain *arm_smmu_get_domain(struct domain *d,
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index e611e72..098b601 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -320,9 +320,7 @@ int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
 return 0;
 
-hd->platform_ops->iotlb_flush(d, gfn, page_count);
-
-return 0;
+return hd->platform_ops->iotlb_flush(d, gfn, page_count);
 }
 
 int iommu_iotlb_flush_all(struct domain *d)
@@ -332,9 +330,7 @@ int iommu_iotlb_flush_all(struct domain *d)
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
 return 0;
 
-hd->platform_ops->iotlb_flush_all(d);
-
-return 0;
+return hd->platform_ops->iotlb_flush_all(d);
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index c2bf1e2..0788a59 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -559,8 +559,10 @@ static int __must_check iommu_flush_all(void)
 return 0;
 }
 
-static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
-int dma_old_pte_present, unsigned int page_count)
+static int __must_check iommu_flush_iotlb(struct domain *d,
+  unsigned long gfn,
+  bool_t dma_old_pte_present,
+  unsigned int page_count)
 {
 struct domain_iommu *hd = dom_iommu(d);
 struct acpi_drhd_unit *drhd;
@@ -598,16 +600,20 @@ static void __intel_iommu_iotlb_flush(struct domain *d, 
unsigned long gfn,
 iommu_flush_write_buffer(iommu);
 }
 }
+
+return 0;
 }
 
-static void intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn, 
unsigned int page_count)
+static int __must_check iommu_flush_iotlb_pages(struct domain *d,
+unsigned long gfn,
+unsigned int page_count)
 {
-__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
+return iommu_flush_iotlb(d, gfn, 1, page_count);
 }
 
-static void intel_iommu_iotlb_flush_all(struct domain *d)
+static int __must_check iommu_flush_iotlb_all(struct domain *d)
 {
-__intel_iommu_iotlb_flush(d, INVALID_GFN, 0, 0);
+return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
 }
 
 /* clear one page's page table */
@@ -616,6 +622,7 @@ static int __must_check dma_pte_clear_one(struct domain 
*domain, u64 addr)
 struct domain_iommu *hd = dom_iommu(domain);
 struct dma_pte *page = NULL, *pte = NULL;
 u64 pg_maddr;
+int rc = 0;
 
 spin_lock(>arch.mapping_lock);
 /* get last l

[Xen-devel] [Patch v6 11/11] vt-d: add __must_check annotation to IOMMU flush pointers and handlers

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/drivers/passthrough/vtd/iommu.c  | 40 ++--
 xen/drivers/passthrough/vtd/iommu.h  | 11 ++
 xen/drivers/passthrough/vtd/qinval.c | 14 ++---
 3 files changed, 34 insertions(+), 31 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 2cc2c93..32202b6 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -335,10 +335,9 @@ static void iommu_flush_write_buffer(struct iommu *iommu)
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_context_reg(
-void *_iommu,
-u16 did, u16 source_id, u8 function_mask, u64 type,
-int flush_non_present_entry)
+static int __must_check flush_context_reg(void *_iommu, u16 did, u16 source_id,
+  u8 function_mask, u64 type,
+  bool_t flush_non_present_entry)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 u64 val = 0;
@@ -389,7 +388,7 @@ static int flush_context_reg(
 }
 
 static int __must_check iommu_flush_context_global(struct iommu *iommu,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
@@ -399,7 +398,7 @@ static int __must_check iommu_flush_context_global(struct 
iommu *iommu,
 static int __must_check iommu_flush_context_device(struct iommu *iommu,
u16 did, u16 source_id,
u8 function_mask,
-   int flush_non_present_entry)
+   bool_t 
flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -408,9 +407,10 @@ static int __must_check iommu_flush_context_device(struct 
iommu *iommu,
 }
 
 /* return value determine if we need a write buffer flush */
-static int flush_iotlb_reg(void *_iommu, u16 did,
-   u64 addr, unsigned int size_order, u64 type,
-   int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check flush_iotlb_reg(void *_iommu, u16 did, u64 addr,
+unsigned int size_order, u64 type,
+bool_t flush_non_present_entry,
+bool_t flush_dev_iotlb)
 {
 struct iommu *iommu = (struct iommu *) _iommu;
 int tlb_offset = ecap_iotlb_offset(iommu->ecap);
@@ -475,8 +475,8 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 }
 
 static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
- int flush_non_present_entry,
- int flush_dev_iotlb)
+ bool_t 
flush_non_present_entry,
+ bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -494,8 +494,8 @@ static int __must_check iommu_flush_iotlb_global(struct 
iommu *iommu,
 }
 
 static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -514,8 +514,8 @@ static int __must_check iommu_flush_iotlb_dsi(struct iommu 
*iommu, u16 did,
 
 static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
   u64 addr, unsigned int order,
-  int flush_non_present_entry,
-  int flush_dev_iotlb)
+  bool_t flush_non_present_entry,
+  bool_t flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -549,7 +549,7 @@ static int __must_check iommu_flush_all(void)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
-int flush_dev_iotlb;
+bool_t flush_dev_iotlb;
 int rc = 0;
 
 flush_all_cache();
@@ -597,7 +597,7 @@ static in

[Xen-devel] [Patch v6 00/11] Check VT-d Device-TLB flush error

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

This patch set is a prereq patch set for Patch:'VT-d Device-TLB flush issue'.

While IOMMU Device-TLB flush timed out, xen calls panic() at present. However 
the existing panic()
is going to be eliminated, so we must propagate the IOMMU Device-TLB flush 
error up to the call trees.

This patch set is also based on the discussion of 'abstract model of IOMMU 
unmaping/mapping failures'

Quan Xu (11):
  IOMMU: handle IOMMU mapping and unmapping failures
  IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU unmapping
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping (top
level ones)
  IOMMU/MMU: propagate IOMMU Device-TLB flush error up to
iommu_iotlb_flush{,_all} (top level ones)
  propagate IOMMU Device-TLB flush error up to EPT update (top level
ones)
  IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending
(top level ones)
  IOMMU: propagate IOMMU Device-TLB flush error (leaf ones).
  vt-d: fix the IOMMU flush issue
  vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom
functions
  vt-d: add __must_check annotation to IOMMU flush pointers and handlers

 xen/arch/arm/p2m.c|   4 +-
 xen/arch/x86/acpi/power.c |  76 +--
 xen/arch/x86/mm.c |  13 +-
 xen/arch/x86/mm/p2m-ept.c |  35 +++-
 xen/arch/x86/mm/p2m-pt.c  |  24 ++-
 xen/arch/x86/mm/p2m.c |  16 +-
 xen/common/memory.c   |  12 +-
 xen/drivers/passthrough/amd/iommu_init.c  |   9 +-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  18 +-
 xen/drivers/passthrough/arm/smmu.c|  19 +-
 xen/drivers/passthrough/iommu.c   |  63 +-
 xen/drivers/passthrough/vtd/extern.h  |   3 +-
 xen/drivers/passthrough/vtd/iommu.c   | 285 ++
 xen/drivers/passthrough/vtd/iommu.h   |  11 +-
 xen/drivers/passthrough/vtd/qinval.c  |  14 +-
 xen/drivers/passthrough/vtd/quirks.c  |  27 ++-
 xen/drivers/passthrough/x86/iommu.c   |   5 +-
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |   8 +-
 xen/include/asm-x86/iommu.h   |   3 +-
 xen/include/xen/iommu.h   |  20 +-
 20 files changed, 479 insertions(+), 186 deletions(-)

-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 07/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Liu Jinsong <jinsong@alibaba-inc.com>
CC: Keir Fraser <k...@xen.org>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Suravee Suthikulpanit <suravee.suthikulpa...@amd.com>
CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v6:
  1. Drop comments in enum dev_power_saved.
  2. If console_suspend() fails, return SAVED_NONE.
  3. Put off leaf ones (in iommu_flush_all()) in later patch.
---
 xen/arch/x86/acpi/power.c | 76 ---
 xen/drivers/passthrough/amd/iommu_init.c  |  9 +++-
 xen/drivers/passthrough/amd/pci_amd_iommu.c   |  2 +-
 xen/drivers/passthrough/iommu.c   |  6 ++-
 xen/drivers/passthrough/vtd/iommu.c   | 35 
 xen/include/asm-x86/hvm/svm/amd-iommu-proto.h |  2 +-
 xen/include/xen/iommu.h   |  4 +-
 7 files changed, 99 insertions(+), 35 deletions(-)

diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
index 2885e31..e516fbd 100644
--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -43,36 +43,70 @@ struct acpi_sleep_info acpi_sinfo;
 
 void do_suspend_lowlevel(void);
 
+enum dev_power_saved
+{
+SAVED_NONE,
+SAVED_CONSOLE,
+SAVED_TIME,
+SAVED_I8259A,
+SAVED_IOAPIC,
+SAVED_IOMMU,
+SAVED_LAPIC,
+SAVED_ALL,
+};
+
 static int device_power_down(void)
 {
-console_suspend();
+if ( console_suspend() )
+return SAVED_NONE;
 
-time_suspend();
+if ( time_suspend() )
+return SAVED_CONSOLE;
 
-i8259A_suspend();
+if ( i8259A_suspend() )
+return SAVED_TIME;
 
+/* ioapic_suspend cannot fail */
 ioapic_suspend();
 
-iommu_suspend();
+if ( iommu_suspend() )
+return SAVED_IOAPIC;
 
-lapic_suspend();
+if ( lapic_suspend() )
+return SAVED_IOMMU;
 
-return 0;
+return SAVED_NONE;
 }
 
-static void device_power_up(void)
+static void device_power_up(enum dev_power_saved saved)
 {
-lapic_resume();
-
-iommu_resume();
-
-ioapic_resume();
-
-i8259A_resume();
-
-time_resume();
-
-console_resume();
+switch ( saved )
+{
+case SAVED_ALL:
+case SAVED_LAPIC:
+lapic_resume();
+/* fall through */
+case SAVED_IOMMU:
+iommu_resume();
+/* fall through */
+case SAVED_IOAPIC:
+ioapic_resume();
+/* fall through */
+case SAVED_I8259A:
+i8259A_resume();
+/* fall through */
+case SAVED_TIME:
+time_resume();
+/* fall through */
+case SAVED_CONSOLE:
+console_resume();
+/* fall through */
+case SAVED_NONE:
+break;
+default:
+BUG();
+break;
+}
 }
 
 static void freeze_domains(void)
@@ -169,6 +203,10 @@ static int enter_state(u32 state)
 {
 printk(XENLOG_ERR "Some devices failed to power down.");
 system_state = SYS_STATE_resume;
+
+if ( error > 0 )
+device_power_up(error);
+
 goto done;
 }
 
@@ -196,7 +234,7 @@ static int enter_state(u32 state)
 write_cr4(cr4 & ~X86_CR4_MCE);
 write_efer(read_efer());
 
-device_power_up();
+device_power_up(SAVED_ALL);
 
 mcheck_init(_cpu_data, 0);
 write_cr4(cr4);
diff --git a/xen/drivers/passthrough/amd/iommu_init.c 
b/xen/drivers/passthrough/amd/iommu_init.c
index 4536106..0b68596 100644
--- a/xen/drivers/passthrough/amd/iommu_init.c
+++ b/xen/drivers/passthrough/amd/iommu_init.c
@@ -1339,7 +1339,14 @@ static void invalidate_all_devices(void)
 iterate_ivrs_mappings(_invalidate_all_devices);
 }
 
-void amd_iommu_suspend(void)
+int amd_iommu_suspend(void)
+{
+amd_iommu_crash_shutdown();
+
+return 0;
+}
+
+void amd_iommu_crash_shutdown(void)
 {
 struct amd_iommu *iommu;
 
diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c 
b/xen/drivers/passthrough/amd/pci_amd_iommu.c
index 17c1f6d..9488eb0 100644
--- a/xen/drivers/passthrough/amd/pci_amd_iommu.c
+++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c
@@ -638,6 +638,6 @@ const struct iommu_ops amd_iommu_ops = {
 .suspend = amd_iommu_suspend,
 .resume = amd_iommu_resume,
 .share_p2m = amd_iommu_share_p2m,
-.crash_shutdown = amd_iommu_suspend,
+.crash_shutdown = amd_iommu_crash_shutdown,
 .dump_p2m_table = amd_dump_p2m_table,
 };
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index e9b2855..e611e72 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -380,10 +380,12 @@ int __init iommu_setup(void)
 return rc;
 }
 
-void iommu_suspend()
+int iommu_suspend()
 {
 if ( iommu_enabled )
-iommu_get_ops()->suspend();
+re

[Xen-devel] [Patch v6 05/11] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Stefano Stabellini <sstabell...@kernel.org>
CC: Julien Grall <julien.gr...@arm.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>

v6: Remove __must_check annotation from xenmem_add_to_physmap().
---
 xen/arch/arm/p2m.c  |  4 +++-
 xen/common/memory.c | 12 ++--
 xen/drivers/passthrough/iommu.c | 13 +
 xen/drivers/passthrough/x86/iommu.c |  5 +++--
 xen/include/xen/iommu.h |  5 +++--
 5 files changed, 28 insertions(+), 11 deletions(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 838d004..68679a3 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -1178,7 +1178,9 @@ out:
 if ( flush )
 {
 flush_tlb_domain(d);
-iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+ret = iommu_iotlb_flush(d, sgfn, egfn - sgfn);
+if ( !rc )
+rc = ret;
 }
 
 while ( (pg = page_list_remove_head(_pages)) )
diff --git a/xen/common/memory.c b/xen/common/memory.c
index ccc6436..46b1d9f 100644
--- a/xen/common/memory.c
+++ b/xen/common/memory.c
@@ -683,9 +683,17 @@ static int xenmem_add_to_physmap(struct domain *d,
 #ifdef CONFIG_HAS_PASSTHROUGH
 if ( need_iommu(d) )
 {
+int ret;
+
 this_cpu(iommu_dont_flush_iotlb) = 0;
-iommu_iotlb_flush(d, xatp->idx - done, done);
-iommu_iotlb_flush(d, xatp->gpfn - done, done);
+
+ret = iommu_iotlb_flush(d, xatp->idx - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
+
+ret = iommu_iotlb_flush(d, xatp->gpfn - done, done);
+if ( unlikely(ret) && rc >= 0 )
+rc = ret;
 }
 #endif
 
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 25501cd..e9b2855 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -312,24 +312,29 @@ static void iommu_free_pagetables(unsigned long unused)
 cpumask_cycle(smp_processor_id(), 
_online_map));
 }
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count)
+int iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+  unsigned int page_count)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->iotlb_flush 
)
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush(d, gfn, page_count);
+
+return 0;
 }
 
-void iommu_iotlb_flush_all(struct domain *d)
+int iommu_iotlb_flush_all(struct domain *d)
 {
 const struct domain_iommu *hd = dom_iommu(d);
 
 if ( !iommu_enabled || !hd->platform_ops || 
!hd->platform_ops->iotlb_flush_all )
-return;
+return 0;
 
 hd->platform_ops->iotlb_flush_all(d);
+
+return 0;
 }
 
 int __init iommu_setup(void)
diff --git a/xen/drivers/passthrough/x86/iommu.c 
b/xen/drivers/passthrough/x86/iommu.c
index b64b98f..a18a608 100644
--- a/xen/drivers/passthrough/x86/iommu.c
+++ b/xen/drivers/passthrough/x86/iommu.c
@@ -104,8 +104,9 @@ int arch_iommu_populate_page_table(struct domain *d)
 this_cpu(iommu_dont_flush_iotlb) = 0;
 
 if ( !rc )
-iommu_iotlb_flush_all(d);
-else if ( rc != -ERESTART )
+rc = iommu_iotlb_flush_all(d);
+
+if ( rc && rc != -ERESTART )
 iommu_teardown(d);
 
 return rc;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 14041a1..22a2002 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -200,8 +200,9 @@ int iommu_do_pci_domctl(struct xen_domctl *, struct domain 
*d,
 int iommu_do_domctl(struct xen_domctl *, struct domain *d,
 XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
 
-void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned int 
page_count);
-void iommu_iotlb_flush_all(struct domain *d);
+int __must_check iommu_iotlb_flush(struct domain *d, unsigned long gfn,
+   unsigned int page_count);
+int __must_check iommu_iotlb_flush_all(struct domain *d);
 
 /*
  * The purpose of the iommu_dont_flush_iotlb optional cpu flag is to
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 06/11] propagate IOMMU Device-TLB flush error up to EPT update (top level ones)

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Propagate the IOMMU Device-TLB flush error up to the ept_set_entry(),
when VT-d shares EPT page table.

Signed-off-by: Quan Xu <quan...@intel.com>
Acked-by: Kevin Tian <kevin.t...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jun Nakajima <jun.nakaj...@intel.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: George Dunlap <george.dun...@eu.citrix.com>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>
CC: Feng Wu <feng...@intel.com>
---
 xen/arch/x86/mm/p2m-ept.c   | 2 +-
 xen/drivers/passthrough/vtd/iommu.c | 6 --
 xen/include/asm-x86/iommu.h | 3 ++-
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index ebb4343..dab3215 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -832,7 +832,7 @@ out:
  need_modify_vtd_table )
 {
 if ( iommu_hap_pt_share )
-iommu_pte_flush(d, gfn, _entry->epte, order, vtd_pte_present);
+rc = iommu_pte_flush(d, gfn, _entry->epte, order, 
vtd_pte_present);
 else
 {
 if ( iommu_flags )
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index e900019..5366267 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1752,8 +1752,8 @@ static int __must_check intel_iommu_unmap_page(struct 
domain *d,
 return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
 }
 
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
- int order, int present)
+int iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+int order, int present)
 {
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu = NULL;
@@ -1778,6 +1778,8 @@ void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
order, !present, flush_dev_iotlb) )
 iommu_flush_write_buffer(iommu);
 }
+
+return 0;
 }
 
 static int __init vtd_ept_page_compatible(struct iommu *iommu)
diff --git a/xen/include/asm-x86/iommu.h b/xen/include/asm-x86/iommu.h
index e82a2f0..815d77e 100644
--- a/xen/include/asm-x86/iommu.h
+++ b/xen/include/asm-x86/iommu.h
@@ -27,7 +27,8 @@ int iommu_setup_hpet_msi(struct msi_desc *);
 
 /* While VT-d specific, this must get declared in a generic header. */
 int adjust_vtd_irq_affinities(void);
-void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int 
present);
+int __must_check iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte,
+ int order, int present);
 bool_t iommu_supports_eim(void);
 int iommu_enable_x2apic_IR(void);
 void iommu_disable_x2apic_IR(void);
-- 
1.9.1


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [Patch v6 10/11] vt-d: propagate the IOMMU Device-TLB flush error up to ME phantom functions

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

Signed-off-by: Quan Xu <quan...@intel.com>
Reviewed-by: Jan Beulich <jbeul...@suse.com>

CC: Jan Beulich <jbeul...@suse.com>
CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>

v6: Don't needlessly split the function header onto two lines.
---
 xen/drivers/passthrough/vtd/extern.h |  3 ++-
 xen/drivers/passthrough/vtd/iommu.c  |  8 
 xen/drivers/passthrough/vtd/quirks.c | 27 +--
 3 files changed, 23 insertions(+), 15 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index cbe0286..6772839 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -91,7 +91,8 @@ int is_igd_vt_enabled_quirk(void);
 void platform_quirks_init(void);
 void vtd_ops_preamble_quirk(struct iommu* iommu);
 void vtd_ops_postamble_quirk(struct iommu* iommu);
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
+int __must_check me_wifi_quirk(struct domain *domain,
+   u8 bus, u8 devfn, int map);
 void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 720b867..2cc2c93 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1473,8 +1473,8 @@ int domain_context_mapping_one(
 
 unmap_vtd_domain_page(context_entries);
 
-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
@@ -1635,8 +1635,8 @@ int domain_context_unmap_one(
 spin_unlock(>lock);
 unmap_vtd_domain_page(context_entries);
 
-if ( !iommu->intel->drhd->segment )
-me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
+if ( !iommu->intel->drhd->segment && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, UNMAP_ME_PHANTOM_FUNC);
 
 return rc;
 }
diff --git a/xen/drivers/passthrough/vtd/quirks.c 
b/xen/drivers/passthrough/vtd/quirks.c
index 473d1fc..91f96ac 100644
--- a/xen/drivers/passthrough/vtd/quirks.c
+++ b/xen/drivers/passthrough/vtd/quirks.c
@@ -331,10 +331,12 @@ void __init platform_quirks_init(void)
  * assigning Intel integrated wifi device to a guest.
  */
 
-static void map_me_phantom_function(struct domain *domain, u32 dev, int map)
+static int __must_check map_me_phantom_function(struct domain *domain,
+u32 dev, int map)
 {
 struct acpi_drhd_unit *drhd;
 struct pci_dev *pdev;
+int rc;
 
 /* find ME VT-d engine base on a real ME device */
 pdev = pci_get_pdev(0, 0, PCI_DEVFN(dev, 0));
@@ -342,23 +344,26 @@ static void map_me_phantom_function(struct domain 
*domain, u32 dev, int map)
 
 /* map or unmap ME phantom function */
 if ( map )
-domain_context_mapping_one(domain, drhd->iommu, 0,
-   PCI_DEVFN(dev, 7), NULL);
+rc = domain_context_mapping_one(domain, drhd->iommu, 0,
+PCI_DEVFN(dev, 7), NULL);
 else
-domain_context_unmap_one(domain, drhd->iommu, 0,
- PCI_DEVFN(dev, 7));
+rc = domain_context_unmap_one(domain, drhd->iommu, 0,
+  PCI_DEVFN(dev, 7));
+
+return rc;
 }
 
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
+int me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
 {
 u32 id;
+int rc = 0;
 
 id = pci_conf_read32(0, 0, 0, 0, 0);
 if ( IS_CTG(id) )
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 3, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:3.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -372,7 +377,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 case 0x423b8086:
 case 0x423c8086:
 case 0x423d8086:
-map_me_phantom_function(domain, 3, map);
+rc = map_me_phantom_function(domain, 3, map);
 break;
 default:
 break;
@@ -382,7 +387,7 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, 
int map)
 {
 /* quit if ME does not exist */
 if ( pci_conf_read32(0, 0, 22, 0, 0) == 0x )
-return;
+return 0;
 
 /* if device is WLAN device, map ME phantom device 0:22.7 */
 id = pci_conf_read32(0, bus, PCI_SLOT(devfn), PCI_FUNC(devfn), 0);
@@ -398,12 +403,14 @@ void me_wifi_quirk(struct domain *domain, u8 bus, u8 
devfn, int m

[Xen-devel] [Patch v6 09/11] vt-d: fix the IOMMU flush issue

[Xu, Quan]

From: Quan Xu <quan...@intel.com>

The propagation value from IOMMU flush interfaces may be positive, which
indicates callers need to flush cache, not one of faliures.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

Signed-off-by: Quan Xu <quan...@intel.com>

CC: Kevin Tian <kevin.t...@intel.com>
CC: Feng Wu <feng...@intel.com>
CC: Keir Fraser <k...@xen.org>
CC: Jan Beulich <jbeul...@suse.com>
CC: Andrew Cooper <andrew.coop...@citrix.com>

v6:
  1. Drop blank lines.
  2. Change '(((u16)bus) << 8) | devfn' to 'PCI_BDF2(bus, devfn)'.
  3. Introduce rc for iommu_flush_context_device() / ret for
 iommu_flush_iotlb_dsi(), adding  ASSERT() behind.
  4. Enhance comments.
  5. Fix iommu_flush_context_*().
---
 xen/drivers/passthrough/vtd/iommu.c | 170 +++-
 1 file changed, 128 insertions(+), 42 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 0788a59..720b867 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -388,17 +388,18 @@ static int flush_context_reg(
 return 0;
 }
 
-static int iommu_flush_context_global(
-struct iommu *iommu, int flush_non_present_entry)
+static int __must_check iommu_flush_context_global(struct iommu *iommu,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL,
  flush_non_present_entry);
 }
 
-static int iommu_flush_context_device(
-struct iommu *iommu, u16 did, u16 source_id,
-u8 function_mask, int flush_non_present_entry)
+static int __must_check iommu_flush_context_device(struct iommu *iommu,
+   u16 did, u16 source_id,
+   u8 function_mask,
+   int flush_non_present_entry)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 return flush->context(iommu, did, source_id, function_mask,
@@ -473,8 +474,9 @@ static int flush_iotlb_reg(void *_iommu, u16 did,
 return 0;
 }
 
-static int iommu_flush_iotlb_global(struct iommu *iommu,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_global(struct iommu *iommu,
+ int flush_non_present_entry,
+ int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -491,8 +493,9 @@ static int iommu_flush_iotlb_global(struct iommu *iommu,
 return status;
 }
 
-static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_dsi(struct iommu *iommu, u16 did,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -509,9 +512,10 @@ static int iommu_flush_iotlb_dsi(struct iommu *iommu, u16 
did,
 return status;
 }
 
-static int iommu_flush_iotlb_psi(
-struct iommu *iommu, u16 did, u64 addr, unsigned int order,
-int flush_non_present_entry, int flush_dev_iotlb)
+static int __must_check iommu_flush_iotlb_psi(struct iommu *iommu, u16 did,
+  u64 addr, unsigned int order,
+  int flush_non_present_entry,
+  int flush_dev_iotlb)
 {
 struct iommu_flush *flush = iommu_get_flush(iommu);
 int status;
@@ -546,17 +550,43 @@ static int __must_check iommu_flush_all(void)
 struct acpi_drhd_unit *drhd;
 struct iommu *iommu;
 int flush_dev_iotlb;
+int rc = 0;
 
 flush_all_cache();
 for_each_drhd_unit ( drhd )
 {
 iommu = drhd->iommu;
-iommu_flush_context_global(iommu, 0);
-flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
+/*
+ * The current logic for rc returns:
+ *   - positive  invoke iommu_flush_write_buffer to flush cache.
+ *   - zero  on success.
+ *   - negative  on failure. Continue to flush IOMMU IOTLB on a
+ *   best effort basis.
+ *
+ * Moreover, IOMMU flush handlers flush_context_qi and flush_iotlb_qi
+ * (or flush_context_reg and flush_iotlb_reg, deep functions in the
+ * call trees of iommu_flush_context_global and 
iommu_flush_iotlb_global)
+ * are with the same logic to 

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On Thursday, May 26, 2016 11:57 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> "Xu, Quan" <quan...@intel.com> 05/26/16 12:38 PM >>>
> >On May 25, 2016 4:30 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> The patch getting too large is easy to deal with: Split it at a
> >> reasonable boundary.
> >I recall your suggestion: top one first, then low level one..
> >I am better not to make this patch as a first one, as this is really a low 
> >level
> one.
> >Then, I need to change condition from 'if ( !rc )'  to ' if ( rc < 0 )'
> >in my series. (but if this series would be merged together, I don't need to
> think about it.) Does it make sense?
> 
> I'm afraid I'm lacking context.

when the propagation value is positive, this patch fixes this flush issue
as follows:
  - call iommu_flush_write_buffer() to flush cache.
  - return zero.

So the condition can be 'if( rc )' for error, but if this patch is not a first 
one in series,
I need to change condition from 'if ( rc )'  to ' if ( rc < 0 )'.. as the 
propagation value may be positive..

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 26, 2016 6:38 PM, Xu, Quan <quan...@intel.com> wrote:
> On May 25, 2016 4:30 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > The patch getting too large is easy to deal with: Split it at a
> > reasonable boundary.
> 
> Jan,
> If I follow the below rule, I need to merge most of patches into this one. I 
> can't
> find a reasonable boundary.
> I recall your suggestion: top one first, then low level one..
> I am better not to make this patch as a first one, as this is really a low 
> level one.
> Then, I need to change condition from 'if ( !rc )'  

Sorry, a typo, 'if ( rc )'

btw, the __must_check annotation is helpful, and  we have  multiple rounds  
review..
I think a big patch is not a big deal. 

Quan

> to ' if ( rc < 0 )' in my series.
> (but if this series would be merged together, I don't need to think about it.)
> Does it make sense?
> 
> Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Question about the usage of spinlock

[Xu, Quan]

On May 26, 2016 9:31 AM, 조현권  wrote:
> I am studying xen 4.6.0 version now and wonder the usage of spinlock which is 
> located in free_heap_pages(xen/common/page_alloc. c)
> As its memory setup is ahead of multicore initialization, spinlock seems to 
> be unnecessary during booting because it uses only single core
> spinlock looks required after multicore initialization finished and when xen 
> needs to free memory space during work
> However i experimented not to use spin lock in free_heap_pages and 
> dom0(linux-kernel) was booted well. It does not cause any error using 
> internet or application program even reboot
> Are therr any special reason to use spinlock?
> My experiment environment was ODROID-XU4 which uses arm architecture

Ah, you can try to test more for DomU..
Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 25, 2016 4:30 PM, Jan Beulich  wrote:
> The patch getting too large is easy to deal with: Split it at a reasonable
> boundary.

Jan, 
If I follow the below rule, I need to merge most of patches into this one. I 
can't find a reasonable boundary.
I recall your suggestion: top one first, then low level one..
I am better not to make this patch as a first one, as this is really a low 
level one.
Then, I need to change condition from 'if ( !rc )'  to ' if ( rc < 0 )' in my 
series. (but if this series would be merged together, I don't need to think 
about it.)
Does it make sense?

Quan


> It's one thing that I want to be clear: Any conversion of a void
> return type of some function to non-void should be accompanied by it at the
> same time becoming __must_check. I dislike having to repeat yet again what I
> have been saying a number of times: Without doing so, it is harder for you as
> the person writing the patch to verify all callers deal with errors, and it's
> perhaps even harder for reviewers to verify you did.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 23, 2016 9:31 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -1391,13 +1399,26 @@ int domain_context_mapping_one(
> >  spin_unlock(>lock);
> >
> >  /* Context entry was previously non-present (with domid 0). */
> > -if ( iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> > -DMA_CCMD_MASK_NOBIT, 1) )
> > -iommu_flush_write_buffer(iommu);
> > -else
> > +rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) |
> > + devfn,
> > +DMA_CCMD_MASK_NOBIT, 1);
> > +
> > +/*
> > + * The current logic for rc returns:
> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > + *   - zero  success.
> > + *   - negative  failure. Continue to flush IOMMU IOTLB on a best
> > + *   effort basis.
> > + */
> > +if ( rc <= 0 )
> >  {
> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> > +
> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> 
> If rc was negative before this call, you may end up returning success without
> having been successful. 

You are right.
IMO I need to add  'ret' here. Then ..

+if ( rc <= 0 )
 {
 int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
-iommu_flush_iotlb_dsi(iommu, iommu_domid, 0, flush_dev_iotlb);
+int ret;
+
+ret = iommu_flush_iotlb_dsi(iommu, iommu_domid, 0, flush_dev_iotlb);
+ASSERT(ret <= 0);
+
+if ( !rc )
+rc = ret;
+}


Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 26, 2016 12:02 AM, Jan Beulich  wrote:
> >>> On 25.05.16 at 17:34,  wrote:
> > On May 23, 2016 10:19 PM, Jan Beulich  wrote:
> >> >>> On 18.05.16 at 10:08,  wrote:
> >> > +unsigned long type,
> >> > +int preemptible)
> >> >  {
> >> >  unsigned long nx, x, y = page->u.inuse.type_info;
> >> > -int rc = 0;
> >> > +int rc = 0, ret = 0;
> >> >
> >> >  ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
> >> >
> >> > @@ -2578,11 +2579,11 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >> >  if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
> >> >  {
> >> >  if ( (x & PGT_type_mask) == PGT_writable_page )
> >> > -iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
> >> > +ret = iommu_unmap_page(d, mfn_to_gmfn(d,
> >> > + page_to_mfn(page)));
> >> >  else if ( type == PGT_writable_page )
> >> > -iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> >> > -   page_to_mfn(page),
> >> > -   IOMMUF_readable|IOMMUF_writable);
> >> > +ret = iommu_map_page(d, mfn_to_gmfn(d,
> page_to_mfn(page)),
> >> > + page_to_mfn(page),
> >> > +
> >> > + IOMMUF_readable|IOMMUF_writable);
> >> >  }
> >> >  }
> >> >
> >> > @@ -2599,6 +2600,9 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >> >  if ( (x & PGT_partial) && !(nx & PGT_partial) )
> >> >  put_page(page);
> >> >
> >> > +if ( !rc )
> >> > +rc = ret;
> >>
> >> I know I've seen this a couple of time already, but with the special
> >> purpose of "ret" I really wonder whether a more specific name
> >> wouldn't be warranted - e.g. "iommu_rc" or "iommu_ret".
> >
> >
> > rc is return value for this function, and no directly association with
> > IOMMU related code ( rc is only for alloc_page_type() ).
> > So the rc cannot be "iommu_rc"..
> >
> > ret can be "iommu_ret", but I think the pair 'rc' / 'ret' may look good.
> 
> Well, I'm not entirely opposed to keeping the names, but I continue to think
> that while at the call sites the shorter name is reasonable, it is quite a 
> bit less
> clear at the point where you conditionally update rc.
> 

I am open to it. What about 'rc' / 'iommu_ret' ?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 23, 2016 10:19 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> 
> > --- a/xen/arch/x86/mm.c
> > +++ b/xen/arch/x86/mm.c
> > @@ -2463,11 +2463,12 @@ static int __put_page_type(struct page_info
> > *page,  }
> >
> >
> > -static int __get_page_type(struct page_info *page, unsigned long type,
> > -   int preemptible)
> > +static int __must_check __get_page_type(struct page_info *page,
> 
> Such a __must_check is relatively pointless when all existing callers already
> check the return value (and surely all code inside mm.c knows it needs to), 
> and
> you don't at the same time make the non-static functions propagating its
> return value also __must_check.

I will drop this __must_check annotation.

> Hence I think best is to limit your effort to IOMMU functions for this patch 
> set.
>

Good idea.
 
> > +unsigned long type,
> > +int preemptible)
> >  {
> >  unsigned long nx, x, y = page->u.inuse.type_info;
> > -int rc = 0;
> > +int rc = 0, ret = 0;
> >
> >  ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
> >
> > @@ -2578,11 +2579,11 @@ static int __get_page_type(struct page_info
> *page, unsigned long type,
> >  if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
> >  {
> >  if ( (x & PGT_type_mask) == PGT_writable_page )
> > -iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
> > +ret = iommu_unmap_page(d, mfn_to_gmfn(d,
> > + page_to_mfn(page)));
> >  else if ( type == PGT_writable_page )
> > -iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> > -   page_to_mfn(page),
> > -   IOMMUF_readable|IOMMUF_writable);
> > +ret = iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> > + page_to_mfn(page),
> > +
> > + IOMMUF_readable|IOMMUF_writable);
> >  }
> >  }
> >
> > @@ -2599,6 +2600,9 @@ static int __get_page_type(struct page_info
> *page, unsigned long type,
> >  if ( (x & PGT_partial) && !(nx & PGT_partial) )
> >  put_page(page);
> >
> > +if ( !rc )
> > +rc = ret;
> 
> I know I've seen this a couple of time already, but with the special purpose 
> of
> "ret" I really wonder whether a more specific name wouldn't be warranted -
> e.g. "iommu_rc" or "iommu_ret".


rc is return value for this function, and no directly association with IOMMU 
related code ( rc is only for alloc_page_type() ).
So the rc cannot be "iommu_rc"..

ret can be "iommu_ret", but I think the pair 'rc' / 'ret' may look good.


> 
> > --- a/xen/arch/x86/mm/p2m-ept.c
> > +++ b/xen/arch/x86/mm/p2m-ept.c
> > @@ -658,7 +658,7 @@ bool_t ept_handle_misconfig(uint64_t gpa)
> >   *
> >   * Returns: 0 for success, -errno for failure
> >   */
> > -static int
> > +static int __must_check
> >  ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
> 
> Now adding the annotation here, without also (first) adding it to the p2m
> method which this gets used for (and which is this function's sole purpose), 
> is
> not useful at all. Please remember - we don't want this annotation because it
> looks good, but in order to make sure errors don't get wrongly ignored. That's
> why, from the beginning, I have been telling you that adding such annotations
> to other than new code means doing it top down (which you clearly don't do
> here).
> 

I will drop this __must_check annotation.

> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -171,20 +171,33 @@ void __hwdom_init iommu_hwdom_init(struct
> domain *d)
> >  {
> >  struct page_info *page;
> >  unsigned int i = 0;
> > +int rc = 0;
> > +
> >  page_list_for_each ( page, >page_list )
> >  {
> >  unsigned long mfn = page_to_mfn(page);
> >  unsigned long gfn = mfn_to_gmfn(d, mfn);
> >  unsigned int mapping = IOMMUF_readable;
> > +int ret;
> >
> >  if ( ((page->u.inuse.type_info & PGT_count_mask) == 0) ||
> >   ((page->u.inuse.type_info & PGT_type_mask)
> >== PGT_writable_page) )
> >  mapping |= IOMMUF_writable;
> > -hd->platform_ops->map_page(d, gfn, mfn, mapping);
> > +
> > +ret = hd->platform_ops->map_page(d, gfn, mfn, mapping);
> > +
> > +if ( unlikely(ret) )
> > +rc = ret;
> 
> This should be good enough, but I think it would be better if, just like
> elsewhere, you returned the first error instead of the last one.
> 

I will also apply it to this series.


(Jan, I know It is not an easy work to review these little things. I very 
appreciate it. Thank you!! )
Quan

___
Xen-devel mailing list

Re: [Xen-devel] [PATCH v5 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 25, 2016 4:07 PM, Jan Beulich  wrote:
> >>> On 25.05.16 at 08:41,  wrote:
> > On May 24, 2016 4:22 PM, Jan Beulich  wrote:
> >> >>> On 18.05.16 at 10:08,  wrote:
> >> >  static int device_power_down(void)  {
> >> > -console_suspend();
> >> > +if ( console_suspend() )
> >> > +return SAVED_CONSOLE;
> >>
> >> I said so on the previous round, and I need to repeat it now: If
> >> console_suspend() fails, you saved _nothing_.
> >>
> >
> > Ah, we may have some different views for SAVED_*, which I mean has
> > been saved and we are no need to resume.
> >
> > e.g. if console_suspend() fails, I did return SAVED_CONSOLE, reading
> > my patch again, and I really resume nothing at all.
> >
> > device_power_up()
> > {
> >  ...
> > +case SAVED_CONSOLE:
> > +break;
> > ...
> > }
> >
> >
> > I know we can also propagate SAVED_NONE for console_suspend() failure,
> > then we need adjust device_power_up() relevantly.
> 
> My main point is that the names of these enumerators should reflect their
> purpose. If one reads "SAVED_CONSOLE", then (s)he should be allowed this
> to mean that console state was saved (and hence needs to be restored upon
> error / resume).
> 


I'll follow this.. it is much better than what I understand.


> >> > -time_suspend();
> >> > +if ( time_suspend() )
> >> > +return SAVED_TIME;
> >> >
> >> > -i8259A_suspend();
> >> > +if ( i8259A_suspend() )
> >> > +return SAVED_I8259A;
> >> >
> >> > +/* ioapic_suspend cannot fail */
> >> >  ioapic_suspend();
> >> >
> >> > -iommu_suspend();
> >> > +if ( iommu_suspend() )
> >> > +return SAVED_IOMMU;
> >> >
> >> > -lapic_suspend();
> >> > +if ( lapic_suspend() )
> >> > +return SAVED_LAPIC;
> >> >
> >> >  return 0;
> >>
> >> And this silently means SAVED_NONE, whereas here you saved everything.
> >> Yielding clearly bogus code ...
> >>
> >
> >
> >  '0' is just on success here.  Look at the condition where we call
> > device_power_up():
> >
> > +if ( error > 0 )
> > +device_power_up(error);
> >
> > Then, it is not bogus code.
> 
> See above: Zero should not mean both "nothing saved" and "saved
> everything".
> 
> >> Also, having come here - did I miss iommu_flush_iotlb_global()
> >> gaining a __must_check annotation somewhere?
> >
> > I will add __must_check annotation to iommu_flush_iotlb_global().
> >
> >> And the struct iommu_flush pointers
> >> and handlers? And, by analogy, iommu_flush_context_*()?
> >
> > I am better only add __must_check annotation to flush->iotlb and
> > handlers, but leaving flush->context/handers and
> > iommu_flush_context_*() as are in current patch set..
> > the coming patch set will fix them.
> 
> I don't follow the logic behind this: The purpose of this series is to make 
> sure
> flushing errors get properly bubbled up, which includes adding __must_check
> annotations. I'm not saying this needs to happen in this patch, but it should
> happen in this series

I will add a patch..

> (and please following the same basic model: A caller or a
> __must_check function should either already be __must_check, or should
> become so at the same time).
> 

Agreed.

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 25, 2016 4:30 PM, Jan Beulich  wrote:
> >>> On 25.05.16 at 10:04,  wrote:
> > On May 23, 2016 11:43 PM, Jan Beulich  wrote:
> >> >>> On 23.05.16 at 17:22,  wrote:
> >> > On May 23, 2016 9:31 PM, Jan Beulich  wrote:
> >> >> >>> On 18.05.16 at 10:08,  wrote:
> >> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >> >> > @@ -557,14 +557,16 @@ static void iommu_flush_all(void)
> >> >> >  }
> >> >> >  }
> >> >> >
> >> >> > -static void __intel_iommu_iotlb_flush(struct domain *d,
> >> >> > unsigned long
> >> gfn,
> >> >> > -int dma_old_pte_present, unsigned int page_count)
> >> >> > +static int __intel_iommu_iotlb_flush(struct domain *d, unsigned
> >> >> > +long
> >> gfn,
> >> >> > + bool_t dma_old_pte_present,
> >> >> > + unsigned int page_count)
> >> >>
> >> >> I realize you say so in the overview mail, but the continuing lack
> >> >> of __must_check here causes review trouble again. And I have a
> >> >> hard time seeing how adding these annotations right away would
> >> >> "disrupt the order", as long as the series is properly ordered / broken 
> >> >> up.
> >> >>
> >> >
> >> > If I add __must_check annotations here right now, e.g.
> >> >
> >> > -static void intel_iommu_iotlb_flush()
> >> > +static int __must_check iommu_flush_iotlb_pages()
> >> >
> >> > ...
> >> >
> >> > @@ -179,8 +179,9 @@ struct iommu_ops {
> >> > -void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned 
> >> > int
> >> > page_count);
> >> > +int __must_check (*iotlb_flush)(struct domain *d, unsigned
> >> > + long gfn,
> >> > unsigned int page_count);
> >> > ...
> >> > }
> >> >
> >> > Should belong  to here too.
> >>
> >> Correct. And where's the problem?
> >>
> >
> > IMO it is not a big deal..
> >
> > I think this makes this patch 1 fat.. why not focus on the positive
> > propagation value from IOMMU flush interfaces in this patch.
> > If we are clear I will add annotation and rename them in another
> > patches, it is acceptable to me.
> 
> The patch getting too large is easy to deal with: Split it at a reasonable
> boundary. It's one thing that I want to be clear: Any conversion of a void
> return type of some function to non-void should be accompanied by it at the
> same time becoming __must_check. I dislike having to repeat yet again what I
> have been saying a number of times: Without doing so, it is harder for you as
> the person writing the patch to verify all callers deal with errors, and it's
> perhaps even harder for reviewers to verify you did.
> 

It is clear to me, but I was lock of attention on reviewers part.
I'll follow your suggestion from now on.. thanks.

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 23, 2016 11:43 PM, Jan Beulich  wrote:
> >>> On 23.05.16 at 17:22,  wrote:
> > On May 23, 2016 9:31 PM, Jan Beulich  wrote:
> >> >>> On 18.05.16 at 10:08,  wrote:
> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >> > @@ -557,14 +557,16 @@ static void iommu_flush_all(void)
> >> >  }
> >> >  }
> >> >
> >> > -static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long
> gfn,
> >> > -int dma_old_pte_present, unsigned int page_count)
> >> > +static int __intel_iommu_iotlb_flush(struct domain *d, unsigned long
> gfn,
> >> > + bool_t dma_old_pte_present,
> >> > + unsigned int page_count)
> >>
> >> I realize you say so in the overview mail, but the continuing lack of
> >> __must_check here causes review trouble again. And I have a hard time
> >> seeing how adding these annotations right away would "disrupt the
> >> order", as long as the series is properly ordered / broken up.
> >>
> >
> > If I add __must_check annotations here right now, e.g.
> >
> > -static void intel_iommu_iotlb_flush()
> > +static int __must_check iommu_flush_iotlb_pages()
> >
> > ...
> >
> > @@ -179,8 +179,9 @@ struct iommu_ops {
> > -void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int
> > page_count);
> > +int __must_check (*iotlb_flush)(struct domain *d, unsigned long
> > + gfn,
> > unsigned int page_count);
> > ...
> > }
> >
> > Should belong  to here too.
> 
> Correct. And where's the problem?
>

IMO it is not a big deal..

I think this makes this patch 1 fat.. why not focus on the positive propagation 
value from IOMMU flush interfaces in this patch.
If we are clear I will add annotation and rename them in another patches, it is 
acceptable to me.

Furthermore, we also need to add (from patch 4):

-static void dma_pte_clear_one(struct domain *domain, u64 addr)
+static int __must_check dma_pte_clear_one(struct domain *domain, u64 addr)
{
...
-__intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
+rc = __intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
...
}

In this patch.

 
> >> > +DMA_CCMD_MASK_NOBIT, 1);
> >> > +
> >> > +/*
> >> > + * The current logic for rc returns:
> >> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> >> > + *   - zero  success.
> >> > + *   - negative  failure. Continue to flush IOMMU IOTLB on a best
> >> > + *   effort basis.
> >> > + */
> >> > +if ( rc <= 0 )
> >> >  {
> >> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >> > +
> >> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >>
> >> If rc was negative before this call, you may end up returning success
> > without
> >> having been successful. Furthermore I think it was you who last time
> >> round reminded me that
> >> iommu_flush_iotlb_dsi() can also return 1, which you don't take care of.
> >>
> >
> > Yes, the iommu_flush_iotlb_dsi() can also return 1.
> > Look at the call tree, at the beginning of
> > flush_context_qi()/flush_iotlb_qi(), or
> > flush_context_reg()/flush_iotlb_reg()..
> >
> > If rc was negative when we call iommu_flush_context_device(), it is
> > impossible to return 1 for iommu_flush_iotlb_dsi().
> 
> This is far from obvious, so please add a respective ASSERT() if you want to
> rely on that.
> 
> > IMO, furthermore, this should not belong  to comment.
> 
> ???

Think twice, I will add comments and a respective ASSERT()..

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 24, 2016 4:22 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > --- a/xen/arch/x86/acpi/power.c
> > +++ b/xen/arch/x86/acpi/power.c
> > @@ -43,36 +43,68 @@ struct acpi_sleep_info acpi_sinfo;
> >
> >  void do_suspend_lowlevel(void);
> >
> > +enum dev_power_saved
> > +{
> > +SAVED_NONE,/* None of device power saved */
> > +SAVED_CONSOLE, /* Device power of console saved */
> > +SAVED_TIME,/* Device power of time saved */
> > +SAVED_I8259A,  /* Device power of I8259A saved */
> > +SAVED_IOAPIC,  /* Device power of IOAPIC saved */
> > +SAVED_IOMMU,   /* Device power of IOMMU saved */
> > +SAVED_LAPIC,   /* Device power of LAPIC saved */
> > +};
> 
> Please avoid comments saying nothing substantially different than the code
> they accompany, and also not helping the reader to understand the
> commented code.
>

I'll drop these comments in v6.

 
> >  static int device_power_down(void)
> >  {
> > -console_suspend();
> > +if ( console_suspend() )
> > +return SAVED_CONSOLE;
> 
> I said so on the previous round, and I need to repeat it now: If
> console_suspend() fails, you saved _nothing_.
> 

Ah, we may have some different views for SAVED_*, which I mean has been saved 
and we are no need to resume.

e.g. if console_suspend() fails, I did return SAVED_CONSOLE, reading my patch 
again, and I really resume nothing at all.

device_power_up()
{
 ...
+case SAVED_CONSOLE:
+break;
...
}


I know we can also propagate SAVED_NONE for console_suspend() failure, then we 
need adjust device_power_up() relevantly.


> > -time_suspend();
> > +if ( time_suspend() )
> > +return SAVED_TIME;
> >
> > -i8259A_suspend();
> > +if ( i8259A_suspend() )
> > +return SAVED_I8259A;
> >
> > +/* ioapic_suspend cannot fail */
> >  ioapic_suspend();
> >
> > -iommu_suspend();
> > +if ( iommu_suspend() )
> > +return SAVED_IOMMU;
> >
> > -lapic_suspend();
> > +if ( lapic_suspend() )
> > +return SAVED_LAPIC;
> >
> >  return 0;
> 
> And this silently means SAVED_NONE, whereas here you saved everything.
> Yielding clearly bogus code ...
>


 '0' is just on success here.  Look at the condition where we call 
device_power_up():

+if ( error > 0 )
+device_power_up(error);

Then, it is not bogus code.
 

> > -static void device_power_up(void)
> > +static void device_power_up(enum dev_power_saved saved)
> >  {
> > -lapic_resume();
> > -
> > -iommu_resume();
> > -
> > -ioapic_resume();
> > -
> > -i8259A_resume();
> > -
> > -time_resume();
> > -
> > -console_resume();
> > +switch ( saved )
> > +{
> > +case SAVED_NONE:
> > +lapic_resume();
> 
> ... here and ...
> 
> > @@ -196,7 +232,7 @@ static int enter_state(u32 state)
> >  write_cr4(cr4 & ~X86_CR4_MCE);
> >  write_efer(read_efer());
> >
> > -device_power_up();
> > +device_power_up(SAVED_NONE);
> 
> ... here.

Then we need to call all of *_resume(). I think the logic is correct, but the 
SAVED_*  may be not what you suggested.

> 
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -541,20 +541,28 @@ static int iommu_flush_iotlb_psi(
> >  return status;
> >  }
> >
> > -static void iommu_flush_all(void)
> > +static int __must_check iommu_flush_all(void)
> >  {
> >  struct acpi_drhd_unit *drhd;
> >  struct iommu *iommu;
> >  int flush_dev_iotlb;
> > +int rc = 0;
> >
> >  flush_all_cache();
> >  for_each_drhd_unit ( drhd )
> >  {
> > +int ret;
> > +
> >  iommu = drhd->iommu;
> >  iommu_flush_context_global(iommu, 0);
> >  flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> > +ret = iommu_flush_iotlb_global(iommu, 0, flush_dev_iotlb);
> > +
> > +if ( unlikely(ret) )
> > +rc = ret;
> 
> Same as for earlier patches - "if ( unlikely(!rc) )" please.
> 


Yes,


> Also, having come here - did I miss iommu_flush_iotlb_global() gaining a
> __must_check annotation somewhere? 

I will add __must_check annotation to iommu_flush_iotlb_global().

> And the struct iommu_flush pointers
> and handlers? And, by analogy, iommu_flush_context_*()?
> 

I am better only add __must_check annotation to flush->iotlb and handlers,
but leaving flush->context/handers and  iommu_flush_context_*() as are in 
current patch set..
the coming patch set will fix them.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 05/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping.

[Xu, Quan]

On May 24, 2016 5:09 PM, Jan Beulich  wrote:
> >>> On 24.05.16 at 11:01,  wrote:
> > On May 23, 2016 11:53 PM, Jan Beulich  wrote:
> >> >>> On 18.05.16 at 10:08,  wrote:
> >> > Propagate the IOMMU Device-TLB flush error up to IOMMU mapping.
> >>
> >> Btw - there's little reason to repeat the title here.
> >>
> >
> > I'll drop it.
> >
> > Can I apply it to  other patches?
> 
> You mean dropping a description that simply repeats the title - yes, of 
> course.

Yes, I think so too.

> Even better would of course be, where meaningful, if you added actual
> descriptions.
> 

I am afraid I would make things worse, so I will just simply drop  the 
description that simply repeats the title.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 23, 2016 9:41 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > No spamming can occur.
> 
> May I suggest "No spamming of the log can occur", to set some context for
> what follows?
> 

Make sense.

> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -240,21 +240,49 @@ int iommu_map_page(struct domain *d,
> unsigned long gfn, unsigned long mfn,
> > unsigned int flags)  {
> >  const struct domain_iommu *hd = dom_iommu(d);
> > +int rc;
> >
> >  if ( !iommu_enabled || !hd->platform_ops )
> >  return 0;
> >
> > -return hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +
> > +if ( unlikely(rc) )
> 
> A more general word on the use of blank lines: I think their use is well 
> advised
> to separate logically (mostly) independent steps. In cases like this, where 
> you
> check the return value of a function, the two things really belong together
> imo. Using too little blank lines negatively affects readability, but using 
> too
> many easily leads to not seeing enough context anymore when looking at
> some code fragment.
> 

I will also apply it to other patches.

> > +{
> > +if ( !d->is_shutting_down && printk_ratelimit() )
> > +printk(XENLOG_ERR
> > +   "d%d: IOMMU mapping gfn %#lx mfn %#lx failed %d.",
> 
> I really dislike having to repeat this yet another time: No stops in log 
> messages
> please. Also to the reader of the log it would be unclear what the number at
> the end represents. May I suggest
> 
> printk(XENLOG_ERR
>"d%d: IOMMU mapping gfn %#lx to mfn %#lx failed: %d",
>d->domain_id, gfn, mfn, rc);
> 
> (arguably one might then also remove the words "gfn" and "mfn").
> 

To me, we are better not to remove 'gfn' / 'mfn', but I really need to add a 
'\n'.. then

printk(XENLOG_ERR
   "d%d: IOMMU mapping gfn %#lx to mfn %#lx failed: %d\n",
   d->domain_id, gfn, mfn, rc);


Quan

> Apart from these cosmetic issues I think this is fine now.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 05/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU mapping.

[Xu, Quan]

On May 23, 2016 11:53 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > Propagate the IOMMU Device-TLB flush error up to IOMMU mapping.
> 
> Btw - there's little reason to repeat the title here.
> 

I'll drop it.

Can I apply it to  other patches?


> > @@ -295,12 +297,23 @@ static void __hwdom_init
> amd_iommu_hwdom_init(struct domain *d)
> >   * a pfn_valid() check would seem desirable here.
> >   */
> >  if ( mfn_valid(pfn) )
> > -amd_iommu_map_page(d, pfn, pfn,
> > -   IOMMUF_readable|IOMMUF_writable);
> > +{
> > +int ret;
> > +
> > +ret = amd_iommu_map_page(d, pfn, pfn,
> > +
> > + IOMMUF_readable|IOMMUF_writable);
> > +
> > +if ( unlikely(ret) )
> > +rc = ret;
> > +}
> 
> So you do the adjustment needed to add __must_check to
> amd_iommu_map_page(), but you don't actually add the annotation.
> Is there a reason for this?
>

Sorry, I missed it.
I really need a __must_check to amd_iommu_map_page() in 
include/asm-x86/hvm/svm/amd-iommu-proto.h.


 
> And of course the comment to an earlier patch applies regarding which error
> to return.

I'll apply it to all of my patch set.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 06/10] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones).

[Xu, Quan]

On May 24, 2016 4:34 PM, Jan Beulich  wrote:
> but indeed if you
> drop the annotations from non-IOMMU functions (unless, as said, you mean
> to also add them further up the call trees), then I think things should be
> coming close.
> 

I'll drop the annotations from non-IOMMU functions.
Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 06/10] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones).

[Xu, Quan]

On May 24, 2016 3:09 PM, Tian, Kevin  wrote:
> > From: Jan Beulich [mailto:jbeul...@suse.com]
> > Sent: Tuesday, May 24, 2016 3:02 PM
> >
> > >>> On 24.05.16 at 03:16,  wrote:
> > > On May 24, 2016 12:06 AM, Jan Beulich  wrote:
> > >> >>> On 18.05.16 at 10:08,  wrote:
> > >> > --- a/xen/common/memory.c
> > >> > +++ b/xen/common/memory.c
> > >> > @@ -633,9 +633,9 @@ static long
> > >>
> memory_exchange(XEN_GUEST_HANDLE_PARAM(xen_memory_exchange_t)
> > >> arg)
> > >> >  return rc;
> > >> >  }
> > >> >
> > >> > -static int xenmem_add_to_physmap(struct domain *d,
> > >> > - struct xen_add_to_physmap *xatp,
> > >> > - unsigned int start)
> > >> > +static int __must_check xenmem_add_to_physmap(struct domain *d,
> > >> > +  struct
> > >> > +xen_add_to_physmap
> > > *xatp,
> > >> > +  unsigned int
> > >> > + start)
> > >> >  {
> > >>
> > >> As before - either you do this adding of annotations completely, or
> > >> you stop
> > > at
> > >> the IOMMU / MM boundary.
> > >
> > > I prefer to stop at the IOMMU / MM boundary. The IOMMU boundary is
> > > obvious, but what's the definition of MM boundary? I thought this is at
> MM boundary.
> >
> > Not sure what you mean to understand. The IOMMU / MM boundary is the
> > boundary between those two components, there's no talk of two
> > boundaries here, and hence the question is unclear to me.
> >
> > Jan
> 
> Hi, Quan,
> 
> A file-based map about IOMMU/MM boundary is under arch/x86/mm. You
> need focus on low-level interaction between IOMMU and MM components,
> i.e. when some state change in MM code (mostly p2m change) needs to
> conduct IOMMU operations.
> 
> Above xenmem is much higher level, which will be routed to various MM
> operations internally so you don't need bother here.

Jan / Kevin,

I thought the IOMMU / MM boundary is the MM functions (high level callers) 
which call iommu_* interfaces (such as,  iommu_map_page / iommu_unmap_page / 
iommu_iotlb_flush ...).
For this case, the xenmem_add_to_physmap() indeed calls iommu_iotlb_flush(),  
but xenmem_add_to_physmap() may be hypervisor interface, instead of MM 
interface.

If I drop this __must_check and fix patch 3 / patch 5, then I think 
__must_check would not be a block issue.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 07/10] IOMMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (leaf ones).

[Xu, Quan]

On May 18, 2016 4:08 PM, Quan Xu <quan...@intel.com> wrote:
> Propagate the IOMMU Device-TLB flush error up to the
> iommu_iotlb_flush{,_all}.
> 
> This patch fixes the leaf ones.
> --- a/xen/drivers/passthrough/vtd/iommu.c
> +++ b/xen/drivers/passthrough/vtd/iommu.c
> -static void intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> unsigned int page_count)
> +static int __must_check iommu_flush_iotlb_pages(struct domain *d,
> +unsigned long gfn,
> +unsigned int
> +page_count)
>  {
> -__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
> +return iommu_flush_iotlb(d, gfn, 1, page_count);
>  }
> 
> -static void intel_iommu_iotlb_flush_all(struct domain *d)
> +static int iommu_flush_iotlb_all(struct domain *d)

Sorry,  I should add __must_check annotation here too.

Quan

>  {
> -__intel_iommu_iotlb_flush(d, INVALID_GFN, 0, 0);
> +return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
>  }
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 06/10] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones).

[Xu, Quan]

On May 24, 2016 12:06 AM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > --- a/xen/common/memory.c
> > +++ b/xen/common/memory.c
> > @@ -633,9 +633,9 @@ static long
> memory_exchange(XEN_GUEST_HANDLE_PARAM(xen_memory_exchange_t)
> arg)
> >  return rc;
> >  }
> >
> > -static int xenmem_add_to_physmap(struct domain *d,
> > - struct xen_add_to_physmap *xatp,
> > - unsigned int start)
> > +static int __must_check xenmem_add_to_physmap(struct domain *d,
> > +  struct xen_add_to_physmap 
> > *xatp,
> > +  unsigned int start)
> >  {
> 
> As before - either you do this adding of annotations completely, or you stop 
> at
> the IOMMU / MM boundary. 

I prefer to stop at the IOMMU / MM boundary. The IOMMU boundary is obvious, but 
what's the definition of MM boundary? I thought this is at MM boundary.

Quan

> And note that this addition was not covered by
> the R-b tag I had offered during the
> v4 review.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 23, 2016 9:31 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -557,14 +557,16 @@ static void iommu_flush_all(void)
> >  }
> >  }
> >
> > -static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> > -int dma_old_pte_present, unsigned int page_count)
> > +static int __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> > + bool_t dma_old_pte_present,
> > + unsigned int page_count)
> 
> I realize you say so in the overview mail, but the continuing lack of
> __must_check here causes review trouble again. And I have a hard time seeing
> how adding these annotations right away would "disrupt the order", as long
> as the series is properly ordered / broken up.
> 

If I add __must_check annotations here right now, e.g. 

-static void intel_iommu_iotlb_flush()
+static int __must_check iommu_flush_iotlb_pages()

... 

@@ -179,8 +179,9 @@ struct iommu_ops {
-void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int 
page_count);
+int __must_check (*iotlb_flush)(struct domain *d, unsigned long gfn, 
unsigned int page_count);
...
}

Should belong  to here too. 




> > @@ -579,23 +581,28 @@ static void __intel_iommu_iotlb_flush(struct
> > domain *d, unsigned long gfn,
> >
> >  flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >  iommu_domid= domain_iommu_domid(d, iommu);
> > +
> >  if ( iommu_domid == -1 )
> 
> I appreciate you adding blank lines where needed, but this one looks spurious.
> 
> > @@ -1391,13 +1399,26 @@ int domain_context_mapping_one(
> >  spin_unlock(>lock);
> >
> >  /* Context entry was previously non-present (with domid 0). */
> > -if ( iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> > -DMA_CCMD_MASK_NOBIT, 1) )
> > -iommu_flush_write_buffer(iommu);
> > -else
> > +rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) |
> > + devfn,
> 
> If you already touch such code, I'd appreciate if you did away with the open
> coding of pre-canned macros or inline functions (PCI_BDF2() in this case).

I will enhance it in v6.

> 
> > +DMA_CCMD_MASK_NOBIT, 1);
> > +
> > +/*
> > + * The current logic for rc returns:
> > + *   - positive  invoke iommu_flush_write_buffer to flush cache.
> > + *   - zero  success.
> > + *   - negative  failure. Continue to flush IOMMU IOTLB on a best
> > + *   effort basis.
> > + */
> > +if ( rc <= 0 )
> >  {
> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> > +
> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> 
> If rc was negative before this call, you may end up returning success without
> having been successful. Furthermore I think it was you who last time round
> reminded me that
> iommu_flush_iotlb_dsi() can also return 1, which you don't take care of.
> 

Yes, the iommu_flush_iotlb_dsi() can also return 1.
Look at the call tree, at the beginning of flush_context_qi()/flush_iotlb_qi(), 
or flush_context_reg()/flush_iotlb_reg()..

If rc was negative when we call iommu_flush_context_device(), it is impossible 
to return 1 for iommu_flush_iotlb_dsi().
 
IMO, furthermore, this should not belong  to comment.

> > @@ -1522,6 +1544,7 @@ int domain_context_unmap_one(
> >  iommu_flush_cache_entry(context, sizeof(struct context_entry));
> >
> >  iommu_domid= domain_iommu_domid(domain, iommu);
> > +
> >  if ( iommu_domid == -1 )
> 
> Seemingly stray addition of blank line again (more such below). And the code
> below has the same issue as that above.
> 
I will enhance it in v6.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On May 20, 2016 5:59 PM, Jan Beulich  wrote:
> >>> On 20.05.16 at 09:15,  wrote:
> > On May 17, 2016 10:00 PM, Jan Beulich  wrote:
> >> >>> On 22.04.16 at 12:54,  wrote:
> >> > --- a/xen/drivers/passthrough/vtd/qinval.c
> >> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> >> > @@ -206,10 +206,71 @@ static int invalidate_sync(struct iommu
> *iommu)
> >> >  return 0;
> >> >  }
> >> >
> >> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> >> > + u16 seg, u8 bus, u8 devfn) {
> >> > +struct domain *d = NULL;
> >> > +struct pci_dev *pdev;
> >> > +
> >> > +if ( test_bit(did, iommu->domid_bitmap) )
> >> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> >> > +
> >> > +/*
> >> > + * In case the domain has been freed or the IOMMU domid bitmap is
> >> > + * not valid, the device no longer belongs to this domain.
> >> > + */
> >> > +if ( d == NULL )
> >> > +return;
> >> > +
> >> > +pcidevs_lock();
> >> > +
> >> > +for_each_pdev(d, pdev)
> >> > +{
> >> > +if ( (pdev->seg == seg) &&
> >> > + (pdev->bus == bus) &&
> >> > + (pdev->devfn == devfn) )
> >> > +{
> >> > +ASSERT(pdev->domain);
> >> > +list_del(>domain_list);
> >> > +pdev->domain = NULL;
> >> > +pci_hide_existing_device(pdev);
> >> > +break;
> >> > +}
> >> > +}
> >>
> >> A loop like this is of course not ideal (especially for Dom0, which
> >> may have many devices). And I wonder why you, ...
> >>
> >> > @@ -134,8 +133,9 @@ int dev_invalidate_iotlb(struct iommu *iommu,
> >> > u16
> >> did,
> >> >  /* invalidate all translations:
> >> > sbit=1,bit_63=0,bit[62:12]=1
> > */
> >> >  sbit = 1;
> >> >  addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFF;
> >> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> >> > -  sid, sbit, addr);
> >> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> did,
> >> > +  pdev->seg, pdev->bus, 
> >> > pdev->devfn,
> >> > +  sbit, addr);
> >> >  break;
> >> >  case DMA_TLB_PSI_FLUSH:
> >> >  if ( !device_in_domain(iommu, pdev, did) ) @@ -154,8
> >> > +154,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> >> >  addr |= (((u64)1 << (size_order - 1)) - 1) << 
> >> > PAGE_SHIFT_4K;
> >> >  }
> >> >
> >> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> >> > -  sid, sbit, addr);
> >> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> did,
> >> > +  pdev->seg, pdev->bus, 
> >> > pdev->devfn,
> >> > +  sbit, addr);
> >> >  break;
> >>
> >> ... holding pdev in your hands here, don't just pass it down (which
> >> at once would make the function signature less convoluted: you could
> >> even eliminate the currently 2nd parameter that way).
> >
> > I am afraid we need to leave it as is.. this pdev , in
> > dev_invalidate_iotlb(), is 'struct pci_ats_dev', but we need a 'struct
> > pci_dev' to hide device in dev_invalidate_iotlb_timeout().
> >
> > 'struct pci_ats_dev' and 'struct pci_dev' are quite different,
> > however, SBDF is connection between them..
> 
> Oh, indeed. Yet - can't enable_ats_device() be passed a struct pci_dev *, and
> that be stored instead of SBDF inside struct pci_ats_dev?
> 

Make sense.  I appreciate your specific advice.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On May 17, 2016 10:00 PM, Jan Beulich  wrote:
> >>> On 22.04.16 at 12:54,  wrote:
> > --- a/xen/drivers/passthrough/vtd/qinval.c
> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> > @@ -206,10 +206,71 @@ static int invalidate_sync(struct iommu *iommu)
> >  return 0;
> >  }
> >
> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> > + u16 seg, u8 bus, u8 devfn) {
> > +struct domain *d = NULL;
> > +struct pci_dev *pdev;
> > +
> > +if ( test_bit(did, iommu->domid_bitmap) )
> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> > +
> > +/*
> > + * In case the domain has been freed or the IOMMU domid bitmap is
> > + * not valid, the device no longer belongs to this domain.
> > + */
> > +if ( d == NULL )
> > +return;
> > +
> > +pcidevs_lock();
> > +
> > +for_each_pdev(d, pdev)
> > +{
> > +if ( (pdev->seg == seg) &&
> > + (pdev->bus == bus) &&
> > + (pdev->devfn == devfn) )
> > +{
> > +ASSERT(pdev->domain);
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +break;
> > +}
> > +}
> 
> A loop like this is of course not ideal (especially for Dom0, which may have
> many devices). And I wonder why you, ...
> 
> > @@ -134,8 +133,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16
> did,
> >  /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
> >  sbit = 1;
> >  addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFF;
> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> > -  sid, sbit, addr);
> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, 
> > did,
> > +  pdev->seg, pdev->bus, 
> > pdev->devfn,
> > +  sbit, addr);
> >  break;
> >  case DMA_TLB_PSI_FLUSH:
> >  if ( !device_in_domain(iommu, pdev, did) ) @@ -154,8
> > +154,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> >  addr |= (((u64)1 << (size_order - 1)) - 1) << 
> > PAGE_SHIFT_4K;
> >  }
> >
> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> > -  sid, sbit, addr);
> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, 
> > did,
> > +  pdev->seg, pdev->bus, 
> > pdev->devfn,
> > +  sbit, addr);
> >  break;
> 
> ... holding pdev in your hands here, don't just pass it down (which at once
> would make the function signature less convoluted: you could even eliminate
> the currently 2nd parameter that way).
> 

Jan, 
I am afraid we need to leave it as is.. this pdev , in 
dev_invalidate_iotlb(), is 'struct pci_ats_dev',
but we need a 'struct pci_dev' to hide device in dev_invalidate_iotlb_timeout().

'struct pci_ats_dev' and 'struct pci_dev' are quite different, however, SBDF is 
connection between them..

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 1/3] vt-d: add a timeout parameter for Queued Invalidation

[Xu, Quan]

On May 19, 2016 7:36 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 19.05.16 at 13:26, <quan...@intel.com> wrote:
> > On May 19, 2016 2:13 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> >>> "Xu, Quan" <quan...@intel.com> 05/19/16 3:35 AM >>>
> >> >On May 19, 2016 8:33 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> >> A single default value for both IOMMU-side and device-side is
> >> >> anyway not optimal. What about introducing a new knob e.g.
> >> >> vtd_qi_device_timeout specifically for device-side flush while
> >> >> using vtd_qi_timeout for other places? If device-side timeout is
> >> >> not specified, it is
> >> then default to vtd_qi_timeout.
> >>
> >> There should imo be a single command line option, allowing for two
> >> values to be passed (e.g. comma-separated).
> >
> > As mentioned, 1 ms is enough for VT-d IOTLB/Context/IEC invalidation,
> > so we are no need to increasing the value of timeout or introduce a
> > boot-time changed parameter.
> > What about a constant (e.g. a macro), 1 ms, for VT-d IOTLB/Context/IEC
> > invalidation timeout.
> 
> If you're absolutely certain no-one will ever find a need to increase that 
> value -
> sure.
> 

Sure.
Also this was mentioned in Kevin's ' Revisit VT-d asynchronous flush issue ' , 
http://lists.xenproject.org/archives/html/xen-devel/2015-11/msg00041.html 

"""-For context/iotlb/iec flush, our measurements show worst cases <10us. We 
also confirmed with hardware team, that 1ms is large  enough for IOMMU internal 
flush. """

> > For Device-TLB invalidation, we can introduce 'vtd_qi_device_timeout',
> > which is boot-time changed, and 1 ms by default.
> 
> One question is whether making this a VT-d specific command line option is a
> good idea: Other IOMMU implementations ought to be in need of doing
> device IOTLB invalidation too, at least sooner or later.
> 

This is indeed farsighted. Agreed.

> The other question is whether any timeout lower than the current one can be
> considered safe (and hence is usable as a default).
> 

Actually the criticality, 1 ms, is from hardware team.
IOW, Our hardware team confirmed that 1ms should be enough for integrated PCI 
devices with ATS.
for discrete PCI devices with ATS, it's uncertain whether 1ms  or 10ms is too 
restrictive to them, but there are only a few devices now in the market.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86/HVM: fix forwarding of internally cached requests

[Xu, Quan]

On May 19, 2016 5:15 PM, Jan Beulich  wrote:
> >>> On 19.05.16 at 10:30,  wrote:
> > On April 28, 2016 11:13 PM, Jan Beulich  wrote:
> >> >>> On 25.04.16 at 10:40,  wrote:
> >> > With other patches also in place, still not work.
> >> > Jianzhong  has been left and Quan will take over the task.
> >>
> >> May I ask for another try, with current tip of staging plus
> >> http://lists.xenproject.org/archives/html/xen-devel/2016-
> >> 04/msg03661.html
> >> ?
> >
> > The same issue for rhel 6.4VM,  which cannot initialize VF driver
> > too.. the below is log  of rhel 6.4 VM:
> >  ..
> >  igbvf: :00.04.0: Invalid MAC Address: 00:00:00:00:00:00
> >  igbvf: probe of :00.04.0 failed with error -5
> >  ..
> >
> > But when I tried to initialize VF MAC in Dom0 with e.g.:
> >   $ip link set eth0 vf 0 mac 00:1E:67:65:93:01
> 
> Makes things even more strange, as things work fine for me with
> SLE11 and SLE12 guests. But I have to admit that the "Invalid MAC Address"
> looks quite unrelated, i.e. is this perhaps some completely different problem
> you have?

I tried to run SLE12 guest. Things are really working fine for me too..
But not for rhel 6.4 guest..

So far, I think the bug is not from xen hypervisor, just from vf driver.

Look at this bug, even from KVM --- igb VF can't work in KVM guest, 
https://bugzilla.kernel.org/show_bug.cgi?id=55421
 
> 
> Again, considering that you have a repro (while I don't), the easiest would be
> to simply log all MSI-X related actions (there shouldn't be too many), to see
> where things go wrong. Such a log alone would maybe already allow me to do
> further investigation.
> 

I need more time to read this part, but sure I will follow it.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 1/3] vt-d: add a timeout parameter for Queued Invalidation

[Xu, Quan]

On May 19, 2016 2:13 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> "Xu, Quan" <quan...@intel.com> 05/19/16 3:35 AM >>>
> >On May 19, 2016 8:33 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> A single default value for both IOMMU-side and device-side is anyway
> >> not optimal. What about introducing a new knob e.g.
> >> vtd_qi_device_timeout specifically for device-side flush while using
> >> vtd_qi_timeout for other places? If device-side timeout is not specified, 
> >> it is
> then default to vtd_qi_timeout.
> 
> There should imo be a single command line option, allowing for two values to
> be passed (e.g. comma-separated).
> 

As mentioned, 1 ms is enough for VT-d IOTLB/Context/IEC invalidation, so we are 
no need to increasing the value of timeout or introduce a boot-time changed 
parameter.
What about a constant (e.g. a macro), 1 ms, for VT-d IOTLB/Context/IEC 
invalidation timeout.

For Device-TLB invalidation, we can introduce 'vtd_qi_device_timeout', which is 
boot-time changed, and 1 ms by default.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86/HVM: fix forwarding of internally cached requests

[Xu, Quan]

On April 28, 2016 11:13 PM, Jan Beulich  wrote:
> >>> On 25.04.16 at 10:40,  wrote:
> > With other patches also in place, still not work.
> > Jianzhong  has been left and Quan will take over the task.
> 
> May I ask for another try, with current tip of staging plus
> http://lists.xenproject.org/archives/html/xen-devel/2016-
> 04/msg03661.html
> ?

Jan,

The same issue for rhel 6.4VM,  which cannot initialize VF driver too.. the 
below is log  of rhel 6.4 VM:
 .. 
 igbvf: :00.04.0: Invalid MAC Address: 00:00:00:00:00:00
 igbvf: probe of :00.04.0 failed with error -5
 ..

But when I tried to initialize VF MAC in Dom0 with e.g.:
  $ip link set eth0 vf 0 mac 00:1E:67:65:93:01

Then to create rhel 6.4 VM again, the VF is working for rhel 6.4 VM.

I will ask QA team to help me verify it on win2k8 guest (I'm not sure whether 
the win2k8 network driver is working or not).

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 1/3] vt-d: add a timeout parameter for Queued Invalidation

[Xu, Quan]

On May 19, 2016 8:33 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Jan Beulich [mailto:jbeul...@suse.com]
> > Sent: Wednesday, May 18, 2016 11:05 PM
> >
> > >>> On 18.05.16 at 14:53, <quan...@intel.com> wrote:
> > > On May 17, 2016 3:48 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > >> >>> On 17.05.16 at 05:19, <kevin.t...@intel.com> wrote:
> > >> >>  From: Xu, Quan
> > >> >> Sent: Monday, May 16, 2016 11:26 PM
> > >> >>
> > >> >> On May 13, 2016 11:28 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > >> >> > >>> On 22.04.16 at 12:54, <quan...@intel.com> wrote:
> > >> >> > > --- a/docs/misc/xen-command-line.markdown
> > >> >> > > +++ b/docs/misc/xen-command-line.markdown
> > >> >> > > @@ -1532,6 +1532,16 @@ Note that if **watchdog** option is
> > >> >> > > also
> > >> >> > specified vpmu will be turned off.
> > >> >> > >  As the virtualisation is not 100% safe, don't use the vpmu
> > >> >> > > flag on production systems (see
> > >> >> > > http://xenbits.xen.org/xsa/advisory-
> > >> 163.html)!
> > >> >> > >
> > >> >> > > +### vtd\_qi\_timeout (VT-d)
> > >> >> > > +> `= `
> > >> >> > > +
> > >> >> > > +> Default: `1`
> > >> >> > > +
> > >> >> > > +Specify the timeout of the VT-d Queued Invalidation in
> milliseconds.
> > >> >> > > +
> > >> >> > > +By default, the timeout is 1ms. When you see error 'Queue
> > >> >> > > +invalidate wait descriptor timed out', try increasing this value.
> > >> >> >
> > >> >> > So when someone enables ATS, will the 1ms timeout apply to the
> > >> >> > dev iotlb invalidations too?
> > >> >>
> > >> >> Yes,
> > >> >> The timeout is the same for IOTLB, Context, IEC and Device-TLB
> invalidation.
> > >> >>
> > >> >> > If so, that's surely too short, and would ideally be adjusted
> > >> >> > automatically, but the need for a higher timeout in that case
> > >> >> > should in any event be mentioned here.
> > >> >>
> > >> >> I can try to use 1ms for IOTLB, Context and  IEC invalidation.
> > >> >> As mentioned, 1 ms is enough for IOTLB, Context and  IEC invalidation.
> > >> >> What about 10 ms for Device-TLB (10 ms is just a higher timeout,
> > >> >> no
> > >> specific meaning)?
> > >> >
> > >> > I remember in earlier discussion we agreed to use 1ms as the
> > >> > default for both IOMMU-side and device-side flushes. For
> > >> > device-side flushes, we checked internal HW team that 1ms is a
> > >> > reasonable threshold for integrated devices. It's likely
> > >> > insufficient for discrete devices. We may check any automatic
> > >> > adjustment method later when it becomes a real problem. For now,
> please elaborate above information in the text.
> > >>
> > >> Well, taking care of automation later is fine with me, but tying
> > >> everything to a single timeout, when device iotlb invalidation may
> > >> require a much larger value, isn't.
> > >>
> > >
> > > A little bit confused. Check it -- could I leave patch 1/3 as is?
> >
> > The patch can imo remain as is only if the new default timeout is
> > large enough for all possible cases (including those users who are
> > adventurous enough to turn on ATS).
> >

Jan,

I only have an ATS device (MYRICOM Inc. Myri-10G Dual-Protocol NIC). 1 ms is 
large enough for invalidation so far.
Any suggestion for this new default timeout?


> A single default value for both IOMMU-side and device-side is anyway not
> optimal. What about introducing a new knob e.g. vtd_qi_device_timeout
> specifically for device-side flush while using vtd_qi_timeout for other 
> places? If
> device-side timeout is not specified, it is then default to vtd_qi_timeout.
>

IMO, we are better to introduce only one  variable  for VT-d invalidation.
The users may be not interested in such a detailed VT-d knowledge. Also this is 
taking into consideration of  consistency.  :)

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 3/3] vt-d: fix vt-d Device-TLB flush timeout issue

[Xu, Quan]

On May 17, 2016 10:00 PM, Jan Beulich  wrote:
> >>> On 22.04.16 at 12:54,  wrote:
> > --- a/xen/drivers/passthrough/vtd/qinval.c
> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> > @@ -206,10 +206,71 @@ static int invalidate_sync(struct iommu *iommu)
> >  return 0;
> >  }
> >
> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> > + u16 seg, u8 bus, u8 devfn) {
> > +struct domain *d = NULL;
> > +struct pci_dev *pdev;
> > +
> > +if ( test_bit(did, iommu->domid_bitmap) )
> > +d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> > +
> > +/*
> > + * In case the domain has been freed or the IOMMU domid bitmap is
> > + * not valid, the device no longer belongs to this domain.
> > + */
> > +if ( d == NULL )
> > +return;
> > +
> > +pcidevs_lock();
> > +
> > +for_each_pdev(d, pdev)
> > +{
> > +if ( (pdev->seg == seg) &&
> > + (pdev->bus == bus) &&
> > + (pdev->devfn == devfn) )
> > +{
> > +ASSERT(pdev->domain);
> > +list_del(>domain_list);
> > +pdev->domain = NULL;
> > +pci_hide_existing_device(pdev);
> > +break;
> > +}
> > +}
> 
> A loop like this is of course not ideal (especially for Dom0, which may have
> many devices). And I wonder why you, ...
> 
> > @@ -134,8 +133,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16
> did,
> >  /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
> >  sbit = 1;
> >  addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFF;
> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> > -  sid, sbit, addr);
> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, 
> > did,
> > +  pdev->seg, pdev->bus, 
> > pdev->devfn,
> > +  sbit, addr);
> >  break;
> >  case DMA_TLB_PSI_FLUSH:
> >  if ( !device_in_domain(iommu, pdev, did) ) @@ -154,8
> > +154,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> >  addr |= (((u64)1 << (size_order - 1)) - 1) << 
> > PAGE_SHIFT_4K;
> >  }
> >
> > -rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> > -  sid, sbit, addr);
> > +rc = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, 
> > did,
> > +  pdev->seg, pdev->bus, 
> > pdev->devfn,
> > +  sbit, addr);
> >  break;
> 
> ... holding pdev in your hands here, don't just pass it down (which at once
> would make the function signature less convoluted: you could even eliminate
> the currently 2nd parameter that way).
> 

Good idea!!

Quan

















___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 1/3] vt-d: add a timeout parameter for Queued Invalidation

[Xu, Quan]

On May 17, 2016 3:48 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 17.05.16 at 05:19, <kevin.t...@intel.com> wrote:
> >>  From: Xu, Quan
> >> Sent: Monday, May 16, 2016 11:26 PM
> >>
> >> On May 13, 2016 11:28 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >> > >>> On 22.04.16 at 12:54, <quan...@intel.com> wrote:
> >> > > --- a/docs/misc/xen-command-line.markdown
> >> > > +++ b/docs/misc/xen-command-line.markdown
> >> > > @@ -1532,6 +1532,16 @@ Note that if **watchdog** option is also
> >> > specified vpmu will be turned off.
> >> > >  As the virtualisation is not 100% safe, don't use the vpmu flag
> >> > > on production systems (see http://xenbits.xen.org/xsa/advisory-
> 163.html)!
> >> > >
> >> > > +### vtd\_qi\_timeout (VT-d)
> >> > > +> `= `
> >> > > +
> >> > > +> Default: `1`
> >> > > +
> >> > > +Specify the timeout of the VT-d Queued Invalidation in milliseconds.
> >> > > +
> >> > > +By default, the timeout is 1ms. When you see error 'Queue
> >> > > +invalidate wait descriptor timed out', try increasing this value.
> >> >
> >> > So when someone enables ATS, will the 1ms timeout apply to the dev
> >> > iotlb invalidations too?
> >>
> >> Yes,
> >> The timeout is the same for IOTLB, Context, IEC and Device-TLB 
> >> invalidation.
> >>
> >> > If so, that's surely too short, and would ideally be adjusted
> >> > automatically, but the need for a higher timeout in that case
> >> > should in any event be mentioned here.
> >>
> >> I can try to use 1ms for IOTLB, Context and  IEC invalidation. As
> >> mentioned, 1 ms is enough for IOTLB, Context and  IEC invalidation.
> >> What about 10 ms for Device-TLB (10 ms is just a higher timeout,  no
> specific meaning)?
> >
> > I remember in earlier discussion we agreed to use 1ms as the default
> > for both IOMMU-side and device-side flushes. For device-side flushes,
> > we checked internal HW team that 1ms is a reasonable threshold for
> > integrated devices. It's likely insufficient for discrete devices. We
> > may check any automatic adjustment method later when it becomes a real
> > problem. For now, please elaborate above information in the text.
> 
> Well, taking care of automation later is fine with me, 
> but tying everything to a
> single timeout, when device iotlb invalidation may require a much larger 
> value,
> isn't.
>

A little bit confused. Check it -- could I leave patch 1/3 as is? 

btw, I have tested it against the last commit, no conflict.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v5 00/10] Check VT-d Device-TLB flush error

[Xu, Quan]

On May 18, 2016 6:21 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:08,  wrote:
> > --Changes in v5:
> >
> > patch 1:
> >   * add the missing blank line.
> >   * add comments.
> >   * if iommu_flush_context_device failed, continue to flush IOMMU IOTLB on
> a best effort basis.
> >   * __defer__ to:
> >   - rename __intel_iommu_iotlb_flush to iommu_flush_iotlb
> >   - rename intel_iommu_iotlb_flush to iommu_flush_iotlb_pages
> >   - rename intel_iommu_iotlb_flush_all to iommu_flush_iotlb_all
> >   - add __must_check annotation
> > in patch 7 / 8,
> > otherwise, this will disrupt the order due to __must_check annotation.
> 
> I've peeked into patch 1, and this information is still not there, despite you
> having been asked before to put this change information into the individual
> patches to aid reviewers.
> 

Sorry, I misunderstood the 'Somewhere' in previous discussion, furthermore, I 
don't like to add so much change information at the beginning of the patch.
But if you like, I will follow it. Sorry again.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 2/3] vt-d: synchronize for Device-TLB flush one by one

[Xu, Quan]

On May 18, 2016 5:29 PM, Jan Beulich  wrote:
> >>> On 18.05.16 at 10:53,  wrote:
> > On May 17, 2016 8:37 PM, Jan Beulich wrote:
> >> >>> On 22.04.16 at 12:54,  wrote:
> >> > -static void queue_invalidate_iotlb(struct iommu *iommu,
> >> > -u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
> >> > +static int __must_check queue_invalidate_iotlb_sync(struct iommu
> >> *iommu,
> >> > +u8 granu, u8 dr, u8 
> >> > dw,
> >> > +u16 did, u8 am, u8 
> >> > ih,
> >> > +u64 addr)
> >> >  {
> >> >  unsigned long flags;
> >> >  unsigned int index;
> >> > @@ -133,10 +141,12 @@ static void queue_invalidate_iotlb(struct
> >> > iommu
> >> *iommu,
> >> >  unmap_vtd_domain_page(qinval_entries);
> >> >  qinval_update_qtail(iommu, index);
> >> >  spin_unlock_irqrestore(>register_lock, flags);
> >> > +
> >> > +return invalidate_sync(iommu);
> >> >  }
> >>
> >> With this, ...
> >>
> >> > @@ -346,9 +353,13 @@ static int flush_iotlb_qi(
> >> >  if (cap_read_drain(iommu->cap))
> >> >  dr = 1;
> >> >  /* Need to conside the ih bit later */
> >> > -queue_invalidate_iotlb(iommu,
> >> > -   type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
> >> > -   dw, did, size_order, 0, addr);
> >> > +ret = queue_invalidate_iotlb_sync(iommu,
> >> > +  type >> 
> >> > DMA_TLB_FLUSH_GRANU_OFFSET,
> >> > +  dr, dw, did, size_order,
> >> > + 0, addr);
> >> > +
> >> > +if ( ret )
> >> > +return ret;
> >> > +
> >> >  if ( flush_dev_iotlb )
> >> >  ret = dev_invalidate_iotlb(iommu, did, addr, size_order, 
> >> > type);
> >> >  rc = invalidate_sync(iommu);
> >>
> >> ... why does this invalidate_sync() not go away?
> >>
> >
> > Oh, it is your suggestion -- leaving the existing logic as is would be
> > better - best effort invalidation even when an error has occurred.
> >
> > http://lists.xenproject.org/archives/html/xen-devel/2016-04/msg00523.h
> > tml
> 
> Look like this was a bad comment of mine (resulting from
> dev_invalidate_iotlb(), other than the other respective functions, not 
> getting a
> _sync tag added), and I would have appreciated if you had simply pointed out
> the redundancy.

I just issued an open for this point in v9 discussion. I felt a strange, but 
really didn't have obvious reasons at that time.
--
I'll  drop this invalidate_sync() in v11.

> Please remember that the review process is bi-directional,
> and hence doesn't mean you need to blindly do everything a reviewer asks for:
> Things you agree with should be changed in code. For things you don't agree
> with you should reply verbally, explaining why a requested change shouldn't
> be done.
> 
Thanks. I will try to follow it.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 2/3] vt-d: synchronize for Device-TLB flush one by one

[Xu, Quan]

On May 17, 2016 8:37 PM, Jan Beulich wrote:
> >>> On 22.04.16 at 12:54,  wrote:
> > --- a/xen/drivers/passthrough/vtd/qinval.c
> > +++ b/xen/drivers/passthrough/vtd/qinval.c
> > @@ -33,6 +33,8 @@ integer_param("vtd_qi_timeout", vtd_qi_timeout);
> >
> >  #define IOMMU_QI_TIMEOUT (vtd_qi_timeout * MILLISECS(1))
> >
> > +static int invalidate_sync(struct iommu *iommu);
> 
> __must_check?
> 

Yes, I will add it.


> > -static void queue_invalidate_iotlb(struct iommu *iommu,
> > -u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
> > +static int __must_check queue_invalidate_iotlb_sync(struct iommu
> *iommu,
> > +u8 granu, u8 dr, u8 dw,
> > +u16 did, u8 am, u8 ih,
> > +u64 addr)
> >  {
> >  unsigned long flags;
> >  unsigned int index;
> > @@ -133,10 +141,12 @@ static void queue_invalidate_iotlb(struct iommu
> *iommu,
> >  unmap_vtd_domain_page(qinval_entries);
> >  qinval_update_qtail(iommu, index);
> >  spin_unlock_irqrestore(>register_lock, flags);
> > +
> > +return invalidate_sync(iommu);
> >  }
> 
> With this, ...
> 
> > @@ -346,9 +353,13 @@ static int flush_iotlb_qi(
> >  if (cap_read_drain(iommu->cap))
> >  dr = 1;
> >  /* Need to conside the ih bit later */
> > -queue_invalidate_iotlb(iommu,
> > -   type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
> > -   dw, did, size_order, 0, addr);
> > +ret = queue_invalidate_iotlb_sync(iommu,
> > +  type >> 
> > DMA_TLB_FLUSH_GRANU_OFFSET,
> > +  dr, dw, did, size_order, 0, 
> > addr);
> > +
> > +if ( ret )
> > +return ret;
> > +
> >  if ( flush_dev_iotlb )
> >  ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
> >  rc = invalidate_sync(iommu);
> 
> ... why does this invalidate_sync() not go away?
> 

Oh, it is your suggestion -- leaving the existing logic as is would be better - 
best effort
invalidation even when an error has occurred.

http://lists.xenproject.org/archives/html/xen-devel/2016-04/msg00523.html

Quan




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 1/3] vt-d: add a timeout parameter for Queued Invalidation

[Xu, Quan]

On May 13, 2016 11:28 PM, Jan Beulich  wrote:
> >>> On 22.04.16 at 12:54,  wrote:
> > --- a/docs/misc/xen-command-line.markdown
> > +++ b/docs/misc/xen-command-line.markdown
> > @@ -1532,6 +1532,16 @@ Note that if **watchdog** option is also
> specified vpmu will be turned off.
> >  As the virtualisation is not 100% safe, don't use the vpmu flag on
> > production systems (see http://xenbits.xen.org/xsa/advisory-163.html)!
> >
> > +### vtd\_qi\_timeout (VT-d)
> > +> `= `
> > +
> > +> Default: `1`
> > +
> > +Specify the timeout of the VT-d Queued Invalidation in milliseconds.
> > +
> > +By default, the timeout is 1ms. When you see error 'Queue invalidate
> > +wait descriptor timed out', try increasing this value.
> 
> So when someone enables ATS, will the 1ms timeout apply to the dev iotlb
> invalidations too?

Yes,
The timeout is the same for IOTLB, Context, IEC and Device-TLB invalidation.

> If so, that's surely too short, and would ideally be adjusted
> automatically, but the need for a higher timeout in that case should in any
> event be mentioned here.

I can try to use 1ms for IOTLB, Context and  IEC invalidation. As mentioned, 1 
ms is enough for IOTLB, Context and  IEC invalidation.
What about 10 ms for Device-TLB (10 ms is just a higher timeout,  no specific 
meaning)?

> 
> Apart from that aspect this patch seems to be ready, but will clearly need a 
> VT-
> d maintainer's ack.
> 

Thanks for your review. I will also test this patch against the last commit ( I 
am still out of office and I will do it around this Wednesday).

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 13, 2016 5:09 PM, Jan Beulich  wrote:
> >>> On 13.05.16 at 10:04,  wrote:
> > On May 12, 2016 11:06 PM, Jan Beulich  wrote:
> >> >>> On 12.05.16 at 16:28,  wrote:
> >> > On May 10, 2016 2:54 PM, Jan Beulich  wrote:
> >> >> >>> On 10.05.16 at 05:41,  wrote:
> >> >> > On May 10, 2016 12:14 AM, Jan Beulich  wrote:
> >> >> >> >>> On 06.05.16 at 10:54,  wrote:
> What I think might at least come close to what Kevin and I would like to see 
> is
> something like
> 
>   if ( !d->is_shutting_down && printk_ratelimit() )
>   printk(...);
>   if ( !is_hardware_domain(d) )
>   domain_crash(d);
> 
> For Dom0 that'll still be more verbose than we'd really like, but it at least
> wouldn't outright flood the console.
> 

Thanks!!
It is really better to come close to Kevin's previous suggestion. I'll follow 
it for v5.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 12, 2016 11:06 PM, Jan Beulich  wrote:
> >>> On 12.05.16 at 16:28,  wrote:
> > On May 10, 2016 2:54 PM, Jan Beulich  wrote:
> >> >>> On 10.05.16 at 05:41,  wrote:
> >> > On May 10, 2016 12:14 AM, Jan Beulich  wrote:
> >> >> >>> On 06.05.16 at 10:54,  wrote:

Jan,

Try it again, I hope I have got it. If not, could you write some sample code 
for me as an exception? :)

--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -240,21 +240,63 @@ int iommu_map_page(struct domain *d, unsigned long gfn, 
unsigned long mfn,
unsigned int flags)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+static int printed = 0;
+int rc;

 if ( !iommu_enabled || !hd->platform_ops )
 return 0;

-return hd->platform_ops->map_page(d, gfn, mfn, flags);
+rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
+
+if ( unlikely(rc) )
+{
+if ( is_hardware_domain(d) )
+{
+if ( !printed )
+{
+printk(XENLOG_ERR
+   "d%d: IOMMU mapping gfn %#lx mfn %#lx failed %d.",
+   d->domain_id, gfn, mfn, rc);
+
+printed = 1;
+}
+}
+else
+domain_crash(d);
+}
+
+return rc;
 }

 int iommu_unmap_page(struct domain *d, unsigned long gfn)
 {
 const struct domain_iommu *hd = dom_iommu(d);
+static int printed = 0;
+int rc;

 if ( !iommu_enabled || !hd->platform_ops )
 return 0;

-return hd->platform_ops->unmap_page(d, gfn);
+rc = hd->platform_ops->unmap_page(d, gfn);
+
+if ( unlikely(rc) )
+{
+if ( is_hardware_domain(d) )
+{
+if ( !printed )
+{
+printk(XENLOG_ERR
+   "d%d: IOMMU unmapping gfn %#lx failed %d.",
+   d->domain_id, gfn, rc);
+
+printed = 1;
+}
+}
+else
+domain_crash(d);
+}
+
+return rc;
 }


Thanks again!!
Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 13, 2016 2:17 PM, Jan Beulich  wrote:
> >>> On 13.05.16 at 05:39,  wrote:
> > On May 10, 2016 5:25 PM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> >> > --- a/xen/arch/x86/acpi/power.c
> >> > +++ b/xen/arch/x86/acpi/power.c
> >> >  static int device_power_down(void)  {
> >> > -console_suspend();
> >> > +if ( console_suspend() )
> >> > +return TYPE_CONSOLE;
> >>
> >> This (together with the resume side) makes me guess that the use of
> >> TYPE_ as a prefix confused not just me, but also you:
> >
> > Yes,  this is really not a good prefix, and probably pretty bad to use
> > 'ERROR_'.
> > What about 'PRIOR_'?  then I also need to adjust  device_power_up() as ...
> 
> What about SAVED_?
>

much better, I'll  take it in v5. Thanks.

> >> > -time_suspend();
> >> > +if ( time_suspend() )
> >> > +return TYPE_TIME;
> >> >
> >> > -i8259A_suspend();
> >> > +if ( i8259A_suspend() )
> >> > +return TYPE_I8259A;
> >> >
> >> > +/* ioapic_suspend should never fail */
> >> >  ioapic_suspend();
> >>
> >> The comment is bogus: "should" means it can in theory. Yet the
> >> function having void return type means it just cannot fail.
> >>
> >
> > I'll use 'cannot', instead of 'should'.
> > Another question, I check the code again, and the rest of the
> > functions (console_suspend/ time_suspend/ i8259A_suspend /
> > ioapic_suspend / lapic_suspend ), in device_power_down(), always returned
> '0'.
> > Maybe I need to fix these functions  annotation from 'int' to 'void',
> > and then I can add a comment on the device_power_down().
> 
> Please don't. Generally the possibility of failure exists, and hence if 
> functions
> have already been written to account for that, we shouldn't strip that
> capability out.
> 

Got it.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 10, 2016 5:25 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/arch/x86/acpi/power.c
> > +++ b/xen/arch/x86/acpi/power.c
> >  static int device_power_down(void)
> >  {
> > -console_suspend();
> > +if ( console_suspend() )
> > +return TYPE_CONSOLE;
> 
> This (together with the resume side) makes me guess that the use of TYPE_ as
> a prefix confused not just me, but also you:

Yes,  this is really not a good prefix, and probably pretty bad to use 'ERROR_'.
What about 'PRIOR_'?  then I also need to adjust  device_power_up() as ...


> Here you want to tell the caller
> that everything _prior_ to console suspend (which happens to be nothing in
> this specific case) needs to be undone. Yet below you use TYPE_CONSOLE as
> an indication that
> console_resume() needs to be called.

... this,

+switch ( prior )
+{
   ...
+time_resume();
+case PRIOR_TIME:
+console_resume();
+case PRIOR_CONSOLE:
...
+}

> 
> > -time_suspend();
> > +if ( time_suspend() )
> > +return TYPE_TIME;
> >
> > -i8259A_suspend();
> > +if ( i8259A_suspend() )
> > +return TYPE_I8259A;
> >
> > +/* ioapic_suspend should never fail */
> >  ioapic_suspend();
> 
> The comment is bogus: "should" means it can in theory. Yet the function
> having void return type means it just cannot fail.
> 

I'll use 'cannot', instead of 'should'.
Another question, I check the code again, and the rest of the functions 
(console_suspend/ time_suspend/ i8259A_suspend / ioapic_suspend / lapic_suspend 
), in device_power_down(), always returned '0'.
Maybe I need to fix these functions  annotation from 'int' to 'void', and then 
I can add a comment on the device_power_down().  

More that, if the   ' iommu_suspend()'  is the only fail, could we re-consider 
the previous v2/v3 solution with 'goto'?

> > @@ -169,6 +204,7 @@ static int enter_state(u32 state)
> >  {
> >  printk(XENLOG_ERR "Some devices failed to power down.");
> >  system_state = SYS_STATE_resume;
> > +device_power_up(error);
> 
> Either error's and device_power_down()'s return type get changed to enum
> dev_power_type, or this needs a "error > 0" conditional.
> 
> > @@ -1267,7 +1273,9 @@ static void __hwdom_init
> intel_iommu_hwdom_init(struct domain *d)
> >  setup_hwdom_pci_devices(d, setup_hwdom_device);
> >  setup_hwdom_rmrr(d);
> >
> > -iommu_flush_all();
> > +if ( iommu_flush_all() )
> > +printk(XENLOG_WARNING VTDPREFIX
> > +   " intel_iommu_hwdom_init: IOMMU flush all failed.\n");
> 
> As said (and I think a number of times) before: At least when you already hold
> the error value in your hands, please also log it.

Agreed, and I will apply for other patches.

> Also personally I'm opposed to
> including function names in non-debug log messages, but I'll leave that
> decision to the VT-d maintainers here.
> 

Albeit Reluctantly, I will fix it.

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 10, 2016 2:54 PM, Jan Beulich  wrote:
> >>> On 10.05.16 at 05:41,  wrote:
> > On May 10, 2016 12:14 AM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> For DomU the solution seems quite obvious: Only log a message if the domain
> is not already marked crashed.

Jan, I am still confused about  this sentence and your another sentence ( _as 
said_ also avoid logging any message for already dying domains).

>  For Dom0 you'll need to get a little more
> creative (but by leveraging the fact that there's only one in the system, this
> can't be too difficult a problem to solve:
> e.g. "manually" rate limit these messages - see printk_ratelimit() et al).
> 

Reading this thread again and again, sorry, I am still inclined to:

+rc = hd->platform_ops->unmap_page(d, gfn);
+
+if ( unlikely(rc) )
+{
+if ( printk_ratelimit() )
+printk(XENLOG_ERR
+   "dom%d: IOMMU unmapping gfn %#lx failed %d.",
+   d->domain_id, gfn, rc);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;


Waiting for Kevin's opinion..


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 12, 2016 9:38 PM, Jan Beulich  wrote:
> >>> On 12.05.16 at 15:29,  wrote:
> > On May 12, 2016 4:53 PM, Jan Beulich  wrote:
> >> >>> On 12.05.16 at 09:50,  wrote:
> >> > On May 10, 2016 12:10 AM, Jan Beulich  wrote:
> >> >> >>> On 06.05.16 at 10:54,  wrote:
> >> >> > @@ -1391,13 +1399,19 @@ int domain_context_mapping_one(
> >> >> >  spin_unlock(>lock);
> >> >> >
> >> >> >  /* Context entry was previously non-present (with domid 0). */
> >> >> > -if ( iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) |
> devfn,
> >> >> > -DMA_CCMD_MASK_NOBIT, 1) )
> >> >> > -iommu_flush_write_buffer(iommu);
> >> >> > -else
> >> >> > +rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8)
> >> >> > + |
> >> devfn,
> >> >> > +DMA_CCMD_MASK_NOBIT, 1);
> >> >> > +
> >> >> > +if ( !rc )
> >> >> >  {
> >> >> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >> >> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >> >> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1,
> >> >> > + flush_dev_iotlb);
> >> >>
> >> >> Please take the opportunity and add the missing blank line
> >> >> (between
> >> >> declaration(s) and statement(s) in cases like this.
> >> >>
> >> >> > +}
> >> >> > +
> >> >> > +if ( rc > 0 )
> >> >>
> >> >> Can iommu_flush_context_device() return a positive value? If so,
> >> >> the logic is now likely wrong. If not (which is what I assume) I'd
> >> >> like to suggest adding a respective ASSERT() (even if only to
> >> >> document the fact). Or alternatively this
> >> >> if() could move into the immediately preceding one.
> >> >
> >> > Check it again. iommu_flush_context_device() can return a positive value.
> >> > [...]
> >> > Could you tell me why the logic is now likely wrong? I will fix it first.
> >>
> >> With
> >>
> >> rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> >> DMA_CCMD_MASK_NOBIT, 1);
> >>
> >> if ( !rc )
> >> {
> >> int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >> rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >> }
> >>
> >> if ( rc > 0 )
> >> {
> >> iommu_flush_write_buffer(iommu);
> >> rc = 0;
> >> }
> >>
> >> it seems pretty clear that you won't call iommu_flush_iotlb_dsi() if
> >> iommu_flush_context_device() returned 1, which doesn't look like what
> >> is wanted at the first glance. But I may be wrong, hence the "likely"
> >> in my
> > earlier
> >> reply.
> >>
> >
> > Oh, this was on purpose.
> >
> > If iommu_flush_context_device() returned 1,  the
> > iommu_flush_iotlb_dsi() returned 1 too.
> > As both flush_context_qi() and  flush_iotlb_qi () are the same at the
> > beginning of the  functions.
> 
> Such implications need to be commented on, so readers (like me) don't
> assume brokenness.
> 

ok, I will add a comment.

> > One concern is if iommu_flush_context_device() is failed, then we
> > won't call iommu_flush_iotlb_dsi(),  which is not best effort to flush.
> 
> Indeed.
> 

I'll fix it as well.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 12, 2016 4:53 PM, Jan Beulich  wrote:
> >>> On 12.05.16 at 09:50,  wrote:
> > On May 10, 2016 12:10 AM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> >> > -static void intel_iommu_iotlb_flush(struct domain *d, unsigned
> >> > long gfn, unsigned int page_count)
> >> > +static void iommu_flush_iotlb_page(struct domain *d, unsigned long
> gfn,
> >> > +   unsigned int page_count)
> >>
> >> The new name suggests just one page. Please use e.g.
> >> iommu_flush_iotlb_pages() instead.
> >>
> >
> > Make sense.
> >
> >> >  {
> >> > -__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
> >> > +iommu_flush_iotlb(d, gfn, 1, page_count);
> >> >  }
> >>
> >> But of course the question is whether having this wrapper is useful
> >> in the first place,
> >
> >
> > This wrapper assumes the 'dma_old_pte_present' is '1', but in another
> > caller intel_iommu_map_page(), i.e.
> >
> >
> >  intel_iommu_map_page()
> > {
> >...
> >  if ( !this_cpu(iommu_dont_flush_iotlb) )
> >   iommu_flush_iotlb(d, gfn, dma_pte_present(old), 1);
> >...
> > }
> >
> >
> > the 'dma_old_pte_present' is not sure.
> 
> I'm sorry, but you're looking at this backwards: I suggested to remove the
> wrapper, not to move any check into iommu_flush_iotlb().
> Removing the wrapper simply means to move the passing of the hard coded 1
> into the current callers of that wrapper.
> 

A little bit confused.
Check one thing, do the wrappers refer to iommu_flush_iotlb_page() and 
iommu_flush_iotlb_all() ?

If yes, we can't ignore another thing:

These two wrappers are also initialized for 2 .callbacks at the bottom of this 
file:


.iotlb_flush = iommu_flush_iotlb_pages,
.iotlb_flush_all = iommu_flush_iotlb_all,





> >> > @@ -1391,13 +1399,19 @@ int domain_context_mapping_one(
> >> >  spin_unlock(>lock);
> >> >
> >> >  /* Context entry was previously non-present (with domid 0). */
> >> > -if ( iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> >> > -DMA_CCMD_MASK_NOBIT, 1) )
> >> > -iommu_flush_write_buffer(iommu);
> >> > -else
> >> > +rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) |
> devfn,
> >> > +DMA_CCMD_MASK_NOBIT, 1);
> >> > +
> >> > +if ( !rc )
> >> >  {
> >> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> >> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> >>
> >> Please take the opportunity and add the missing blank line (between
> >> declaration(s) and statement(s) in cases like this.
> >>
> >> > +}
> >> > +
> >> > +if ( rc > 0 )
> >>
> >> Can iommu_flush_context_device() return a positive value? If so, the
> >> logic is now likely wrong. If not (which is what I assume) I'd like
> >> to suggest adding a respective ASSERT() (even if only to document the
> >> fact). Or alternatively this
> >> if() could move into the immediately preceding one.
> >
> > Check it again. iommu_flush_context_device() can return a positive value.
> > [...]
> > Could you tell me why the logic is now likely wrong? I will fix it first.
> 
> With
> 
> rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> DMA_CCMD_MASK_NOBIT, 1);
> 
> if ( !rc )
> {
> int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> }
> 
> if ( rc > 0 )
> {
> iommu_flush_write_buffer(iommu);
> rc = 0;
> }
> 
> it seems pretty clear that you won't call iommu_flush_iotlb_dsi() if
> iommu_flush_context_device() returned 1, which doesn't look like what is
> wanted at the first glance. But I may be wrong, hence the "likely" in my 
> earlier
> reply.
> 

Oh, this was on purpose.

If iommu_flush_context_device() returned 1,  the iommu_flush_iotlb_dsi() 
returned 1 too.
As both flush_context_qi() and  flush_iotlb_qi () are the same at the beginning 
of the  functions.

One concern is if iommu_flush_context_device() is failed, then we won't call 
iommu_flush_iotlb_dsi(),  which is not best effort to flush.


Quan










___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 10/10] vt-d: propagate error up to ME phantom function mapping and unmapping

[Xu, Quan]

On May 12, 2016 4:45 PM, Jan Beulich  wrote:
> >>> On 12.05.16 at 07:16,  wrote:
> > Taken together, there are 3 call trees to me_wifi_quirk():
> >
> >  1).
> > ...--me_wifi_quirk()--domain_context_mapping_one()--
> domain_context_map
> > ping()--se
> > tup_hwdom_device()
> >
> > There is no use in calling this function if an
> > earlier error occurred. The change can be more lightweight (the
> > detailed change is pending).
> >
> > 2).  me_wifi_quirk()--domain_context_unmap_one()--...
> >
> >As you mentioned,  while in the unmap case it
> > should probably stay as is, to fit the "best effort" theme.
> >
> >   Then I need to remove the  __must_check annotation
> > of me_wifi_quirk().
> 
> This does not follow from the above. You again should propagate the error in
> all cases (unless it would overwrite an earlier error - as you're doing in 
> various
> other places).
> 

Sorry, I know the item 2).  is tricky,  as I am confused about ' while in the 
unmap case it should probably stay as is, to fit the "best effort" theme '.


Actually, what I need to enhance the p10 are:

- drop ret
- replace the 

 if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+{
+ret = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+
+if ( !rc )
+rc = ret;
+}


TO_


-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);



btw, I found I am struggling in this v4 and I will spend more time fixing. 
thanks for your patience.
Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 10, 2016 12:10 AM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > -static void intel_iommu_iotlb_flush(struct domain *d, unsigned long
> > gfn, unsigned int page_count)
> > +static void iommu_flush_iotlb_page(struct domain *d, unsigned long gfn,
> > +   unsigned int page_count)
> 
> The new name suggests just one page. Please use e.g.
> iommu_flush_iotlb_pages() instead.
> 

Make sense. 

> >  {
> > -__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
> > +iommu_flush_iotlb(d, gfn, 1, page_count);
> >  }
> 
> But of course the question is whether having this wrapper is useful in the 
> first
> place,


This wrapper assumes the 'dma_old_pte_present' is '1', but in another caller 
intel_iommu_map_page(), i.e. 


 intel_iommu_map_page()
{
   ...
 if ( !this_cpu(iommu_dont_flush_iotlb) )
  iommu_flush_iotlb(d, gfn, dma_pte_present(old), 1);
   ...
}


the 'dma_old_pte_present' is not sure. 

in intel_iommu_map_page(),  if we can check the 'dma_pte_present(old)':
  -- 1, flush the pages.
  -- 0, don't flush the pages.

Then we can remove this wrapper.

If my description is not clear, I can send out the related change.

> the more that ...
> 
> > @@ -639,7 +646,7 @@ static void dma_pte_clear_one(struct domain
> *domain, u64 addr)
> >  iommu_flush_cache_entry(pte, sizeof(struct dma_pte));
> >
> >  if ( !this_cpu(iommu_dont_flush_iotlb) )
> > -__intel_iommu_iotlb_flush(domain, addr >> PAGE_SHIFT_4K, 1, 1);
> > +iommu_flush_iotlb(domain, addr >> PAGE_SHIFT_4K, 1, 1);
> 
> ... it's being open coded here. IOW if you want to retain the wrapper, please
> use it here.
>

Waiting for the above discussion, if we still need the wrapper, I will use it 
here.


 
> > @@ -1391,13 +1399,19 @@ int domain_context_mapping_one(
> >  spin_unlock(>lock);
> >
> >  /* Context entry was previously non-present (with domid 0). */
> > -if ( iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> > -DMA_CCMD_MASK_NOBIT, 1) )
> > -iommu_flush_write_buffer(iommu);
> > -else
> > +rc = iommu_flush_context_device(iommu, 0, (((u16)bus) << 8) | devfn,
> > +DMA_CCMD_MASK_NOBIT, 1);
> > +
> > +if ( !rc )
> >  {
> >  int flush_dev_iotlb = find_ats_dev_drhd(iommu) ? 1 : 0;
> > -iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> > +rc = iommu_flush_iotlb_dsi(iommu, 0, 1, flush_dev_iotlb);
> 
> Please take the opportunity and add the missing blank line (between
> declaration(s) and statement(s) in cases like this.
> 
> > +}
> > +
> > +if ( rc > 0 )
> 
> Can iommu_flush_context_device() return a positive value? If so, the logic is
> now likely wrong. If not (which is what I assume) I'd like to suggest adding a
> respective ASSERT() (even if only to document the fact). Or alternatively this
> if() could move into the immediately preceding one.
> 


Check it again. iommu_flush_context_device() can return a positive value.

If VT-d QI is enabled, the call tree up to iommu_flush_context_device():
 -- flush_context_qi()  -- iommu_flush_context_device() 


i.e. 

In flush_context_qi()
{
...
if ( flush_non_present_entry )
{
if ( !cap_caching_mode(iommu->cap) )
return 1;
else
did = 0;
}
...
}


and the ' flush_non_present_entry '  is really '1' for above code.



Could you tell me why the logic is now likely wrong? I will fix it first.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 10/10] vt-d: propagate error up to ME phantom function mapping and unmapping

[Xu, Quan]

On May 11, 2016 5:07 PM, Jan Beulich  wrote:
> >>> On 11.05.16 at 10:35,  wrote:
> > On May 10, 2016 5:29 PM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> >> > @@ -1430,7 +1430,12 @@ int domain_context_mapping_one(
> >> >  unmap_vtd_domain_page(context_entries);
> >> >
> >> >  if ( !seg )
> >> > -me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
> >> > +{
> >> > +ret = me_wifi_quirk(domain, bus, devfn,
> >> > + MAP_ME_PHANTOM_FUNC);
> >> > +
> >> > +if ( !rc )
> >> > +rc = ret;
> >> > +}
> >>
> >> Is there any use in calling this function if an earlier error occurred?
> >> If not,
> >
> > It is  no use.
> 
> With this I don't understand ...
> 
> > We may need to consider this call tree:
> >$
> > me_wifi_quirk()--domain_context_mapping_one()--
> domain_context_mapping(
> > )--reass
> > ign_device_ownership()--...
> >
> > Then, what about dropping this patch? Leave it as is,  or remove '
> > __must_check' annotation and propagate error up to the above call tree
> > only?
> 
> ... this. If calling the function is pointless if an earlier error occurred, 
> why don't
> you just check rc to be zero alongside the !seg check?
> 

---
Good idea.

---

Taken together, there are 3 call trees to me_wifi_quirk():

 1). 
...--me_wifi_quirk()--domain_context_mapping_one()--domain_context_mapping()--setup_hwdom_device()

There is no use in calling this function if an earlier 
error occurred. The change can be more lightweight (the detailed change is 
pending).

2).  me_wifi_quirk()--domain_context_unmap_one()--...

   As you mentioned,  while in the unmap case it should 
probably stay as is, to fit the "best effort" theme. 

  Then I need to remove the  __must_check annotation  of 
me_wifi_quirk().

3). 
me_wifi_quirk()--domain_context_mapping_one()--domain_context_mapping()--reassign_device_ownership()
This is not an earlier error, so we need propagate the 
error up to the call tree (the detailed change is pending).



The below is based on previous v4 p1...p9:
---

diff --git a/xen/drivers/passthrough/vtd/extern.h 
b/xen/drivers/passthrough/vtd/extern.h
index cbe0286..d4d37c3 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -91,7 +91,7 @@ int is_igd_vt_enabled_quirk(void);
 void platform_quirks_init(void);
 void vtd_ops_preamble_quirk(struct iommu* iommu);
 void vtd_ops_postamble_quirk(struct iommu* iommu);
-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
+int me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map);
 void pci_vtd_quirk(const struct pci_dev *);
 bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
diff --git a/xen/drivers/passthrough/vtd/iommu.c 
b/xen/drivers/passthrough/vtd/iommu.c
index 29cf870..0ac7894 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1429,8 +1429,8 @@ int domain_context_mapping_one(

 unmap_vtd_domain_page(context_entries);

-if ( !seg )
-me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
+if ( !seg && !rc )
+rc = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);

 return rc;
 }
diff --git a/xen/drivers/passthrough/vtd/quirks.c 
b/xen/drivers/passthrough/vtd/quirks.c
index 473d1fc..3606b52 100644
--- a/xen/drivers/passthrough/vtd/quirks.c
+++ b/xen/drivers/passthrough/vtd/quirks.c
@@ -331,10 +331,12 @@ void __init platform_quirks_init(void)
  * assigning Intel integrated wifi device to a guest.
  */

-static void map_me_phantom_function(struct domain *domain, u32 dev, int map)
+static int __must_check map_me_phantom_function(struct domain *domain,
+u32 dev, int map)
 {
 struct acpi_drhd_unit *drhd;
 struct pci_dev *pdev;
+int rc;

 /* find ME VT-d engine base on a real ME device */
 pdev = pci_get_pdev(0, 0, PCI_DEVFN(dev, 0));
@@ -342,23 +344,27 @@ static void map_me_phantom_function(struct domain 
*domain, u32 dev, int map)

 /* map or unmap ME phantom function */
 if ( map )
-domain_context_mapping_one(domain, drhd->iommu, 0,
-   PCI_DEVFN(dev, 7), NULL);
+rc = domain_context_mapping_one(domain, drhd->iommu, 0,
+PCI_DEVFN(dev, 7), NULL);
 else
-domain_context_unmap_one(domain, drhd->iommu, 0,
- PCI_DEVFN(dev, 7));
+rc = domain_context_unmap_one(domain, drhd->iommu, 0,
+  PCI_DEVFN(dev, 7));
+
+return rc;
 }

-void me_wifi_quirk(struct domain *domain, u8 bus, u8 devfn, int map)
+int me_wifi_quirk(struct domain *domain,
+  u8 bus, u8 devfn, int map)
 {
 u32 id;
+int rc = 0;

 id = 

Re: [Xen-devel] [PATCH v4 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 11, 2016 4:46 PM, George Dunlap <george.dun...@eu.citrix.com> wrote:
> On Wed, May 11, 2016 at 3:26 AM, Xu, Quan <quan...@intel.com> wrote:
> > Agreed. Thanks for your careful checking.
> >
> > Check it again --
> > 1. then I am no need to check 'rc' as below:
> >
> >   if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >need_modify_vtd_table )
> >  {
> > +if ( !rc )
> > +rc = ret;
> > ...
> >
> > +if ( !rc && unlikely(ret) )
> > +rc = ret;
> >  }
> 
> Actually, in the case of iommu_map_page(), you can just use rc directly and
> not bother using ret at all.  (In the case of iommu_unmap_page(), you still
> need to use ret and check that rc != 0 to make sure you get the first error.)
> 
> Something like applying the attached patch (not built or nitpicked for style,
> just for clarity).
> 

Got it, thanks.

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 10/10] vt-d: propagate error up to ME phantom function mapping and unmapping

[Xu, Quan]

On May 10, 2016 5:29 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > @@ -1430,7 +1430,12 @@ int domain_context_mapping_one(
> >  unmap_vtd_domain_page(context_entries);
> >
> >  if ( !seg )
> > -me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
> > +{
> > +ret = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
> > +
> > +if ( !rc )
> > +rc = ret;
> > +}
> 
> Is there any use in calling this function if an earlier error occurred?
> If not,

It is  no use.


We may need to consider this call tree:
   $ 
me_wifi_quirk()--domain_context_mapping_one()--domain_context_mapping()--reassign_device_ownership()--...

Then, what about dropping this patch? Leave it as is,
 or remove ' __must_check' annotation and propagate error up to the above call 
tree only?


>  the change can be more lightweight (while in the unmap case it should
> probably stay as is, to fit the "best effort" theme).
> 

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 08/10] vt-d/ept: propagate IOMMU Device-TLB flush error up to EPT update.

[Xu, Quan]

On Tuesday, May 10, 2016 5:10 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/arch/x86/mm/p2m-ept.c
> > +++ b/xen/arch/x86/mm/p2m-ept.c
> > @@ -832,7 +832,8 @@ out:
> >   need_modify_vtd_table )
> >  {
> >  if ( iommu_hap_pt_share )
> > -iommu_pte_flush(d, gfn, _entry->epte, order, 
> > vtd_pte_present);
> > +ret = iommu_pte_flush(d, gfn, _entry->epte,
> > +  order, vtd_pte_present);
> >  else
> >  {
> >  if ( iommu_flags )
> 
> Looking at this in conjunction with patch 3, I can't see where "ret"
> would get consumed.
> 

Sorry, this 'ret' should be 'rc' here, as I removed  the pending

"
+if ( !rc )
+rc = ret;
+
"

from v4.


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 07/10] IOMMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (leaf ones).

[Xu, Quan]

On May 11, 2016 3:17 PM, Jan Beulich  wrote:
> >>> On 11.05.16 at 09:12,  wrote:
> > On May 11, 2016 3:06 PM, Jan Beulich  wrote:
> >> >>> On 11.05.16 at 08:47,  wrote:
> >> > On May 10, 2016 5:07 PM, Jan Beulich  wrote:
> >> >> >>> On 06.05.16 at 10:54,  wrote:
> >> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >> >> > @@ -604,15 +604,15 @@ static int iommu_flush_iotlb(struct domain
> >> >> > *d,
> >> >> unsigned long gfn,
> >> >> >  return rc;
> >> >> >  }
> >> >> >
> >> >> > -static void iommu_flush_iotlb_page(struct domain *d, unsigned
> >> >> > long
> >> gfn,
> >> >> > -   unsigned int page_count)
> >> >> > +static int iommu_flush_iotlb_page(struct domain *d, unsigned long
> gfn,
> >> >> > +  unsigned int page_count)
> >> >> >  {
> >> >> > -iommu_flush_iotlb(d, gfn, 1, page_count);
> >> >> > +return iommu_flush_iotlb(d, gfn, 1, page_count);
> >> >> >  }
> >> >> >
> >> >> > -static void iommu_flush_iotlb_all(struct domain *d)
> >> >> > +static int iommu_flush_iotlb_all(struct domain *d)
> >> >> >  {
> >> >> > -iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >> >> > +return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >> >> >  }
> >> >>
> >> >> As already indicated in a reply to an earlier patch, despite what
> >> >> was said on the earlier version I think we should have
> >> >> __must_check here
> >> >
> >> > If the static one is initialized for .callback, is it really
> >> > necessary to add __must_check here?
> >> > I check it with compiler, and it is ok when I didn't add __must_check 
> >> > here.
> >>
> >> Without you telling us what exactly you checked, I can't respond to this.
> >> Extending from the reply just sent to patch 3(?) and for the
> >> avoidance of doubt, you now obviously also need to
> >> __must_check-annotate the function pointer (to match the desire of
> >> wanting to never lose such an annotation on the way back up the call tree).
> >>
> >
> > I checked -- without __must_check for iommu_flush_iotlb_page() /
> > iommu_flush_iotlb_all().
> 
> But _what_ did you check? I.e. the question isn't which functions you did your
> check with, but what behavioral checking you did.
> 
without __must_check for iommu_flush_iotlb_page() /iommu_flush_iotlb_all(), I 
can run 'make xen' successfully. Sorry.

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 07/10] IOMMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (leaf ones).

[Xu, Quan]

On May 11, 2016 3:06 PM, Jan Beulich  wrote:
> >>> On 11.05.16 at 08:47,  wrote:
> > On May 10, 2016 5:07 PM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >> > @@ -604,15 +604,15 @@ static int iommu_flush_iotlb(struct domain
> >> > *d,
> >> unsigned long gfn,
> >> >  return rc;
> >> >  }
> >> >
> >> > -static void iommu_flush_iotlb_page(struct domain *d, unsigned long
> gfn,
> >> > -   unsigned int page_count)
> >> > +static int iommu_flush_iotlb_page(struct domain *d, unsigned long gfn,
> >> > +  unsigned int page_count)
> >> >  {
> >> > -iommu_flush_iotlb(d, gfn, 1, page_count);
> >> > +return iommu_flush_iotlb(d, gfn, 1, page_count);
> >> >  }
> >> >
> >> > -static void iommu_flush_iotlb_all(struct domain *d)
> >> > +static int iommu_flush_iotlb_all(struct domain *d)
> >> >  {
> >> > -iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >> > +return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >> >  }
> >>
> >> As already indicated in a reply to an earlier patch, despite what was
> >> said on the earlier version I think we should have __must_check here
> >
> > If the static one is initialized for .callback, is it really necessary
> > to add __must_check here?
> > I check it with compiler, and it is ok when I didn't add __must_check here.
> 
> Without you telling us what exactly you checked, I can't respond to this.
> Extending from the reply just sent to patch 3(?) and for the avoidance of
> doubt, you now obviously also need to __must_check-annotate the function
> pointer (to match the desire of wanting to never lose such an annotation on
> the way back up the call tree).
> 

I checked -- without __must_check for iommu_flush_iotlb_page() / 
iommu_flush_iotlb_all().


> > If yes, I'll add __must_check for ARM one as well.
> 
> Of course.
> 
Got it.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 07/10] IOMMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (leaf ones).

[Xu, Quan]

On May 10, 2016 5:07 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -604,15 +604,15 @@ static int iommu_flush_iotlb(struct domain *d,
> unsigned long gfn,
> >  return rc;
> >  }
> >
> > -static void iommu_flush_iotlb_page(struct domain *d, unsigned long gfn,
> > -   unsigned int page_count)
> > +static int iommu_flush_iotlb_page(struct domain *d, unsigned long gfn,
> > +  unsigned int page_count)
> >  {
> > -iommu_flush_iotlb(d, gfn, 1, page_count);
> > +return iommu_flush_iotlb(d, gfn, 1, page_count);
> >  }
> >
> > -static void iommu_flush_iotlb_all(struct domain *d)
> > +static int iommu_flush_iotlb_all(struct domain *d)
> >  {
> > -iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> > +return iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >  }
> 
> As already indicated in a reply to an earlier patch, despite what was said on 
> the
> earlier version I think we should have __must_check here

If the static one is initialized for .callback, is it really necessary to add 
__must_check here? 
I check it with compiler, and it is ok when I didn't add __must_check here.

If yes, I'll add __must_check for ARM one as well.

> and ...
> 
> > --- a/xen/include/xen/iommu.h
> > +++ b/xen/include/xen/iommu.h
> > @@ -179,8 +179,8 @@ struct iommu_ops {
> >  void (*resume)(void);
> >  void (*share_p2m)(struct domain *d);
> >  void (*crash_shutdown)(void);
> > -void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int
> page_count);
> > -void (*iotlb_flush_all)(struct domain *d);
> > +int (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int
> page_count);
> > +int (*iotlb_flush_all)(struct domain *d);
> 
> ... here.

Yes,  it is necessary here.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 06/10] IOMMU/MMU: propagate IOMMU Device-TLB flush error up to iommu_iotlb_flush{, _all} (top level ones).

[Xu, Quan]

On May 10, 2016 5:04 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/common/memory.c
> > +++ b/xen/common/memory.c
> > @@ -677,9 +677,19 @@ static int xenmem_add_to_physmap(struct
> domain
> > *d,  #ifdef CONFIG_HAS_PASSTHROUGH
> >  if ( need_iommu(d) )
> >  {
> > +int ret;
> > +
> >  this_cpu(iommu_dont_flush_iotlb) = 0;
> > -iommu_iotlb_flush(d, xatp->idx - done, done);
> > -iommu_iotlb_flush(d, xatp->gpfn - done, done);
> > +
> > +ret = iommu_iotlb_flush(d, xatp->idx - done, done);
> > +
> > +if ( rc >= 0 && unlikely(ret) )
> > +rc = ret;
> > +
> > +ret = iommu_iotlb_flush(d, xatp->gpfn - done, done);
> > +
> > +if ( rc >= 0 && unlikely(ret) )
> > +rc = ret;
> 
> In both cases I think it would be preferable to switch the two sides of the 
> &&.
> 

Make sense. Thanks for your careful check.


> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -312,24 +312,29 @@ static void iommu_free_pagetables(unsigned
> long unused)
> >  cpumask_cycle(smp_processor_id(),
> > _online_map));  }
> >
> > -void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned
> > int page_count)
> > +int __must_check iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> > +   unsigned int page_count)
> 
> The annotation generally belongs on declarations; the only exception are 
> static
> functions which don't have any (i.e. they get defined before first getting
> used).
>

You are right. I will drop these __must_check annotation here.
 
> > --- a/xen/include/xen/iommu.h
> > +++ b/xen/include/xen/iommu.h
> > @@ -200,8 +200,9 @@ int iommu_do_pci_domctl(struct xen_domctl *,
> > struct domain *d,  int iommu_do_domctl(struct xen_domctl *, struct domain
> *d,
> >  XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
> >
> > -void iommu_iotlb_flush(struct domain *d, unsigned long gfn, unsigned
> > int page_count); -void iommu_iotlb_flush_all(struct domain *d);
> > +int __must_check iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> > +   unsigned int page_count); int
> > +__must_check iommu_iotlb_flush_all(struct domain *d);
> 
> I.e. these changes suffice, repeating the annotations on the function
> definitions is redundant with the ones put here.
> 

Agreed.

> With respective adjustments done
> Reviewed-by: Jan Beulich 

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 04/10] vt-d: propagate IOMMU Device-TLB flush error up to IOMMU unmapping.

[Xu, Quan]

On May 10, 2016 4:51 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 06.05.16 at 10:54, <quan...@intel.com> wrote:
> > Propagate the IOMMU Device-TLB flush error up to IOMMU unmapping.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> > Acked-by: Kevin Tian <kevin.t...@intel.com>
> 
> Reviewed-by: Jan Beulich <jbeul...@suse.com>
> 
> but note ...
> 
> > @@ -1766,9 +1769,7 @@ static int intel_iommu_unmap_page(struct
> domain *d, unsigned long gfn)
> >  if ( iommu_passthrough && is_hardware_domain(d) )
> >  return 0;
> >
> > -dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
> > -
> > -return 0;
> > +return dma_pte_clear_one(d, (paddr_t)gfn << PAGE_SHIFT_4K);
> >  }
> 
> ... how you lose the __must_check here, since
> intel_iommu_unmap_page() isn't __must_check (which we said you may skip
> as long as the common code wrapper has it, but in the context here I'm no
> longer convinced skipping this at any layer is a good idea, as that makes
> validation of the call trees more difficult).
> (This is just a remark regarding the comment on the earlier patch, i.e. not
> something needing any further change here.)
> 


I'll be bold to add __must_check..
Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 10, 2016 4:44 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/arch/x86/mm/p2m.c
> > +++ b/xen/arch/x86/mm/p2m.c
> > @@ -638,13 +638,20 @@ p2m_remove_page(struct p2m_domain *p2m,
> unsigned long gfn, unsigned long mfn,
> >  mfn_t mfn_return;
> >  p2m_type_t t;
> >  p2m_access_t a;
> > +int rc = 0, ret;
> >
> >  if ( !paging_mode_translate(p2m->domain) )
> >  {
> >  if ( need_iommu(p2m->domain) )
> >  for ( i = 0; i < (1 << page_order); i++ )
> > -iommu_unmap_page(p2m->domain, mfn + i);
> > -return 0;
> > +{
> > +   ret = iommu_unmap_page(p2m->domain, mfn + i);
> > +
> > +   if ( !rc )
> > +   rc = ret;
> > +}
> > +
> > +return rc;
> >  }
> 
> In code like this, btw., restricting the scope of "ret" to the innermost block
> would help future readers see immediately that the value of "ret" is of no
> further interest outside of that block.
> 
> Having reached the end of the patch, I'm missing the __must_check additions
> that you said you would do in this new iteration. Is there any reason for 
> their
> absence? Did I overlook something?
> 

Sorry, I did overlook something.
Checked the v2/v3 replies again, I still can't find it.
I only add the __must_check annotation for these functions you point out.
Do I need to add the __must_check annotation for these  functions (but not void 
function) in this patch?

Quan 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 10, 2016 10:45 PM, George Dunlap  wrote:
> On Tue, May 10, 2016 at 9:44 AM, Jan Beulich  wrote:
>  On 06.05.16 at 10:54,  wrote:
> >> When IOMMU mapping is failed, we issue a best effort rollback,
> >> stopping IOMMU mapping, unmapping the previous IOMMU maps and
> then
> >> reporting the error up to the call trees. When rollback is not
> >> feasible (in early initialization phase or trade-off of complexity)
> >> for the hardware domain, we do things on a best effort basis, only
> throwing out an error message.
> >>
> >> IOMMU unmapping should perhaps continue despite an error, in an
> >> attempt to do best effort cleanup.
> >>
> >> --- a/xen/arch/x86/mm/p2m.c
> >> +++ b/xen/arch/x86/mm/p2m.c
> >> @@ -638,13 +638,20 @@ p2m_remove_page(struct p2m_domain *p2m,
> unsigned long gfn, unsigned long mfn,
> >>  mfn_t mfn_return;
> >>  p2m_type_t t;
> >>  p2m_access_t a;
> >> +int rc = 0, ret;
> >>
> >>  if ( !paging_mode_translate(p2m->domain) )
> >>  {
> >>  if ( need_iommu(p2m->domain) )
> >>  for ( i = 0; i < (1 << page_order); i++ )
> >> -iommu_unmap_page(p2m->domain, mfn + i);
> >> -return 0;
> >> +{
> >> +   ret = iommu_unmap_page(p2m->domain, mfn + i);
> >> +
> >> +   if ( !rc )
> >> +   rc = ret;
> >> +}
> >> +
> >> +return rc;
> >>  }
> >
> > In code like this, btw., restricting the scope of "ret" to the
> > innermost block would help future readers see immediately that the
> > value of "ret" is of no further interest outside of that block.
> 
> I wouldn't ask for re-send just for this, but...
> 
> > Having reached the end of the patch, I'm missing the __must_check
> > additions that you said you would do in this new iteration. Is there
> > any reason for their absence? Did I overlook something?
> 
> If it's going to be re-sent anyway, moving the ret declaration inside the loop
> might as well be done.
> 
> Other than that, it looks good to me, thanks.
> 


I'll fix and re-send it in next v5.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 10, 2016 11:00 PM, George Dunlap <george.dun...@eu.citrix.com> wrote:
> On Tue, May 10, 2016 at 3:45 PM, George Dunlap
> <george.dun...@eu.citrix.com> wrote:
> > On Tue, May 10, 2016 at 9:44 AM, Jan Beulich <jbeul...@suse.com> wrote:
> >>>>> On 06.05.16 at 10:54, <quan...@intel.com> wrote:
> >>> When IOMMU mapping is failed, we issue a best effort rollback,
> >>> stopping IOMMU mapping, unmapping the previous IOMMU maps and
> then
> >>> reporting the error up to the call trees. When rollback is not
> >>> feasible (in early initialization phase or trade-off of complexity)
> >>> for the hardware domain, we do things on a best effort basis, only
> throwing out an error message.
> >>>
> >>> IOMMU unmapping should perhaps continue despite an error, in an
> >>> attempt to do best effort cleanup.
> >>>
> >>> Signed-off-by: Quan Xu <quan...@intel.com>
> >>>
> >>> CC: Keir Fraser <k...@xen.org>
> >>> CC: Jan Beulich <jbeul...@suse.com>
> >>> CC: Andrew Cooper <andrew.coop...@citrix.com>
> >>> CC: Jun Nakajima <jun.nakaj...@intel.com>
> >>> CC: Kevin Tian <kevin.t...@intel.com>
> >>> CC: George Dunlap <george.dun...@eu.citrix.com>
> >>> ---
> >>
> >> Somewhere here I continue to miss a summary on what has changed
> >> compared to the previous version. For review especially of larger
> >> patches (where preferably one wouldn't want to re-review the entire
> >> thing) this is more than just a nice-to-have.
> >>
> >>> @@ -812,17 +813,22 @@ ept_set_entry(struct p2m_domain *p2m,
> unsigned long gfn, mfn_t mfn,
> >>>  rc = atomic_write_ept_entry(ept_entry, new_entry, target);
> >>>  if ( unlikely(rc) )
> >>>  old_entry.epte = 0;
> >>> -else if ( p2mt != p2m_invalid &&
> >>> -  (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
> >>> -/* Track the highest gfn for which we have ever had a valid 
> >>> mapping
> */
> >>> -p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
> >>> +else
> >>> +{
> >>> +entry_written = 1;
> >>> +
> >>> +if ( p2mt != p2m_invalid &&
> >>> + (gfn + (1UL << order) - 1 > p2m->max_mapped_pfn) )
> >>> +/* Track the highest gfn for which we have ever had a valid
> mapping */
> >>> +p2m->max_mapped_pfn = gfn + (1UL << order) - 1;
> >>> +}
> >>>
> >>>  out:
> >>>  if ( needs_sync )
> >>>  ept_sync_domain(p2m);
> >>>
> >>>  /* For host p2m, may need to change VT-d page table.*/
> >>> -if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >>> +if ( entry_written && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >>>   need_modify_vtd_table )
> >>>  {
> >>
> >> I'd prefer this conditional to remain untouched, but I'll leave the
> >> decision to the maintainers of the file.
> >
> > Any particular reason you think it would be better untouched?
> >
> > I asked for it to be changed to "entry_written", because it seemed to
> > me that's what was actually wanted (i.e., you're checking whether rc
> > == 0 to determine whether the entry was written or not).  At the
> > moment the checks will be identical, but if someone changed something
> > later, rc might be non-zero even though the entry had been written, in
> > which case (I think) you'd want the iommu update to happen.
> >
> > It's not that big a deal to me, but I do prefer it this way (unless
> > I've misunderstood something).
> 
> See the discussion on patch 8 regarding why I now think Jan is probably right.
> 

Agreed. Thanks for your careful checking.

Check it again --
1. then I am no need to check 'rc' as below:

  if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
   need_modify_vtd_table )
 {
+if ( !rc ) 
+rc = ret;
...

+if ( !rc && unlikely(ret) )
+rc = ret;
 }


2.  leave the below as is:

-if ( rc == 0 && p2m_is_hostp2m(p2m) )
+if ( entry_written && p2m_is_hostp2m(p2m) )

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 10, 2016 4:03 PM, Jan Beulich  wrote:
> >>> On 10.05.16 at 09:53,  wrote:
> > +rc = hd->platform_ops->unmap_page(d, gfn);
> > +
> > +if ( unlikely(rc) )
> > +{
> > +if ( printk_ratelimit() )
> > +printk(XENLOG_ERR
> > +   "iommu_unmap_page: IOMMU unmapping gfn %#lx failed for
> dom%d.",
> > +   gfn, d->domain_id);
> > +
> > +if ( !is_hardware_domain(d) )
> > +domain_crash(d);
> > +}
> > +
> > +return rc;
> 
> But please - as said - also avoid logging any message for already dying
> domains.
> 


Kept Kevin's opinion for later, I hope I have got your point as below:
...
+rc = hd->platform_ops->unmap_page(d, gfn);
+
+if ( unlikely(rc) )
+{
+if ( is_hardware_domain(d) )
+if ( printk_ratelimit() )
+printk(XENLOG_ERR
+   "iommu_unmap_page: IOMMU unmapping gfn %#lx failed for 
dom%d.",
+   gfn, d->domain_id);
+else
+domain_crash(d);
+}
+
+return rc;
...

Thanks for your patience.
Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 10, 2016 2:54 PM, Jan Beulich  wrote:
> >>> On 10.05.16 at 05:41,  wrote:
> > On May 10, 2016 12:14 AM, Jan Beulich  wrote:
> >> >>> On 06.05.16 at 10:54,  wrote:
> >> > --- a/xen/drivers/passthrough/iommu.c
> >> > +++ b/xen/drivers/passthrough/iommu.c
> >> > @@ -240,21 +240,47 @@ int iommu_map_page(struct domain *d,
> >> unsigned long gfn, unsigned long mfn,
> >> > unsigned int flags)  {
> >> >  const struct domain_iommu *hd = dom_iommu(d);
> >> > +int rc;
> >> >
> >> >  if ( !iommu_enabled || !hd->platform_ops )
> >> >  return 0;
> >> >
> >> > -return hd->platform_ops->map_page(d, gfn, mfn, flags);
> >> > +rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
> >> > +
> >> > +if ( unlikely(rc) )
> >> > +{
> >> > +printk(XENLOG_ERR
> >> > +   "iommu_map_page: IOMMU mapping gfn %#lx mfn %#lx
> >> > + failed for
> >> dom%d.",
> >> > +   gfn, mfn, d->domain_id);
> >> > +
> >> > +if ( !is_hardware_domain(d) )
> >> > +domain_crash(d);
> >> > +}
> >>
> >> This still may spam the console in at least the case of Dom0.
> >
> > I am afraid we may need a minor trade-off. What about:
> >
> >dprintk(XENLOG_ERR, "...");
> >
> > to print out in debug mode.
> 
> And be silent in non-debug mode? That's not what we want.
> 

Without your below suggestion, this is really my best solution.

> >>  For DomU I'd
> >> really expect you to state in the commit message why no spamming can
> >> occur (of course assuming it really can't, which I'm not convinced of).
> >>
> >
> > In this v4, I think we will still spam the console in extreme cases :(:(..
> >
> > For mapping:
> > +ret = iommu_map_page();
> > +if ( unlikely(ret) )
> > +{
> > +while ( i-- )
> > +iommu_unmap_page();
> > +}
> >
> > We'll  stop map against any error and unmapping the previous mappings.
> > The extreme case is error for unmapping the previous mappings.
> >
> > Again -- I think dprintk is a better solution. Any suggestion?
> 
> For DomU the solution seems quite obvious: Only log a message if the domain
> is not already marked crashed. For Dom0 you'll need to get a little more
> creative (but by leveraging the fact that there's only one in the system, this
> can't be too difficult a problem to solve:
> e.g. "manually" rate limit these messages - see printk_ratelimit() et al).

Amazing!!
As the comment said, printk_ratelimit() is lifted from Linux. referred to the 
Linux, __iiuc__ , I will fix this issue as below (a variant):

...
+rc = hd->platform_ops->unmap_page(d, gfn);
+
+if ( unlikely(rc) )
+{
+if ( printk_ratelimit() )
+printk(XENLOG_ERR
+   "iommu_unmap_page: IOMMU unmapping gfn %#lx failed for 
dom%d.",
+   gfn, d->domain_id);
+
+if ( !is_hardware_domain(d) )
+domain_crash(d);
+}
+
+return rc;
...

Thanks!!



Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 10, 2016 12:14 AM, Jan Beulich  wrote:
> >>> On 06.05.16 at 10:54,  wrote:
> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -240,21 +240,47 @@ int iommu_map_page(struct domain *d,
> unsigned long gfn, unsigned long mfn,
> > unsigned int flags)  {
> >  const struct domain_iommu *hd = dom_iommu(d);
> > +int rc;
> >
> >  if ( !iommu_enabled || !hd->platform_ops )
> >  return 0;
> >
> > -return hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +
> > +if ( unlikely(rc) )
> > +{
> > +printk(XENLOG_ERR
> > +   "iommu_map_page: IOMMU mapping gfn %#lx mfn %#lx failed for
> dom%d.",
> > +   gfn, mfn, d->domain_id);
> > +
> > +if ( !is_hardware_domain(d) )
> > +domain_crash(d);
> > +}
> 
> This still may spam the console in at least the case of Dom0.

I am afraid we may need a minor trade-off. What about:

   dprintk(XENLOG_ERR, "...");

to print out in debug mode.

>  For DomU I'd
> really expect you to state in the commit message why no spamming can occur
> (of course assuming it really can't, which I'm not convinced of).
>

In this v4, I think we will still spam the console in extreme cases :(:(..

For mapping:
+ret = iommu_map_page();
+if ( unlikely(ret) )
+{
+while ( i-- )
+iommu_unmap_page();
+}

We'll  stop map against any error and unmapping the previous mappings.  The 
extreme case is error for unmapping the previous mappings.

Again -- I think dprintk is a better solution. Any suggestion?

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] IOMMU: don't BUG() on exotic hardware

[Xu, Quan]

On May 09, 2016 4:24 PM, Jan Beulich  wrote:
> >>> On 09.05.16 at 09:55,  wrote:
> > On May 06, 2016 10:24 PM, Jan Beulich  wrote:
> >> On x86, iommu_get_ops() BUG()s when running on non-Intel, non-AMD
> >> hardware. While, with our current code, that's a correct prerequisite
> >> assumption for IOMMU presence, this is wrong on systems without
> IOMMU.
> >> Hence iommu_enabled (and alike) checks should be done prior to
> >> calling that function, not after.
> >>
> >> Also move iommu_suspend() next to iommu_resume() - it escapes me why
> >> iommu_do_domctl() had got put between the two.
> >>
> >> Signed-off-by: Jan Beulich 
> >>
> >>  void iommu_crash_shutdown(void)
> >>  {
> >> -const struct iommu_ops *ops = iommu_get_ops();
> >>  if ( iommu_enabled )
> >> -ops->crash_shutdown();
> >> +iommu_get_ops()->crash_shutdown();
> >>  iommu_enabled = iommu_intremap = iommu_intpost = 0;
> >
> > btw, is this line still a code style issue?
> 
> Which one - the changed one or the context one? In the latter case, even if
> there were a coding style issue (which I don't see) correcting it wouldn't
> belong here.
> 
The context one -- "iommu_enabled = iommu_intremap = iommu_intpost = 0;"

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] IOMMU: don't BUG() on exotic hardware

[Xu, Quan]

On May 06, 2016 10:24 PM, Jan Beulich  wrote:
> On x86, iommu_get_ops() BUG()s when running on non-Intel, non-AMD
> hardware. While, with our current code, that's a correct prerequisite
> assumption for IOMMU presence, this is wrong on systems without IOMMU.
> Hence iommu_enabled (and alike) checks should be done prior to calling that
> function, not after.
> 
> Also move iommu_suspend() next to iommu_resume() - it escapes me why
> iommu_do_domctl() had got put between the two.
> 
> Signed-off-by: Jan Beulich 
> 
> --- a/xen/drivers/passthrough/iommu.c
> +++ b/xen/drivers/passthrough/iommu.c
> @@ -337,11 +337,16 @@ int __init iommu_setup(void)
>  return rc;
>  }
> 
> +void iommu_suspend()
> +{
> +if ( iommu_enabled )
> +iommu_get_ops()->suspend();
> +}
> +


What about this style:

+void iommu_suspend()
+{
+if ( iommu_enabled &&
+ iommu_get_ops()->suspend )
+iommu_get_ops()->suspend();
+}
+

At least for AMD, not all of the .callback are  initialized.

>  void iommu_crash_shutdown(void)
>  {
> -const struct iommu_ops *ops = iommu_get_ops();
>  if ( iommu_enabled )
> -ops->crash_shutdown();
> +iommu_get_ops()->crash_shutdown();
>  iommu_enabled = iommu_intremap = iommu_intpost = 0;

btw, is this line still a code style issue?

}


Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 0/3] Re: VT-d Device-TLB flush issue

[Xu, Quan]

On May 06, 2016 9:36 PM, Jan Beulich  wrote:
> >>> On 06.05.16 at 14:21,  wrote:
> > If i didn't miss something, there is still no comment for this patch set.
> > I think this patch set is close to being ready. I hope this patch set
> > is still in your waiting-review list, but not missed.:)
> 
> As said before, with this series going on top of the other one there's little 
> point
> trying to accelerate this one with the other still being worked on.
> 

Got it, thanks.
Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 06, 2016 3:03 PM, Jan Beulich  wrote:
> >>> On 05.05.16 at 12:18,  wrote:
> > What about this one as below:
> 
> If done properly (coding style, comments to silence Coverity, consistency
> throughout not just your change, but its effects on the rest of the code in 
> that
> file), that's certainly an option.
> 

Thanks. I'll try it in next v4.

-Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 05, 2016 7:03 PM, George Dunlap <george.dun...@citrix.com> wrote:
> On 05/05/16 07:53, Xu, Quan wrote:
> > On May 04, 2016 9:49 PM, George Dunlap <george.dun...@eu.citrix.com>
> wrote:
> >> On Fri, Apr 29, 2016 at 10:25 AM, Quan Xu <quan...@intel.com> wrote:
> >>> When IOMMU mapping is failed, we issue a best effort rollback,
> >>> stopping IOMMU mapping, unmapping the previous IOMMU maps and
> >> then
> >>> reporting the error up to the call trees. When rollback is not
> >>> feasible (in early initialization phase or trade-off of complexity)
> >>> for the hardware domain, we do things on a best effort basis, only
> >>> throwing
> >> out an error message.
> >>>
> >>> IOMMU unmapping should perhaps continue despite an error, in an
> >>> attempt to do best effort cleanup.
> >>>
> >>> Signed-off-by: Quan Xu <quan...@intel.com>
> >>>
> >>> CC: Keir Fraser <k...@xen.org>
> >>> CC: Jan Beulich <jbeul...@suse.com>
> >>> CC: Andrew Cooper <andrew.coop...@citrix.com>
> >>> CC: Jun Nakajima <jun.nakaj...@intel.com>
> >>> CC: Kevin Tian <kevin.t...@intel.com>
> >>> CC: George Dunlap <george.dun...@eu.citrix.com>
> >>> ---
> >>>  xen/arch/x86/mm.c   | 13 -
> >>>  xen/arch/x86/mm/p2m-ept.c   | 27 +--
> >>>  xen/arch/x86/mm/p2m-pt.c| 24 
> >>>  xen/arch/x86/mm/p2m.c   | 11 +--
> >>>  xen/drivers/passthrough/iommu.c | 14 +-
> >>>  5 files changed, 75 insertions(+), 14 deletions(-)
> >>>
> >>> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index
> >>> a42097f..427097d 100644
> >>> --- a/xen/arch/x86/mm.c
> >>> +++ b/xen/arch/x86/mm.c
> >>> @@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >>> int preemptible)  {
> >>>  unsigned long nx, x, y = page->u.inuse.type_info;
> >>> -int rc = 0;
> >>> +int rc = 0, ret = 0;
> >>>
> >>>  ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
> >>>
> >>> @@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >>>  if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
> >>>  {
> >>>  if ( (x & PGT_type_mask) == PGT_writable_page )
> >>> -iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
> >>> +ret = iommu_unmap_page(d, mfn_to_gmfn(d,
> >> page_to_mfn(page)));
> >>>  else if ( type == PGT_writable_page )
> >>> -iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> >>> -   page_to_mfn(page),
> >>> -   IOMMUF_readable|IOMMUF_writable);
> >>> +ret = iommu_map_page(d, mfn_to_gmfn(d,
> page_to_mfn(page)),
> >>> + page_to_mfn(page),
> >>> +
> >>> + IOMMUF_readable|IOMMUF_writable);
> >>>  }
> >>>  }
> >>>
> >>> @@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >>>  if ( (x & PGT_partial) && !(nx & PGT_partial) )
> >>>  put_page(page);
> >>>
> >>> +if ( !rc )
> >>> +rc = ret;
> >>> +
> >>>  return rc;
> >>>  }
> >>>
> >>> diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
> >>> index 1ed5b47..df87944 100644
> >>> --- a/xen/arch/x86/mm/p2m-ept.c
> >>> +++ b/xen/arch/x86/mm/p2m-ept.c
> >>> @@ -821,6 +821,8 @@ out:
> >>>  if ( needs_sync )
> >>>  ept_sync_domain(p2m);
> >>>
> >>> +ret = 0;
> >>> +
> >>>  /* For host p2m, may need to change VT-d page table.*/
> >>>  if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >>>   need_modify_vtd_table )
> >>> @@ -831,11 +833,29 @@ out:
> >>>  {
> >>>  if ( iommu_flags 

Re: [Xen-devel] [PATCH v3 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 04, 2016 4:43 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 04.05.16 at 04:14, <quan...@intel.com> wrote:
> > On May 04, 2016 10:00 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Xu, Quan
> >> > Sent: Friday, April 29, 2016 5:25 PM diff --git
> >> > a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index
> >> > 2885e31..9097333 100644
> >> > --- a/xen/arch/x86/acpi/power.c
> >> > +++ b/xen/arch/x86/acpi/power.c
> >> > @@ -45,6 +45,8 @@ void do_suspend_lowlevel(void);
> >> >
> >> >  static int device_power_down(void)  {
> >> > +int err;
> >> > +
> >> >  console_suspend();
> >> >
> >> >  time_suspend();
> >> > @@ -53,11 +55,22 @@ static int device_power_down(void)
> >> >
> >> >  ioapic_suspend();
> >> >
> >> > -iommu_suspend();
> >> > +err = iommu_suspend();
> >> > +
> >> > +if ( err )
> >> > +goto iommu_suspend_error;
> >> >
> >> >  lapic_suspend();
> >> >
> >> >  return 0;
> >> > +
> >> > + iommu_suspend_error:
> >> > +ioapic_resume();
> >> > +i8259A_resume();
> >> > +time_resume();
> >> > +console_resume();
> >> > +
> >> > +return err;
> >> >  }
> >>
> >> Jan had comment to better reuse device_power_up... looks no change in
> >> this version.
> >
> > Yes,  __iiuc__, this may be an optimization, but not a must.
> > We can discuss this in detail In this version.
> 
> As an optimization it would indeed be quite pointless here. My request was
> more for maintainability: By re-using the function future changes don't need
> to go to two places, and hence there's no risk of one of them getting
> forgotten.
> 

Jan,
What about this one as below:


--- a/xen/arch/x86/acpi/power.c
+++ b/xen/arch/x86/acpi/power.c
@@ -43,6 +43,17 @@ struct acpi_sleep_info acpi_sinfo;

 void do_suspend_lowlevel(void);

+enum dev_power_type {
+TYPE_ALL,
+TYPE_CONSOLE,
+TYPE_TIME,
+TYPE_I8259A,
+TYPE_IOAPIC,
+TYPE_IOMMU,
+TYPE_LAPIC,
+TYPE_UNKNOWN
+};
+
 static int device_power_down(void)
 {
 console_suspend();
@@ -53,26 +64,35 @@ static int device_power_down(void)

 ioapic_suspend();

-iommu_suspend();
+if ( iommu_suspend() )
+return TYPE_IOMMU;

 lapic_suspend();

 return 0;
 }

-static void device_power_up(void)
+static void device_power_up(enum dev_power_type type)
 {
-lapic_resume();
-
-iommu_resume();
-
-ioapic_resume();
-
-i8259A_resume();
-
-time_resume();
-
-console_resume();
+switch ( type ) {
+case TYPE_ALL:
+case TYPE_LAPIC:
+lapic_resume();
+case TYPE_IOMMU:
+iommu_resume();
+case TYPE_IOAPIC:
+ioapic_resume();
+case TYPE_I8259A:
+i8259A_resume();
+case TYPE_TIME:
+time_resume();
+case TYPE_CONSOLE:
+console_resume();
+break;
+default:
+BUG();
+break;
+}
 }

 static void freeze_domains(void)
@@ -169,6 +189,7 @@ static int enter_state(u32 state)
 {
 printk(XENLOG_ERR "Some devices failed to power down.");
 system_state = SYS_STATE_resume;
+device_power_up(error);
 goto done;
 }

@@ -196,7 +217,7 @@ static int enter_state(u32 state)
 write_cr4(cr4 & ~X86_CR4_MCE);
 write_efer(read_efer());

-device_power_up();
+device_power_up(TYPE_ALL);

 mcheck_init(_cpu_data, 0);
 write_cr4(cr4);

--
Quan




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 04, 2016 9:49 PM, George Dunlap <george.dun...@eu.citrix.com> wrote:
> On Fri, Apr 29, 2016 at 10:25 AM, Quan Xu <quan...@intel.com> wrote:
> > When IOMMU mapping is failed, we issue a best effort rollback,
> > stopping IOMMU mapping, unmapping the previous IOMMU maps and
> then
> > reporting the error up to the call trees. When rollback is not
> > feasible (in early initialization phase or trade-off of complexity)
> > for the hardware domain, we do things on a best effort basis, only throwing
> out an error message.
> >
> > IOMMU unmapping should perhaps continue despite an error, in an
> > attempt to do best effort cleanup.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Keir Fraser <k...@xen.org>
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Andrew Cooper <andrew.coop...@citrix.com>
> > CC: Jun Nakajima <jun.nakaj...@intel.com>
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: George Dunlap <george.dun...@eu.citrix.com>
> > ---
> >  xen/arch/x86/mm.c   | 13 -
> >  xen/arch/x86/mm/p2m-ept.c   | 27 +--
> >  xen/arch/x86/mm/p2m-pt.c| 24 
> >  xen/arch/x86/mm/p2m.c   | 11 +--
> >  xen/drivers/passthrough/iommu.c | 14 +-
> >  5 files changed, 75 insertions(+), 14 deletions(-)
> >
> > diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index
> > a42097f..427097d 100644
> > --- a/xen/arch/x86/mm.c
> > +++ b/xen/arch/x86/mm.c
> > @@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info
> *page, unsigned long type,
> > int preemptible)  {
> >  unsigned long nx, x, y = page->u.inuse.type_info;
> > -int rc = 0;
> > +int rc = 0, ret = 0;
> >
> >  ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
> >
> > @@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info
> *page, unsigned long type,
> >  if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
> >  {
> >  if ( (x & PGT_type_mask) == PGT_writable_page )
> > -iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
> > +ret = iommu_unmap_page(d, mfn_to_gmfn(d,
> page_to_mfn(page)));
> >  else if ( type == PGT_writable_page )
> > -iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> > -   page_to_mfn(page),
> > -   IOMMUF_readable|IOMMUF_writable);
> > +ret = iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> > + page_to_mfn(page),
> > + IOMMUF_readable|IOMMUF_writable);
> >  }
> >  }
> >
> > @@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info
> *page, unsigned long type,
> >  if ( (x & PGT_partial) && !(nx & PGT_partial) )
> >  put_page(page);
> >
> > +if ( !rc )
> > +rc = ret;
> > +
> >  return rc;
> >  }
> >
> > diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
> > index 1ed5b47..df87944 100644
> > --- a/xen/arch/x86/mm/p2m-ept.c
> > +++ b/xen/arch/x86/mm/p2m-ept.c
> > @@ -821,6 +821,8 @@ out:
> >  if ( needs_sync )
> >  ept_sync_domain(p2m);
> >
> > +ret = 0;
> > +
> >  /* For host p2m, may need to change VT-d page table.*/
> >  if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >   need_modify_vtd_table )
> > @@ -831,11 +833,29 @@ out:
> >  {
> >  if ( iommu_flags )
> >  for ( i = 0; i < (1 << order); i++ )
> > -iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
> > iommu_flags);
> > +{
> > +rc = iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
> > iommu_flags);
> > +
> > +if ( !ret && unlikely(rc) )
> > +{
> > +while ( i-- )
> > +iommu_unmap_page(d, gfn + i);
> > +
> > +ret = rc;
> > +break;
> > +}
> > +}
> >  else
> >  for ( i = 0; i < (1 << order); i++ )
> > -iommu_unmap_pa

Re: [Xen-devel] [PATCH v3 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 04, 2016 9:44 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 04.05.16 at 13:49, <quan...@intel.com> wrote:
> > On May 04, 2016 9:29 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Quan Xu
> >> > Sent: Friday, April 29, 2016 5:25 PM
> >> >
> >> > Treat IOMMU mapping and unmapping failures as a fatal to the domain
> >> > (with the exception of the hardware domain).
> >> >
> >> > If IOMMU mapping and unmapping failed, crash the domain (with the
> >> > exception of the hardware domain) and propagate the error up to the
> >> > call trees.
> >> >
> >> > Signed-off-by: Quan Xu <quan...@intel.com>
> >> > Reviewed-by: Kevin Tian <kevin.t...@intel.com>
> >> >
> >> > CC: Jan Beulich <jbeul...@suse.com>
> >> > ---
> >> >  xen/drivers/passthrough/iommu.c | 30
> >> > --
> >> >  1 file changed, 28 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/xen/drivers/passthrough/iommu.c
> >> > b/xen/drivers/passthrough/iommu.c index b64676f..a0003ac 100644
> >> > --- a/xen/drivers/passthrough/iommu.c
> >> > +++ b/xen/drivers/passthrough/iommu.c
> >> > @@ -243,21 +243,47 @@ int iommu_map_page(struct domain *d,
> >> unsigned
> >> > long gfn, unsigned long mfn,
> >> > unsigned int flags)  {
> >> >  struct hvm_iommu *hd = domain_hvm_iommu(d);
> >> > +int rc;
> >> >
> >> >  if ( !iommu_enabled || !hd->platform_ops )
> >> >  return 0;
> >> >
> >> > -return hd->platform_ops->map_page(d, gfn, mfn, flags);
> >> > +rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
> >> > +
> >> > +if ( rc )
> >> > +{
> >> > +if ( is_hardware_domain(d) )
> >> > +printk(XENLOG_ERR
> >> > +   "iommu_map_page: IOMMU mapping gfn %#lx mfn
> >> > + %#lx failed for
> >> > dom%d.",
> >> > +   gfn, mfn, d->domain_id);
> >> > +else
> >> > +domain_crash(d);
> >> > +}
> >>
> >> It makes sense to print error messages for all domains, and then
> >> selectively crash domain:
> >>
> >> printk(XENLOG_ERR ...);
> >> if ( !is_hardware_domain(d) )
> >>domain_crash(d);
> >>
> >
> > But Jan said,
> > ..
> > if ( is_hardware_domain() )
> > printk();
> > else
> > domain_crash();
> > ..
> > is the better approach.
> 
> Not exactly. All I complained about was that the Dom0 case went completely
> silently.
> 
> > I remain neutral for this point. I think this is not a technical
> > problem, but a matter of preference.
> 
> Indeed.
>

If no other suggestions, I'll take Kevin's suggestion as a final  conclusion 
for both iommu_map_page() and iommu_unmap_page().
 

Quan

> > This gap is subject to preventing spamming the log.
> >
> > For iommu_map_page(), I think Kevin's suggestion is better, much more
> > information for the crash domain, And also won't spam the log as we
> > stop mapping against any error.
> >
> > For iommu_unmap_page(),IOMMU unmapping should perhaps continue
> despite
> > an error, in an attempt to do best effort cleanup. Then, Kevin's
> > suggestion may spam the log.
> 
> But I've always been saying that for multiple successive operations you
> shouldn't issue one message each.


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 04, 2016 9:52 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 04.05.16 at 14:09, <quan...@intel.com> wrote:
> > On May 04, 2016 9:26 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Xu, Quan
> >> > Sent: Friday, April 29, 2016 5:25 PM  static void
> >> > intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> >> > unsigned int
> >> > page_count)
> >>
> >> could we remove "Intel_" prefix completely? You can rename this one
> >> to iommu_flush_iotlb_page...
> >>
> >
> > Sure, I am ok.
> > I wonder why to remove "Intel_" prefix.
> > In this xen/drivers/passthrough/vtd/iommu.c file, most of functions
> > are beginning with "intel_" as intel specific.
> > In xen/drivers/passthrough/iommu.c file, most of functions are
> > beginning with 'iommu_' as common part.
> 
> For non-static functions I'd suggest to keep the prefix (or use vtd_
> alternatively). For static functions there's no need to disambiguate them, and
> hence an intel_ prefix is just mostly useless baggage.
> 

Thanks. The above two are both static functions. So  I will follow Kevin's 
suggestion.
Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 02/10] IOMMU: handle IOMMU mapping and unmapping failures

[Xu, Quan]

On May 04, 2016 9:29 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Quan Xu
> > Sent: Friday, April 29, 2016 5:25 PM
> >
> > Treat IOMMU mapping and unmapping failures as a fatal to the domain
> > (with the exception of the hardware domain).
> >
> > If IOMMU mapping and unmapping failed, crash the domain (with the
> > exception of the hardware domain) and propagate the error up to the
> > call trees.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> > Reviewed-by: Kevin Tian <kevin.t...@intel.com>
> >
> > CC: Jan Beulich <jbeul...@suse.com>
> > ---
> >  xen/drivers/passthrough/iommu.c | 30 --
> >  1 file changed, 28 insertions(+), 2 deletions(-)
> >
> > diff --git a/xen/drivers/passthrough/iommu.c
> > b/xen/drivers/passthrough/iommu.c index b64676f..a0003ac 100644
> > --- a/xen/drivers/passthrough/iommu.c
> > +++ b/xen/drivers/passthrough/iommu.c
> > @@ -243,21 +243,47 @@ int iommu_map_page(struct domain *d,
> unsigned
> > long gfn, unsigned long mfn,
> > unsigned int flags)  {
> >  struct hvm_iommu *hd = domain_hvm_iommu(d);
> > +int rc;
> >
> >  if ( !iommu_enabled || !hd->platform_ops )
> >  return 0;
> >
> > -return hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +rc = hd->platform_ops->map_page(d, gfn, mfn, flags);
> > +
> > +if ( rc )
> > +{
> > +if ( is_hardware_domain(d) )
> > +printk(XENLOG_ERR
> > +   "iommu_map_page: IOMMU mapping gfn %#lx mfn %#lx
> > + failed for
> > dom%d.",
> > +   gfn, mfn, d->domain_id);
> > +else
> > +domain_crash(d);
> > +}
> 
> It makes sense to print error messages for all domains, and then selectively
> crash domain:
> 
> printk(XENLOG_ERR ...);
> if ( !is_hardware_domain(d) )
>   domain_crash(d);
> 

But Jan said,
..
if ( is_hardware_domain() )
printk();
else
domain_crash();
..
is the better approach.


I remain neutral for this point. I think this is not a technical problem, but a 
matter of preference.
This gap is subject to preventing spamming the log. 

For iommu_map_page(), I think Kevin's suggestion is better, much more 
information for the crash domain,
And also won't spam the log as we stop mapping against any error.

For iommu_unmap_page(),IOMMU unmapping should perhaps continue despite an 
error, in an attempt
to do best effort cleanup. Then, Kevin's suggestion may spam the log.


Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 01/10] vt-d: fix the IOMMU flush issue

[Xu, Quan]

On May 04, 2016 9:26 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, April 29, 2016 5:25 PM
> >
> > The propagation value from IOMMU flush interfaces may be positive,
> > which indicates callers need to flush cache, not one of faliures.
> >
> > when the propagation value is positive, this patch fixes this flush
> > issue as follows:
> >   - call iommu_flush_write_buffer() to flush cache.
> >   - return zero.
> >
> > Signed-off-by: Quan Xu <quan...@intel.com>
> >
> > CC: Kevin Tian <kevin.t...@intel.com>
> > CC: Feng Wu <feng...@intel.com>
> > CC: Keir Fraser <k...@xen.org>
> > CC: Jan Beulich <jbeul...@suse.com>
> > CC: Andrew Cooper <andrew.coop...@citrix.com>
> > ---
> >  xen/drivers/passthrough/vtd/iommu.c | 94
> > +
> >  xen/include/asm-x86/iommu.h |  2 +-
> >  2 files changed, 64 insertions(+), 32 deletions(-)
> >
> > diff --git a/xen/drivers/passthrough/vtd/iommu.c
> > b/xen/drivers/passthrough/vtd/iommu.c
> > index 5ad25dc..6e2e43a 100644
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -558,14 +558,16 @@ static void iommu_flush_all(void)
> >  }
> >  }
> >
> > -static void __intel_iommu_iotlb_flush(struct domain *d, unsigned long gfn,
> > -int dma_old_pte_present, unsigned int page_count)
> > +static int iommu_flush_iotlb(struct domain *d, unsigned long gfn,
> > + bool_t dma_old_pte_present,
> > + unsigned int page_count)
> >  {
> >  struct hvm_iommu *hd = domain_hvm_iommu(d);
> >  struct acpi_drhd_unit *drhd;
> >  struct iommu *iommu;
> >  int flush_dev_iotlb;
> >  int iommu_domid;
> > +int rc, ret = 0;
> >
> >  /*
> >   * No need pcideves_lock here because we have flush @@ -584,29
> > +586,35 @@ static void __intel_iommu_iotlb_flush(struct domain *d,
> > unsigned long gfn,
> >  continue;
> >
> >  if ( page_count != 1 || gfn == INVALID_GFN )
> > -{
> > -if ( iommu_flush_iotlb_dsi(iommu, iommu_domid,
> > -0, flush_dev_iotlb) )
> > -iommu_flush_write_buffer(iommu);
> > -}
> > +rc = iommu_flush_iotlb_dsi(iommu, iommu_domid,
> > +   0, flush_dev_iotlb);
> >  else
> > +rc = iommu_flush_iotlb_psi(iommu, iommu_domid,
> > +   (paddr_t)gfn << PAGE_SHIFT_4K,
> > +   PAGE_ORDER_4K,
> > +   !dma_old_pte_present,
> > +   flush_dev_iotlb);
> > +
> > +if ( rc > 0 )
> >  {
> > -if ( iommu_flush_iotlb_psi(iommu, iommu_domid,
> > -(paddr_t)gfn << PAGE_SHIFT_4K, PAGE_ORDER_4K,
> > -!dma_old_pte_present, flush_dev_iotlb) )
> > -iommu_flush_write_buffer(iommu);
> > +iommu_flush_write_buffer(iommu);
> > +ret = 0;
> 
> You don't need 'ret' here. Just change 'rc' to 0.
>

Agreed, this is really a good idea.
 
> >  }
> > +else if ( rc < 0 )
> > +ret = rc;
> 
> Then this change is not required
> 

Ditto.

> >  }
> > +
> > +return ret;
> 
> Then return 'rc' instead.
> 

Ditto.

> >  }
> >
> >  static void intel_iommu_iotlb_flush(struct domain *d, unsigned long
> > gfn, unsigned int
> > page_count)
> 
> could we remove "Intel_" prefix completely? You can rename this one to
> iommu_flush_iotlb_page...
>

Sure, I am ok.
I wonder why to remove "Intel_" prefix.
In this xen/drivers/passthrough/vtd/iommu.c file, most of functions are 
beginning with "intel_" as intel specific.
In xen/drivers/passthrough/iommu.c file, most of functions are beginning with 
'iommu_' as common part.

Quan


 
> >  {
> > -__intel_iommu_iotlb_flush(d, gfn, 1, page_count);
> > +iommu_flush_iotlb(d, gfn, 1, page_count);
> >  }
> >
> >  static void intel_iommu_iotlb_flush_all(struct domain *d)
> 
> and this one just iommu_flush_iotlb_all
> 
> >  {
> > -__intel_iommu_iotlb_flush(d, INVALID_GFN, 0, 0);
> > +iommu_flush_iotlb(d, INVALID_GFN, 0, 0);
> >  }
> >
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 03/10] IOMMU/MMU: enhance the call trees of IOMMU unmapping and mapping

[Xu, Quan]

On May 04, 2016 4:41 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 04.05.16 at 03:45, <kevin.t...@intel.com> wrote:
> >>  From: Xu, Quan
> >> Sent: Friday, April 29, 2016 5:25 PM
> >> --- a/xen/arch/x86/mm.c
> >> +++ b/xen/arch/x86/mm.c
> >> @@ -2467,7 +2467,7 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >> int preemptible)  {
> >>  unsigned long nx, x, y = page->u.inuse.type_info;
> >> -int rc = 0;
> >> +int rc = 0, ret = 0;
> >>
> >>  ASSERT(!(type & ~(PGT_type_mask | PGT_pae_xen_l2)));
> >>
> >> @@ -2578,11 +2578,11 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >>  if ( d && is_pv_domain(d) && unlikely(need_iommu(d)) )
> >>  {
> >>  if ( (x & PGT_type_mask) == PGT_writable_page )
> >> -iommu_unmap_page(d, mfn_to_gmfn(d, page_to_mfn(page)));
> >> +ret = iommu_unmap_page(d, mfn_to_gmfn(d,
> >> + page_to_mfn(page)));
> >>  else if ( type == PGT_writable_page )
> >> -iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> >> -   page_to_mfn(page),
> >> -   IOMMUF_readable|IOMMUF_writable);
> >> +ret = iommu_map_page(d, mfn_to_gmfn(d, page_to_mfn(page)),
> >> + page_to_mfn(page),
> >> +
> >> + IOMMUF_readable|IOMMUF_writable);
> >>  }
> >>  }
> >>
> >> @@ -2599,6 +2599,9 @@ static int __get_page_type(struct page_info
> >> *page, unsigned long type,
> >>  if ( (x & PGT_partial) && !(nx & PGT_partial) )
> >>  put_page(page);
> >>
> >> +if ( !rc )
> >> +rc = ret;
> >> +
> >>  return rc;
> >>  }
> >
> > I know there were quite some discussions before around above change
> > (sorry I didn't remember all of them). Just based on mental picture we
> > should return error where it firstly occurs. However above change
> > looks favoring errors in later "rc = alloc_page_type" over earlier
> > iommu_map/unmap_page error. Is it what we want?
> 
> Yes, as that's the primary operation here.
> 
> > If there is a reason that we cannot return immediately upon
> > iommu_map/unmap,
> 
> Since for Dom0 we don't call domain_crash(), we must not bypass
> alloc_page_type() here. And even for DomU it would seem at least fragile if we
> did - we better don't alter the refcounting behavior.
> 

A little bit confused.
Check it,  for this __get_page_type(), can I leave my modification as is?

> >> --- a/xen/arch/x86/mm/p2m-ept.c
> >> +++ b/xen/arch/x86/mm/p2m-ept.c
> >> @@ -821,6 +821,8 @@ out:
> >>  if ( needs_sync )
> >>  ept_sync_domain(p2m);
> >>
> >> +ret = 0;
> >> +
> >>  /* For host p2m, may need to change VT-d page table.*/
> >>  if ( rc == 0 && p2m_is_hostp2m(p2m) && need_iommu(d) &&
> >>   need_modify_vtd_table )
> >> @@ -831,11 +833,29 @@ out:
> >>  {
> >>  if ( iommu_flags )
> >>  for ( i = 0; i < (1 << order); i++ )
> >> -iommu_map_page(d, gfn + i, mfn_x(mfn) + i, 
> >> iommu_flags);
> >> +{
> >> +rc = iommu_map_page(d, gfn + i, mfn_x(mfn) + i,
> >> + iommu_flags);
> >> +
> >> +if ( !ret && unlikely(rc) )
> >
> > I think you should move check of ret before iommu_map_page, since we
> > should stop map against any error (either from best-effort unmap error
> side).
> 
> Considering ret getting set to zero ahead of the loop plus ...
> 
> >> +{
> >> +while ( i-- )
> >> +iommu_unmap_page(d, gfn + i);
> >> +
> >> +ret = rc;
> >> +break;
> 
> ... this, it looks to me as if the checking of ret above is simply 
> unnecessary.
> 

Make sense. I'll drop ret check.

Quan



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 04, 2016 4:43 PM, Jan Beulich <jbeul...@suse.com> wrote:
> >>> On 04.05.16 at 04:14, <quan...@intel.com> wrote:
> > On May 04, 2016 10:00 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> >> > From: Xu, Quan
> >> > Sent: Friday, April 29, 2016 5:25 PM diff --git
> >> > a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c index
> >> > 2885e31..9097333 100644
> >> > --- a/xen/arch/x86/acpi/power.c
> >> > +++ b/xen/arch/x86/acpi/power.c
> >> > @@ -45,6 +45,8 @@ void do_suspend_lowlevel(void);
> >> >
> >> >  static int device_power_down(void)  {
> >> > +int err;
> >> > +
> >> >  console_suspend();
> >> >
> >> >  time_suspend();
> >> > @@ -53,11 +55,22 @@ static int device_power_down(void)
> >> >
> >> >  ioapic_suspend();
> >> >
> >> > -iommu_suspend();
> >> > +err = iommu_suspend();
> >> > +
> >> > +if ( err )
> >> > +goto iommu_suspend_error;
> >> >
> >> >  lapic_suspend();
> >> >
> >> >  return 0;
> >> > +
> >> > + iommu_suspend_error:
> >> > +ioapic_resume();
> >> > +i8259A_resume();
> >> > +time_resume();
> >> > +console_resume();
> >> > +
> >> > +return err;
> >> >  }
> >>
> >> Jan had comment to better reuse device_power_up... looks no change in
> >> this version.
> >
> > Yes,  __iiuc__, this may be an optimization, but not a must.
> > We can discuss this in detail In this version.
> 
> As an optimization it would indeed be quite pointless here. My request was
> more for maintainability: By re-using the function future changes don't need
> to go to two places, and hence there's no risk of one of them getting
> forgotten.
> 

Got it, I will fix it in next v4.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86/HVM: fix forwarding of internally cached requests

[Xu, Quan]

On May 03, 2016 9:39 PM, Jan Beulich  wrote:
> >>> On 03.05.16 at 14:50,  wrote:
> > On April 29, 2016 12:11 AM,  wrote:
> >> On April 28, 2016 11:13 PM, Jan Beulich  wrote:
> >> > >>> On 25.04.16 at 10:40,  wrote:
> >> > > With other patches also in place, still not work.
> >> > > Jianzhong  has been left and Quan will take over the task.
> >> >
> >> > May I ask for another try, with current tip of staging plus
> >> > http://lists.xenproject.org/archives/html/xen-devel/2016-
> >> > 04/msg03661.html
> >> > ?
> >>
> >>
> >> Sure, I will try it on tomorrow afternoon or next Tuesday.
> >
> > Jan,
> > Sorry, this is still working in progress.
> >  The original env is broken. I am rebuilding a new env, with some
> > problems of Windows image.
> > I checked the staging branch, and I found your patches are in staging
> > branch now,  so I am just going to test staging branch. Right?
> 
> Yes.

Jan,
The VF is still _not_ working for Windows_Server2008, but the PF does.
My nic is  'Intel Corporation I350 Gigabit Network Connection',
and the last commit is 'c2994f86'.
I also tested the RHEL6 VM, the same result as Windows_Server2008.

I'll also go into this issue.
Any update, let me help you test it.

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 10/10] vt-d: propagate error up to ME phantom function mapping and unmapping

[Xu, Quan]

On May 04, 2016 10:02 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, April 29, 2016 5:25 PM
> > diff --git a/xen/drivers/passthrough/vtd/iommu.c
> > b/xen/drivers/passthrough/vtd/iommu.c
> > index cf847ec..bfee299 100644
> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > @@ -1301,7 +1301,7 @@ int domain_context_mapping_one(
> >  u64 maddr, pgd_maddr;
> >  u16 seg = iommu->intel->drhd->segment;
> >  int agaw;
> > -int rc;
> > +int rc, ret;
> >
> >  ASSERT(pcidevs_locked());
> >  spin_lock(>lock);
> > @@ -1435,7 +1435,12 @@ int domain_context_mapping_one(
> >  unmap_vtd_domain_page(context_entries);
> >
> >  if ( !seg )
> > -me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
> > +{
> > +ret = me_wifi_quirk(domain, bus, devfn, MAP_ME_PHANTOM_FUNC);
> > +
> > +if ( !rc && unlikely(ret) )
> > +rc = ret;
> > +}
> 
> similarly to make it simple no need to check ret again.

Agreed, I also hesitated to add this check. I'll drop similar ret check in this 
patch set.

Quan

> 
> >
> >  return rc;
> >  }


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 09/10] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On May 04, 2016 10:00 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, April 29, 2016 5:25 PM
> > diff --git a/xen/arch/x86/acpi/power.c b/xen/arch/x86/acpi/power.c
> > index 2885e31..9097333 100644
> > --- a/xen/arch/x86/acpi/power.c
> > +++ b/xen/arch/x86/acpi/power.c
> > @@ -45,6 +45,8 @@ void do_suspend_lowlevel(void);
> >
> >  static int device_power_down(void)
> >  {
> > +int err;
> > +
> >  console_suspend();
> >
> >  time_suspend();
> > @@ -53,11 +55,22 @@ static int device_power_down(void)
> >
> >  ioapic_suspend();
> >
> > -iommu_suspend();
> > +err = iommu_suspend();
> > +
> > +if ( err )
> > +goto iommu_suspend_error;
> >
> >  lapic_suspend();
> >
> >  return 0;
> > +
> > + iommu_suspend_error:
> > +ioapic_resume();
> > +i8259A_resume();
> > +time_resume();
> > +console_resume();
> > +
> > +return err;
> >  }
> 
> Jan had comment to better reuse device_power_up... looks no change in this
> version.

Yes,  __iiuc__, this may be an optimization, but not a must.
We can discuss this in detail In this version. 

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86/HVM: fix forwarding of internally cached requests

[Xu, Quan]

On April 29, 2016 12:11 AM,  wrote:
> On April 28, 2016 11:13 PM, Jan Beulich  wrote:
> > >>> On 25.04.16 at 10:40,  wrote:
> > > With other patches also in place, still not work.
> > > Jianzhong  has been left and Quan will take over the task.
> >
> > May I ask for another try, with current tip of staging plus
> > http://lists.xenproject.org/archives/html/xen-devel/2016-
> > 04/msg03661.html
> > ?
> 
> 
> Sure, I will try it on tomorrow afternoon or next Tuesday.

Jan,
Sorry, this is still working in progress.
 The original env is broken. I am rebuilding a new env, with some problems of 
Windows image.
I checked the staging branch, and I found your patches are in staging branch 
now,
 so I am just going to test staging branch. Right?

Quan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] question about ept_set_entry

[Xu, Quan]

On April 29, 2016 5:24 PM, Jan Beulich  wrote:
> >>> On 29.04.16 at 11:14,  wrote:
> > hi all
> >
> > in
> > - p2m-ept.c
> >   - ept_set_entry
> >
> > i see code about iommu,
> > those code  for what?

Maybe Chunyu's question is -- why  EPT update is  along with IOMMU page table 
update.
I guess :(.

Quan

> 
> For keeping the IOMMU translation tables in sync with the CPU ones?
> But perhaps I'm simply not understanding what you try to get at...
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 09/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On April 29, 2016 3:14 PM, Jan Beulich  wrote:
>  >>> On 29.04.16 at 04:41,  wrote:
> > Now I check the status in caller to make the print include caller
> > which is failed, instead print in iommu_flush_all().
> > i.e.,
> > vtd_crash_shutdown()
> > {
> > ..
> > if ( iommu_flush_all() )
> > printk(XENLOG_WARNING VTDPREFIX
> >" vtd_crash_shutdown: IOMMU flush all failed.\n"); ..
> > }
> >
> > I am afraid I still don't get the point. To be honest, in such a fix,
> > The print is not so useful to me ( Correct me, I will continue to
> > enhance it).
> 
> So do you think it would be more useful to leave the admin with no clue why a
> system misbehaves, instead of providing clear indication?
> IOW - what usefulness concerns do you have?
> 

Jan, I will follow your suggestion for v3. I look forward to hearing  the other 
CCed maintainers' opinions on this.

...
i.e., In above case, in case I am an admin, I may be much more interested in 
what causes vt-d crash,
keeping quite during crash. It is just a matter of my personal preference. 
Ignore me.
...

Quan

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 09/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

 On April 29, 2016 10:21 AM, Tian, Kevin <kevin.t...@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, April 29, 2016 12:09 AM
> > On April 28, 2016 11:12 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > > >>> On 28.04.16 at 17:03, <quan...@intel.com> wrote:
> > > > On April 28, 2016 10:36 PM, Jan Beulich <jbeul...@suse.com> wrote:
> > > >> >>> On 28.04.16 at 16:14, <quan...@intel.com> wrote:
> > > >> > On April 25, 2016 7:53 PM, Jan Beulich <jbeul...@suse.com>
> wrote:
> > > >> >> >>> On 18.04.16 at 16:00, <quan...@intel.com> wrote:
> > > >> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> > > >> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> > > >> >
> > > >> >
> > > >> >> > -static void iommu_flush_all(void)
> > > >> >> > +static int iommu_flush_all(void)
> > > >> >>
> > > >> >> __must_check
> > > >> >>
> > > >> >
> > > >> > The iommu_flush_all() is also called in
> > > >> > intel_iommu_hwdom_init() and vtd_crash_shutdown().
> > > >> > As we were on the same page, we can ignore the error code
> > > >> > propagation for these two call trees.
> > > >>
> > > >> I don't know what you're referring to here with "we were on the
> > > >> same
> > > page".
> > > > I
> > > >> don't think I've ever agreed (in the context of this series) to
> > > >> ignore any
> > > > error
> > > >> returns.
> > > >>
> > > >
> > > >
> > > > Look at the below link.
> > > > http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg032
> > > > 34.h
> > > > tml
> > >
> > > Which still talks about (conditionally) logging messages, not ignoring of
> errors.
> > >
> >
> > Sorry, check it again.
> > in intel_iommu_hwdom_init() and  vtd_crash_shutdown(), I still need to
> > log messages as:
> >
> > if ( iommu_flush_all() )
> >  printk()
> >
> > ?
> >
> > Thanks for your patience.
> > Quan
> 
> Yes, but please make sure you only print once to be useful.
>

 
Thanks.
Now I check the status in caller to make the print include caller which is 
failed, instead print in iommu_flush_all().
i.e., 
vtd_crash_shutdown()
{
..
if ( iommu_flush_all() )
printk(XENLOG_WARNING VTDPREFIX
   " vtd_crash_shutdown: IOMMU flush all failed.\n");
..
}

I am afraid I still don't get the point. To be honest, in such a fix,
The print is not so useful to me ( Correct me, I will continue to enhance it).

Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2] x86/HVM: fix forwarding of internally cached requests

[Xu, Quan]

On April 28, 2016 11:13 PM, Jan Beulich  wrote:
> >>> On 25.04.16 at 10:40,  wrote:
> > With other patches also in place, still not work.
> > Jianzhong  has been left and Quan will take over the task.
> 
> May I ask for another try, with current tip of staging plus
> http://lists.xenproject.org/archives/html/xen-devel/2016-
> 04/msg03661.html
> ?


Sure, I will try it on tomorrow afternoon or next Tuesday.
Quan
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 09/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On April 28, 2016 11:12 PM, Jan Beulich  wrote:
> >>> On 28.04.16 at 17:03,  wrote:
> > On April 28, 2016 10:36 PM, Jan Beulich  wrote:
> >> >>> On 28.04.16 at 16:14,  wrote:
> >> > On April 25, 2016 7:53 PM, Jan Beulich  wrote:
> >> >> >>> On 18.04.16 at 16:00,  wrote:
> >> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >> >
> >> >
> >> >> > -static void iommu_flush_all(void)
> >> >> > +static int iommu_flush_all(void)
> >> >>
> >> >> __must_check
> >> >>
> >> >
> >> > The iommu_flush_all() is also called in intel_iommu_hwdom_init()
> >> > and vtd_crash_shutdown().
> >> > As we were on the same page, we can ignore the error code
> >> > propagation for these two call trees.
> >>
> >> I don't know what you're referring to here with "we were on the same
> page".
> > I
> >> don't think I've ever agreed (in the context of this series) to
> >> ignore any
> > error
> >> returns.
> >>
> >
> >
> > Look at the below link.
> > http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg03234.h
> > tml
> 
> Which still talks about (conditionally) logging messages, not ignoring of 
> errors.
> 

Sorry, check it again.
in intel_iommu_hwdom_init() and  vtd_crash_shutdown(), 
I still need to log messages as:

if ( iommu_flush_all() )
 printk()

?

Thanks for your patience.
Quan






___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 09/11] IOMMU: propagate IOMMU Device-TLB flush error up to IOMMU suspending

[Xu, Quan]

On April 28, 2016 10:36 PM, Jan Beulich  wrote:
> >>> On 28.04.16 at 16:14,  wrote:
> > On April 25, 2016 7:53 PM, Jan Beulich  wrote:
> >> >>> On 18.04.16 at 16:00,  wrote:
> >> > --- a/xen/drivers/passthrough/vtd/iommu.c
> >> > +++ b/xen/drivers/passthrough/vtd/iommu.c
> >
> >
> >> > -static void iommu_flush_all(void)
> >> > +static int iommu_flush_all(void)
> >>
> >> __must_check
> >>
> >
> > The iommu_flush_all() is also called in intel_iommu_hwdom_init()  and
> > vtd_crash_shutdown().
> > As we were on the same page, we can ignore the error code propagation
> > for these two call trees.
> 
> I don't know what you're referring to here with "we were on the same page". I
> don't think I've ever agreed (in the context of this series) to ignore any 
> error
> returns.
> 


Look at the below link. 
http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg03234.html

I hope I understand this correctly.

Quan

> Jan
> 
> > I wonder whether we really need this '__must_check' or not,
> > furthermore, print-out message Is pointless (at least, to me).
> >
> > Jan, could I ignore this '__must_check '?
> >
> > Quan
> 
> 
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel