Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-25 Thread Jan Beulich
>>> On 09.09.15 at 10:56,  wrote:
 On 08.09.15 at 15:21,  wrote:
>> I got Xen booting now with VT-d enabled, lot's of errors and the system is
>> unstable. xl dmesg output:
> 
> Lots of errors? I don't see much - can you point out the errors you see?
> There are two interesting things:
> 
>> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = 
>> 82c000201000
>> (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b
>>[...]
>> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = 
>> 82c000203000
>> (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df
> 
> Two IOMMUs with different capabilities. I'll have to look at what
> specifically one supports which the other doesn't, but I wouldn't
> be surprised if code wouldn't fully cope with that.

cap[40..42] represent the number of fault registers (afaict not an
issue if there's just one).

ecap[2] (device IOTLB) is relevant only when using ATS, which you
don't enable.

ecap[7] (snoop control) unavailable on any IOMMU will just lead
to it not getting used at all (equivalent to "iommu=no-snoop" on
the command line).

So all in all the feature differences are benign.

>> (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
> 
> But later ...
> 
>> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
>> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
>> (XEN) Intel VT-d Snoop Control not enabled.
>> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
>> (XEN) Intel VT-d Queued Invalidation enabled.
>> (XEN) Intel VT-d Interrupt Remapping enabled.
> 
> ... and ...
> 
>> (XEN) I/O virtualisation enabled
>> (XEN)  - Dom0 mode: Relaxed
>> (XEN) Interrupt remapping enabled
> 
> I'll have to check what this inconsistency means.

So from all I can tell this is bogus message wording when the firmware
requests the OS not to enabled x2APIC mode. Below a patch hopefully
making this less confusing.

Jan

--- unstable.orig/xen/arch/x86/apic.c
+++ unstable/xen/arch/x86/apic.c
@@ -943,8 +943,18 @@ void __init x2apic_bsp_setup(void)
 mask_8259A();
 mask_IO_APIC_setup(ioapic_entries);
 
-if ( iommu_enable_x2apic_IR() )
+switch ( iommu_enable_x2apic_IR() )
 {
+case 0:
+break;
+case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */
+if ( !x2apic_enabled )
+{
+printk("Not enabling x2APIC (upon firmware request)\n");
+goto restore_out;
+}
+/* fall through */
+default:
 if ( x2apic_enabled )
 panic("Interrupt remapping could not be enabled while "
   "x2APIC is already enabled by BIOS");
--- unstable.orig/xen/drivers/passthrough/vtd/intremap.c
+++ unstable/xen/drivers/passthrough/vtd/intremap.c
@@ -143,10 +143,10 @@ static void set_hpet_source_id(unsigned 
 set_ire_sid(ire, SVT_VERIFY_SID_SQ, SQ_13_IGNORE_3, hpetid_to_bdf(id));
 }
 
-int iommu_supports_eim(void)
+bool_t iommu_supports_eim(void)
 {
 struct acpi_drhd_unit *drhd;
-int apic;
+unsigned int apic;
 
 if ( !iommu_qinval || !iommu_intremap || list_empty(&acpi_drhd_units) )
 return 0;
@@ -154,12 +154,12 @@ int iommu_supports_eim(void)
 /* We MUST have a DRHD unit for each IOAPIC. */
 for ( apic = 0; apic < nr_ioapics; apic++ )
 if ( !ioapic_to_drhd(IO_APIC_ID(apic)) )
-{
+{
 dprintk(XENLOG_WARNING VTDPREFIX,
 "There is not a DRHD for IOAPIC %#x (id: %#x)!\n",
 apic, IO_APIC_ID(apic));
 return 0;
-}
+}
 
 for_each_drhd_unit ( drhd )
 if ( !ecap_queued_inval(drhd->iommu->ecap) ||
@@ -833,10 +833,10 @@ int iommu_enable_x2apic_IR(void)
 struct iommu *iommu;
 
 if ( !iommu_supports_eim() )
-return -1;
+return -EOPNOTSUPP;
 
 if ( !platform_supports_x2apic() )
-return -1;
+return -ENXIO;
 
 for_each_drhd_unit ( drhd )
 {
@@ -861,7 +861,7 @@ int iommu_enable_x2apic_IR(void)
 {
 dprintk(XENLOG_INFO VTDPREFIX,
 "Failed to enable Queued Invalidation!\n");
-return -1;
+return -EIO;
 }
 }
 
@@ -873,7 +873,7 @@ int iommu_enable_x2apic_IR(void)
 {
 dprintk(XENLOG_INFO VTDPREFIX,
 "Failed to enable Interrupt Remapping!\n");
-return -1;
+return -EIO;
 }
 }
 
--- unstable.orig/xen/include/asm-x86/iommu.h
+++ unstable/xen/include/asm-x86/iommu.h
@@ -27,7 +27,7 @@ int iommu_setup_hpet_msi(struct msi_desc
 /* While VT-d specific, this must get declared in a generic header. */
 int adjust_vtd_irq_affinities(void);
 void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int 
present);
-int iommu_supports_eim(void);
+bool_t iommu_supports_eim(void);
 int iommu_enable_x2apic_IR(void);
 void iommu_disable_x2apic_IR(void);
 


___
Xen-devel mailing l

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-10 Thread Jan Beulich
>>> On 09.09.15 at 19:02,  wrote:
> Well, now the system is booting but would you have any idea why the system
> only boots with iommu=verbose,debug but not with iommu=1? If I pass iommu=1
> as a boot parameter it fails detecting SATA drives (AHCI) with "ata1.00:
> failed command: IDENTIFY DEVICE" error. Also USB fails with fatal error,
> "ehci_hcd :xx:xxx:xx: HC died".

Short of logs thereof - no explanation other than the extra printing
possibly changing timing somewhere (albeit one would think that
while the printing happens timing shouldn't be that much of an issue).

> Also if I try to pass dom0pvh=1 it also crashes with iommu=verbose,debug -
> screen just goes completely black and system restarts itself - no way to
> see any error messages at all.

Likely a different problem, and definitely one where again we'd need
to see full logs of. You may want to try "noreboot" as a command line
option to see whether that makes the machine hang instead of reboot
(which would be an indication that proper logging will give us useful
insight).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-09 Thread Valtteri Kiviniemi
Hi,

I thought that some of those verbose logging notifications were errors like
for example "traps.c:2685:d0v10 Domain attempted WRMSR"

Well, now the system is booting but would you have any idea why the system
only boots with iommu=verbose,debug but not with iommu=1? If I pass iommu=1
as a boot parameter it fails detecting SATA drives (AHCI) with "ata1.00:
failed command: IDENTIFY DEVICE" error. Also USB fails with fatal error,
"ehci_hcd :xx:xxx:xx: HC died".

Also if I try to pass dom0pvh=1 it also crashes with iommu=verbose,debug -
screen just goes completely black and system restarts itself - no way to
see any error messages at all.

2015-09-09 11:56 GMT+03:00 Jan Beulich :

> >>> On 08.09.15 at 15:21,  wrote:
> > I got Xen booting now with VT-d enabled, lot's of errors and the system
> is
> > unstable. xl dmesg output:
>
> Lots of errors? I don't see much - can you point out the errors you see?
> There are two interesting things:
>
> > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg =
> 82c000201000
> > (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b
> >[...]
> > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg =
> 82c000203000
> > (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df
>
> Two IOMMUs with different capabilities. I'll have to look at what
> specifically one supports which the other doesn't, but I wouldn't
> be surprised if code wouldn't fully cope with that.
>
> > (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
>
> But later ...
>
> > (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
> > (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
> > (XEN) Intel VT-d Snoop Control not enabled.
> > (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> > (XEN) Intel VT-d Queued Invalidation enabled.
> > (XEN) Intel VT-d Interrupt Remapping enabled.
>
> ... and ...
>
> > (XEN) I/O virtualisation enabled
> > (XEN)  - Dom0 mode: Relaxed
> > (XEN) Interrupt remapping enabled
>
> I'll have to check what this inconsistency means.
>
> Apart from that the request for a native kernel log with IOMMU
> enabled still stands (for comparison as well as quirk identification
> purposes).
>
> Jan
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-09 Thread Jan Beulich
>>> On 08.09.15 at 15:21,  wrote:
> I got Xen booting now with VT-d enabled, lot's of errors and the system is
> unstable. xl dmesg output:

Lots of errors? I don't see much - can you point out the errors you see?
There are two interesting things:

> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = 
> 82c000201000
> (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b
>[...]
> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = 
> 82c000203000
> (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df

Two IOMMUs with different capabilities. I'll have to look at what
specifically one supports which the other doesn't, but I wouldn't
be surprised if code wouldn't fully cope with that.

> (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.

But later ...

> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.

... and ...

> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled

I'll have to check what this inconsistency means.

Apart from that the request for a native kernel log with IOMMU
enabled still stands (for comparison as well as quirk identification
purposes).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Valtteri Kiviniemi
Hi,

I got Xen booting now with VT-d enabled, lot's of errors and the system is
unstable. xl dmesg output:

 Xen 4.6.0-rc
(XEN) Xen version 4.6.0-rc (r...@easyanticheat.net) (gcc (Debian 4.9.2-10)
4.9.2) debug=y Mon Sep  7 20:46:47 EEST 2015
(XEN) Latest ChangeSet: Fri Aug 28 16:01:55 2015 +0100 hg:0b7f5657e0d1-dirty
(XEN) Console output is synchronous.
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=2048M dom0pvh=0 cpufreq=xen:performance
loglevel=all guest_loglevel=all iommu=verbose,debug sync_console
lapic=debug apic_verbosity=debug apic=debug
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)   - 0009e800 (usable)
(XEN)  0009e800 - 000a (reserved)
(XEN)  000e - 0010 (reserved)
(XEN)  0010 - b9bec000 (usable)
(XEN)  b9bec000 - ba761000 (reserved)
(XEN)  ba761000 - ba7af000 (ACPI data)
(XEN)  ba7af000 - bc9b3000 (ACPI NVS)
(XEN)  bc9b3000 - bd078000 (reserved)
(XEN)  bd078000 - bd079000 (usable)
(XEN)  bd079000 - bd0ff000 (reserved)
(XEN)  bd0ff000 - bd318000 (usable)
(XEN)  bd318000 - bdff9000 (reserved)
(XEN)  bdff9000 - be00 (usable)
(XEN)  e000 - f000 (reserved)
(XEN)  fed1c000 - fed2 (reserved)
(XEN)  ff00 - 0001 (reserved)
(XEN)  0001 - 00084000 (usable)
(XEN) ACPI: RSDP 000F0540, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT BA770098, 00A4 (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FACP BA79F6E8, 010C (r5 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: DSDT BA7701D0, 2F515 (r2 ALASKA   A M I   1072009 INTL 20091013)
(XEN) ACPI: FACS BC9AFF80, 0040
(XEN) ACPI: APIC BA79F7F8, 0100 (r3 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FPDT BA79F8F8, 0044 (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FIDT BA79F940, 009C (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: MCFG BA79F9E0, 003C (r1 ALASKAA M I  1072009 MSFT   97)
(XEN) ACPI: ASF! BA7AE218, 00A0 (r32 INTEL   HCG1 TFSMF4240)
(XEN) ACPI: UEFI BA79FA78, 0042 (r1 ALASKA   A M I   1072009 0)
(XEN) ACPI: BDAT BA79FAC0, 0030 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: HPET BA79FAF0, 0038 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: MSCT BA79FB28, 0090 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: PMCT BA79FBB8, 0064 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: SLIT BA79FC20, 002D (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: SRAT BA79FC50, 0E58 (r3 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: WDDT BA7A0AA8, 0040 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: SSDT BA7A0AE8, D647 (r1 ALASKAPmMgt1 INTL 20120913)
(XEN) ACPI: DMAR BA7AE130, 00E4 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) System RAM: 32669MB (33453740kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 6 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 8 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 10 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 7 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 9 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 11 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-c000
(XEN) SRAT: Node 0 PXM 0 1-84000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fd8b0
(XEN) DMI 2.7 present.
(XEN) APIC boot state is 'xapic'
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
bc9aff80/, using 32
(XEN) ACPI: wakeup_vec[bc9aff8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee0
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) Processor #4 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
(XEN) Processor #6 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
(XEN) Processor #8 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
(XEN) Processor #10 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) 

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Jan Beulich
>>> On 08.09.15 at 15:12,  wrote:
> Sorry about that, I'm using gmail and misclicked reply instead of
> reply-all. No Idea how to fix the line wrapping though.
> 
> Any ideas how to enable IOMMU in vanilla kernel? I checked from kernel
> documentation that the parameters are intel_iommu=on and iommu=force and
> they were passed as a boot parameters.

Yes, that's how it ought to work (and how it works for me). I suppose
you checked that your kernel configuration has CONFIG_INTEL_IOMMU=y?
In which case the most likely reason for the missing messages is that
your log level doesn't allow them to go to wherever you collect the log.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Valtteri Kiviniemi
Hi,

Sorry about that, I'm using gmail and misclicked reply instead of
reply-all. No Idea how to fix the line wrapping though.

Any ideas how to enable IOMMU in vanilla kernel? I checked from kernel
documentation that the parameters are intel_iommu=on and iommu=force and
they were passed as a boot parameters.

2015-09-08 15:58 GMT+03:00 Jan Beulich :

> >>> On 08.09.15 at 14:18,  wrote:
> > Well, getting a boot log with Xen seems to be really hard. Meanwhile,
> > here's a boot log from native 4.2.0 kernel with IOMMU enabled.
>
> There are a couple of issues here:
>
> - you sent this to me privately, instead of to the list (so others can also
> look at it)
> - your mailer line wrapped the log, which makes it hard to read/grep
> (better to attach the file if you can't fix your mail client)
> - there's no sign of any IOMMU initialization (there ought to be the
> result of
>
> pr_info("Intel(R) Virtualization Technology for Directed I/O\n");
>
> as well as a number of other info level messages (not sure whether
> that's a sign of your log level not being high enough, or some other
> setup failing).
>
> Jan
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Jan Beulich
>>> On 07.09.15 at 20:28,  wrote:
> On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote:
>>I'm happy to help to debug this further if needed, but so far I'm not able
>>to boot the system at all without iommu=0 boot parameter.
>> 
> 
> It'd be helpful if you could use a serial console (PCI serial card perhaps? 
> Or if your motherboard has a built-in IPMI/BMC/AMT management processor with 
> Serial-Over-LAN port that works aswell..) and capture a logfile from the full 
> Xen + Linux boot.. with all the verbose/debugging options enabled.

Indeed - without seeing a log of what goes wrong I don't think there's
much we can do. (One thing that might help is to post a native Linux
boot log with the IOMMU fully enabled, provided of course that one
boots up. This might provide hints at workarounds they have an place
but we don't.)

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel


Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-07 Thread Pasi Kärkkäinen
On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote:
>Hi,
> 

Hi,

>I'm using a Asus X99-A motherboard with latest bios which in paper (and in
>BIOS) supports VT-d. However, with Xen 4.5.1 and Xen unstable the VT-d
>fails completely on SATA AHCI with "failed to IDENTIFY".
> 
>I also tried to disable the SATA controller completely from BIOS and try
>to boot from USB stick (to debug further). However, also the USB-ports
>fail when using VT-d with error message "Fatal error, HC died" and then
>the kernel panics.
> 
>I googled around and tried the pci-phantom boot parameter for both the
>SATA and USB controllers and also the iommu=workaround_bios_bug but with
>no success.
> 
>The SATA controller that my system is using is:
> 
>00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA
>Controller [AHCI mode] (rev 05)
> 
>And the USB controllers:
> 
>00:14.0 USB controller: Intel Corporation Wellsburg USB xHCI Host
>Controller (rev 05)
>00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host
>Controller #1 (rev 05)
>07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
>Controller
> 
>And lastly: I installed ESXI 6.0 and it installed successfully with VT-d
>enabled. So I suspect that this might be a combination of BIOS/Xen bug
>that VMware has been able to solve (or then they do something
>differently).
> 
>I'm happy to help to debug this further if needed, but so far I'm not able
>to boot the system at all without iommu=0 boot parameter.
> 

It'd be helpful if you could use a serial console (PCI serial card perhaps? Or 
if your motherboard has a built-in IPMI/BMC/AMT management processor with 
Serial-Over-LAN port that works aswell..) and capture a logfile from the full 
Xen + Linux boot.. with all the verbose/debugging options enabled.

>Best regards,
>Valtteri Kiviniemi


-- Pasi


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel