Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-25 Thread Jan Beulich 
>>> On 09.09.15 at 10:56,  wrote:
 On 08.09.15 at 15:21,  wrote:
>> I got Xen booting now with VT-d enabled, lot's of errors and the system is
>> unstable. xl dmesg output:
> 
> Lots of errors? I don't see much - can you point out the errors you see?
> There are two interesting things:
> 
>> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = 
>> 82c000201000
>> (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b
>>[...]
>> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = 
>> 82c000203000
>> (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df
> 
> Two IOMMUs with different capabilities. I'll have to look at what
> specifically one supports which the other doesn't, but I wouldn't
> be surprised if code wouldn't fully cope with that.

cap[40..42] represent the number of fault registers (afaict not an
issue if there's just one).

ecap[2] (device IOTLB) is relevant only when using ATS, which you
don't enable.

ecap[7] (snoop control) unavailable on any IOMMU will just lead
to it not getting used at all (equivalent to "iommu=no-snoop" on
the command line).

So all in all the feature differences are benign.

>> (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
> 
> But later ...
> 
>> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
>> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
>> (XEN) Intel VT-d Snoop Control not enabled.
>> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
>> (XEN) Intel VT-d Queued Invalidation enabled.
>> (XEN) Intel VT-d Interrupt Remapping enabled.
> 
> ... and ...
> 
>> (XEN) I/O virtualisation enabled
>> (XEN)  - Dom0 mode: Relaxed
>> (XEN) Interrupt remapping enabled
> 
> I'll have to check what this inconsistency means.

So from all I can tell this is bogus message wording when the firmware
requests the OS not to enabled x2APIC mode. Below a patch hopefully
making this less confusing.

Jan

--- unstable.orig/xen/arch/x86/apic.c
+++ unstable/xen/arch/x86/apic.c
@@ -943,8 +943,18 @@ void __init x2apic_bsp_setup(void)
 mask_8259A();
 mask_IO_APIC_setup(ioapic_entries);
 
-if ( iommu_enable_x2apic_IR() )
+switch ( iommu_enable_x2apic_IR() )
 {
+case 0:
+break;
+case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */
+if ( !x2apic_enabled )
+{
+printk("Not enabling x2APIC (upon firmware request)\n");
+goto restore_out;
+}
+/* fall through */
+default:
 if ( x2apic_enabled )
 panic("Interrupt remapping could not be enabled while "
   "x2APIC is already enabled by BIOS");
--- unstable.orig/xen/drivers/passthrough/vtd/intremap.c
+++ unstable/xen/drivers/passthrough/vtd/intremap.c
@@ -143,10 +143,10 @@ static void set_hpet_source_id(unsigned 
 set_ire_sid(ire, SVT_VERIFY_SID_SQ, SQ_13_IGNORE_3, hpetid_to_bdf(id));
 }
 
-int iommu_supports_eim(void)
+bool_t iommu_supports_eim(void)
 {
 struct acpi_drhd_unit *drhd;
-int apic;
+unsigned int apic;
 
 if ( !iommu_qinval || !iommu_intremap || list_empty(_drhd_units) )
 return 0;
@@ -154,12 +154,12 @@ int iommu_supports_eim(void)
 /* We MUST have a DRHD unit for each IOAPIC. */
 for ( apic = 0; apic < nr_ioapics; apic++ )
 if ( !ioapic_to_drhd(IO_APIC_ID(apic)) )
-{
+{
 dprintk(XENLOG_WARNING VTDPREFIX,
 "There is not a DRHD for IOAPIC %#x (id: %#x)!\n",
 apic, IO_APIC_ID(apic));
 return 0;
-}
+}
 
 for_each_drhd_unit ( drhd )
 if ( !ecap_queued_inval(drhd->iommu->ecap) ||
@@ -833,10 +833,10 @@ int iommu_enable_x2apic_IR(void)
 struct iommu *iommu;
 
 if ( !iommu_supports_eim() )
-return -1;
+return -EOPNOTSUPP;
 
 if ( !platform_supports_x2apic() )
-return -1;
+return -ENXIO;
 
 for_each_drhd_unit ( drhd )
 {
@@ -861,7 +861,7 @@ int iommu_enable_x2apic_IR(void)
 {
 dprintk(XENLOG_INFO VTDPREFIX,
 "Failed to enable Queued Invalidation!\n");
-return -1;
+return -EIO;
 }
 }
 
@@ -873,7 +873,7 @@ int iommu_enable_x2apic_IR(void)
 {
 dprintk(XENLOG_INFO VTDPREFIX,
 "Failed to enable Interrupt Remapping!\n");
-return -1;
+return -EIO;
 }
 }
 
--- unstable.orig/xen/include/asm-x86/iommu.h
+++ unstable/xen/include/asm-x86/iommu.h
@@ -27,7 +27,7 @@ int iommu_setup_hpet_msi(struct msi_desc
 /* While VT-d specific, this must get declared in a generic header. */
 int adjust_vtd_irq_affinities(void);
 void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int 
present);
-int iommu_supports_eim(void);
+bool_t iommu_supports_eim(void);
 int iommu_enable_x2apic_IR(void);
 void iommu_disable_x2apic_IR(void);

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-10 Thread Jan Beulich 
>>> On 09.09.15 at 19:02,  wrote:
> Well, now the system is booting but would you have any idea why the system
> only boots with iommu=verbose,debug but not with iommu=1? If I pass iommu=1
> as a boot parameter it fails detecting SATA drives (AHCI) with "ata1.00:
> failed command: IDENTIFY DEVICE" error. Also USB fails with fatal error,
> "ehci_hcd :xx:xxx:xx: HC died".

Short of logs thereof - no explanation other than the extra printing
possibly changing timing somewhere (albeit one would think that
while the printing happens timing shouldn't be that much of an issue).

> Also if I try to pass dom0pvh=1 it also crashes with iommu=verbose,debug -
> screen just goes completely black and system restarts itself - no way to
> see any error messages at all.

Likely a different problem, and definitely one where again we'd need
to see full logs of. You may want to try "noreboot" as a command line
option to see whether that makes the machine hang instead of reboot
(which would be an indication that proper logging will give us useful
insight).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-09 Thread Jan Beulich 
>>> On 08.09.15 at 15:21,  wrote:
> I got Xen booting now with VT-d enabled, lot's of errors and the system is
> unstable. xl dmesg output:

Lots of errors? I don't see much - can you point out the errors you see?
There are two interesting things:

> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = 
> 82c000201000
> (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b
>[...]
> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = 
> 82c000203000
> (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df

Two IOMMUs with different capabilities. I'll have to look at what
specifically one supports which the other doesn't, but I wouldn't
be surprised if code wouldn't fully cope with that.

> (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.

But later ...

> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
> (XEN) Intel VT-d Snoop Control not enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.

... and ...

> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled

I'll have to check what this inconsistency means.

Apart from that the request for a native kernel log with IOMMU
enabled still stands (for comparison as well as quirk identification
purposes).

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Jan Beulich 
>>> On 07.09.15 at 20:28,  wrote:
> On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote:
>>I'm happy to help to debug this further if needed, but so far I'm not able
>>to boot the system at all without iommu=0 boot parameter.
>> 
> 
> It'd be helpful if you could use a serial console (PCI serial card perhaps? 
> Or if your motherboard has a built-in IPMI/BMC/AMT management processor with 
> Serial-Over-LAN port that works aswell..) and capture a logfile from the full 
> Xen + Linux boot.. with all the verbose/debugging options enabled.

Indeed - without seeing a log of what goes wrong I don't think there's
much we can do. (One thing that might help is to post a native Linux
boot log with the IOMMU fully enabled, provided of course that one
boots up. This might provide hints at workarounds they have an place
but we don't.)

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Jan Beulich 
>>> On 08.09.15 at 15:12,  wrote:
> Sorry about that, I'm using gmail and misclicked reply instead of
> reply-all. No Idea how to fix the line wrapping though.
> 
> Any ideas how to enable IOMMU in vanilla kernel? I checked from kernel
> documentation that the parameters are intel_iommu=on and iommu=force and
> they were passed as a boot parameters.

Yes, that's how it ought to work (and how it works for me). I suppose
you checked that your kernel configuration has CONFIG_INTEL_IOMMU=y?
In which case the most likely reason for the missing messages is that
your log level doesn't allow them to go to wherever you collect the log.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-08 Thread Valtteri Kiviniemi 
Hi,

I got Xen booting now with VT-d enabled, lot's of errors and the system is
unstable. xl dmesg output:

 Xen 4.6.0-rc
(XEN) Xen version 4.6.0-rc (r...@easyanticheat.net) (gcc (Debian 4.9.2-10)
4.9.2) debug=y Mon Sep  7 20:46:47 EEST 2015
(XEN) Latest ChangeSet: Fri Aug 28 16:01:55 2015 +0100 hg:0b7f5657e0d1-dirty
(XEN) Console output is synchronous.
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=2048M dom0pvh=0 cpufreq=xen:performance
loglevel=all guest_loglevel=all iommu=verbose,debug sync_console
lapic=debug apic_verbosity=debug apic=debug
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)   - 0009e800 (usable)
(XEN)  0009e800 - 000a (reserved)
(XEN)  000e - 0010 (reserved)
(XEN)  0010 - b9bec000 (usable)
(XEN)  b9bec000 - ba761000 (reserved)
(XEN)  ba761000 - ba7af000 (ACPI data)
(XEN)  ba7af000 - bc9b3000 (ACPI NVS)
(XEN)  bc9b3000 - bd078000 (reserved)
(XEN)  bd078000 - bd079000 (usable)
(XEN)  bd079000 - bd0ff000 (reserved)
(XEN)  bd0ff000 - bd318000 (usable)
(XEN)  bd318000 - bdff9000 (reserved)
(XEN)  bdff9000 - be00 (usable)
(XEN)  e000 - f000 (reserved)
(XEN)  fed1c000 - fed2 (reserved)
(XEN)  ff00 - 0001 (reserved)
(XEN)  0001 - 00084000 (usable)
(XEN) ACPI: RSDP 000F0540, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT BA770098, 00A4 (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FACP BA79F6E8, 010C (r5 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: DSDT BA7701D0, 2F515 (r2 ALASKA   A M I   1072009 INTL 20091013)
(XEN) ACPI: FACS BC9AFF80, 0040
(XEN) ACPI: APIC BA79F7F8, 0100 (r3 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FPDT BA79F8F8, 0044 (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: FIDT BA79F940, 009C (r1 ALASKA   A M I   1072009 AMI 10013)
(XEN) ACPI: MCFG BA79F9E0, 003C (r1 ALASKAA M I  1072009 MSFT   97)
(XEN) ACPI: ASF! BA7AE218, 00A0 (r32 INTEL   HCG1 TFSMF4240)
(XEN) ACPI: UEFI BA79FA78, 0042 (r1 ALASKA   A M I   1072009 0)
(XEN) ACPI: BDAT BA79FAC0, 0030 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: HPET BA79FAF0, 0038 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: MSCT BA79FB28, 0090 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: PMCT BA79FBB8, 0064 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: SLIT BA79FC20, 002D (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: SRAT BA79FC50, 0E58 (r3 ALASKA   A M I 1 INTL 20091013)
(XEN) ACPI: WDDT BA7A0AA8, 0040 (r1 ALASKA   A M I 0 INTL 20091013)
(XEN) ACPI: SSDT BA7A0AE8, D647 (r1 ALASKAPmMgt1 INTL 20120913)
(XEN) ACPI: DMAR BA7AE130, 00E4 (r1 ALASKA   A M I 1 INTL 20091013)
(XEN) System RAM: 32669MB (33453740kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 6 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 8 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 10 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 7 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 9 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 11 -> Node 0
(XEN) SRAT: Node 0 PXM 0 0-c000
(XEN) SRAT: Node 0 PXM 0 1-84000
(XEN) NUMA: Using 20 for the hash shift.
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000fd8b0
(XEN) DMI 2.7 present.
(XEN) APIC boot state is 'xapic'
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
bc9aff80/, using 32
(XEN) ACPI: wakeup_vec[bc9aff8c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee0
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) Processor #4 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
(XEN) Processor #6 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled)
(XEN) Processor #8 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled)
(XEN) Processor #10 7:15 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN)

[Xen-devel] Asus X99-A VT-d problems

2015-09-07 Thread Valtteri Kiviniemi 
Hi,

I'm using a Asus X99-A motherboard with latest bios which in paper (and in
BIOS) supports VT-d. However, with Xen 4.5.1 and Xen unstable the VT-d
fails completely on SATA AHCI with "failed to IDENTIFY".


I also tried to disable the SATA controller completely from BIOS and try to
boot from USB stick (to debug further). However, also the USB-ports fail
when using VT-d with error message "Fatal error, HC died" and then the
kernel panics.

I googled around and tried the pci-phantom boot parameter for both the SATA
and USB controllers and also the iommu=workaround_bios_bug but with no
success.

The SATA controller that my system is using is:

00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA Controller
[AHCI mode] (rev 05)

And the USB controllers:

00:14.0 USB controller: Intel Corporation Wellsburg USB xHCI Host
Controller (rev 05)
00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host
Controller #1 (rev 05)
07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
Controller

And lastly: I installed ESXI 6.0 and it installed successfully with VT-d
enabled. So I suspect that this might be a combination of BIOS/Xen bug that
VMware has been able to solve (or then they do something differently).

I'm happy to help to debug this further if needed, but so far I'm not able
to boot the system at all without iommu=0 boot parameter.

Best regards,
Valtteri Kiviniemi
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Asus X99-A VT-d problems

2015-09-07 Thread Pasi Kärkkäinen 
On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote:
>Hi,
> 

Hi,

>I'm using a Asus X99-A motherboard with latest bios which in paper (and in
>BIOS) supports VT-d. However, with Xen 4.5.1 and Xen unstable the VT-d
>fails completely on SATA AHCI with "failed to IDENTIFY".
> 
>I also tried to disable the SATA controller completely from BIOS and try
>to boot from USB stick (to debug further). However, also the USB-ports
>fail when using VT-d with error message "Fatal error, HC died" and then
>the kernel panics.
> 
>I googled around and tried the pci-phantom boot parameter for both the
>SATA and USB controllers and also the iommu=workaround_bios_bug but with
>no success.
> 
>The SATA controller that my system is using is:
> 
>00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA
>Controller [AHCI mode] (rev 05)
> 
>And the USB controllers:
> 
>00:14.0 USB controller: Intel Corporation Wellsburg USB xHCI Host
>Controller (rev 05)
>00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host
>Controller #1 (rev 05)
>07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host
>Controller
> 
>And lastly: I installed ESXI 6.0 and it installed successfully with VT-d
>enabled. So I suspect that this might be a combination of BIOS/Xen bug
>that VMware has been able to solve (or then they do something
>differently).
> 
>I'm happy to help to debug this further if needed, but so far I'm not able
>to boot the system at all without iommu=0 boot parameter.
> 

It'd be helpful if you could use a serial console (PCI serial card perhaps? Or 
if your motherboard has a built-in IPMI/BMC/AMT management processor with 
Serial-Over-LAN port that works aswell..) and capture a logfile from the full 
Xen + Linux boot.. with all the verbose/debugging options enabled.

>Best regards,
>Valtteri Kiviniemi


-- Pasi


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel