Re: [Xen-devel] Asus X99-A VT-d problems
>>> On 09.09.15 at 10:56, wrote: On 08.09.15 at 15:21, wrote: >> I got Xen booting now with VT-d enabled, lot's of errors and the system is >> unstable. xl dmesg output: > > Lots of errors? I don't see much - can you point out the errors you see? > There are two interesting things: > >> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = >> 82c000201000 >> (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b >>[...] >> (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = >> 82c000203000 >> (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df > > Two IOMMUs with different capabilities. I'll have to look at what > specifically one supports which the other doesn't, but I wouldn't > be surprised if code wouldn't fully cope with that. cap[40..42] represent the number of fault registers (afaict not an issue if there's just one). ecap[2] (device IOTLB) is relevant only when using ATS, which you don't enable. ecap[7] (snoop control) unavailable on any IOMMU will just lead to it not getting used at all (equivalent to "iommu=no-snoop" on the command line). So all in all the feature differences are benign. >> (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC. > > But later ... > >> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB. >> (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB. >> (XEN) Intel VT-d Snoop Control not enabled. >> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. >> (XEN) Intel VT-d Queued Invalidation enabled. >> (XEN) Intel VT-d Interrupt Remapping enabled. > > ... and ... > >> (XEN) I/O virtualisation enabled >> (XEN) - Dom0 mode: Relaxed >> (XEN) Interrupt remapping enabled > > I'll have to check what this inconsistency means. So from all I can tell this is bogus message wording when the firmware requests the OS not to enabled x2APIC mode. Below a patch hopefully making this less confusing. Jan --- unstable.orig/xen/arch/x86/apic.c +++ unstable/xen/arch/x86/apic.c @@ -943,8 +943,18 @@ void __init x2apic_bsp_setup(void) mask_8259A(); mask_IO_APIC_setup(ioapic_entries); -if ( iommu_enable_x2apic_IR() ) +switch ( iommu_enable_x2apic_IR() ) { +case 0: +break; +case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */ +if ( !x2apic_enabled ) +{ +printk("Not enabling x2APIC (upon firmware request)\n"); +goto restore_out; +} +/* fall through */ +default: if ( x2apic_enabled ) panic("Interrupt remapping could not be enabled while " "x2APIC is already enabled by BIOS"); --- unstable.orig/xen/drivers/passthrough/vtd/intremap.c +++ unstable/xen/drivers/passthrough/vtd/intremap.c @@ -143,10 +143,10 @@ static void set_hpet_source_id(unsigned set_ire_sid(ire, SVT_VERIFY_SID_SQ, SQ_13_IGNORE_3, hpetid_to_bdf(id)); } -int iommu_supports_eim(void) +bool_t iommu_supports_eim(void) { struct acpi_drhd_unit *drhd; -int apic; +unsigned int apic; if ( !iommu_qinval || !iommu_intremap || list_empty(&acpi_drhd_units) ) return 0; @@ -154,12 +154,12 @@ int iommu_supports_eim(void) /* We MUST have a DRHD unit for each IOAPIC. */ for ( apic = 0; apic < nr_ioapics; apic++ ) if ( !ioapic_to_drhd(IO_APIC_ID(apic)) ) -{ +{ dprintk(XENLOG_WARNING VTDPREFIX, "There is not a DRHD for IOAPIC %#x (id: %#x)!\n", apic, IO_APIC_ID(apic)); return 0; -} +} for_each_drhd_unit ( drhd ) if ( !ecap_queued_inval(drhd->iommu->ecap) || @@ -833,10 +833,10 @@ int iommu_enable_x2apic_IR(void) struct iommu *iommu; if ( !iommu_supports_eim() ) -return -1; +return -EOPNOTSUPP; if ( !platform_supports_x2apic() ) -return -1; +return -ENXIO; for_each_drhd_unit ( drhd ) { @@ -861,7 +861,7 @@ int iommu_enable_x2apic_IR(void) { dprintk(XENLOG_INFO VTDPREFIX, "Failed to enable Queued Invalidation!\n"); -return -1; +return -EIO; } } @@ -873,7 +873,7 @@ int iommu_enable_x2apic_IR(void) { dprintk(XENLOG_INFO VTDPREFIX, "Failed to enable Interrupt Remapping!\n"); -return -1; +return -EIO; } } --- unstable.orig/xen/include/asm-x86/iommu.h +++ unstable/xen/include/asm-x86/iommu.h @@ -27,7 +27,7 @@ int iommu_setup_hpet_msi(struct msi_desc /* While VT-d specific, this must get declared in a generic header. */ int adjust_vtd_irq_affinities(void); void iommu_pte_flush(struct domain *d, u64 gfn, u64 *pte, int order, int present); -int iommu_supports_eim(void); +bool_t iommu_supports_eim(void); int iommu_enable_x2apic_IR(void); void iommu_disable_x2apic_IR(void); ___ Xen-devel mailing l
Re: [Xen-devel] Asus X99-A VT-d problems
>>> On 09.09.15 at 19:02, wrote: > Well, now the system is booting but would you have any idea why the system > only boots with iommu=verbose,debug but not with iommu=1? If I pass iommu=1 > as a boot parameter it fails detecting SATA drives (AHCI) with "ata1.00: > failed command: IDENTIFY DEVICE" error. Also USB fails with fatal error, > "ehci_hcd :xx:xxx:xx: HC died". Short of logs thereof - no explanation other than the extra printing possibly changing timing somewhere (albeit one would think that while the printing happens timing shouldn't be that much of an issue). > Also if I try to pass dom0pvh=1 it also crashes with iommu=verbose,debug - > screen just goes completely black and system restarts itself - no way to > see any error messages at all. Likely a different problem, and definitely one where again we'd need to see full logs of. You may want to try "noreboot" as a command line option to see whether that makes the machine hang instead of reboot (which would be an indication that proper logging will give us useful insight). Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
Hi, I thought that some of those verbose logging notifications were errors like for example "traps.c:2685:d0v10 Domain attempted WRMSR" Well, now the system is booting but would you have any idea why the system only boots with iommu=verbose,debug but not with iommu=1? If I pass iommu=1 as a boot parameter it fails detecting SATA drives (AHCI) with "ata1.00: failed command: IDENTIFY DEVICE" error. Also USB fails with fatal error, "ehci_hcd :xx:xxx:xx: HC died". Also if I try to pass dom0pvh=1 it also crashes with iommu=verbose,debug - screen just goes completely black and system restarts itself - no way to see any error messages at all. 2015-09-09 11:56 GMT+03:00 Jan Beulich : > >>> On 08.09.15 at 15:21, wrote: > > I got Xen booting now with VT-d enabled, lot's of errors and the system > is > > unstable. xl dmesg output: > > Lots of errors? I don't see much - can you point out the errors you see? > There are two interesting things: > > > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = > 82c000201000 > > (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b > >[...] > > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = > 82c000203000 > > (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df > > Two IOMMUs with different capabilities. I'll have to look at what > specifically one supports which the other doesn't, but I wouldn't > be surprised if code wouldn't fully cope with that. > > > (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC. > > But later ... > > > (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB. > > (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB. > > (XEN) Intel VT-d Snoop Control not enabled. > > (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. > > (XEN) Intel VT-d Queued Invalidation enabled. > > (XEN) Intel VT-d Interrupt Remapping enabled. > > ... and ... > > > (XEN) I/O virtualisation enabled > > (XEN) - Dom0 mode: Relaxed > > (XEN) Interrupt remapping enabled > > I'll have to check what this inconsistency means. > > Apart from that the request for a native kernel log with IOMMU > enabled still stands (for comparison as well as quirk identification > purposes). > > Jan > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
>>> On 08.09.15 at 15:21, wrote: > I got Xen booting now with VT-d enabled, lot's of errors and the system is > unstable. xl dmesg output: Lots of errors? I don't see much - can you point out the errors you see? There are two interesting things: > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffd000 iommu->reg = > 82c000201000 > (XEN) [VT-D]iommu.c:1151: cap = d2008c10ef0466 ecap = f0205b >[...] > (XEN) [VT-D]iommu.c:1149: drhd->address = fbffc000 iommu->reg = > 82c000203000 > (XEN) [VT-D]iommu.c:1151: cap = d2078c106f0466 ecap = f020df Two IOMMUs with different capabilities. I'll have to look at what specifically one supports which the other doesn't, but I wouldn't be surprised if code wouldn't fully cope with that. > (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC. But later ... > (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB. > (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB. > (XEN) Intel VT-d Snoop Control not enabled. > (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. > (XEN) Intel VT-d Queued Invalidation enabled. > (XEN) Intel VT-d Interrupt Remapping enabled. ... and ... > (XEN) I/O virtualisation enabled > (XEN) - Dom0 mode: Relaxed > (XEN) Interrupt remapping enabled I'll have to check what this inconsistency means. Apart from that the request for a native kernel log with IOMMU enabled still stands (for comparison as well as quirk identification purposes). Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
Hi, I got Xen booting now with VT-d enabled, lot's of errors and the system is unstable. xl dmesg output: Xen 4.6.0-rc (XEN) Xen version 4.6.0-rc (r...@easyanticheat.net) (gcc (Debian 4.9.2-10) 4.9.2) debug=y Mon Sep 7 20:46:47 EEST 2015 (XEN) Latest ChangeSet: Fri Aug 28 16:01:55 2015 +0100 hg:0b7f5657e0d1-dirty (XEN) Console output is synchronous. (XEN) Bootloader: GNU GRUB 0.97 (XEN) Command line: dom0_mem=2048M dom0pvh=0 cpufreq=xen:performance loglevel=all guest_loglevel=all iommu=verbose,debug sync_console lapic=debug apic_verbosity=debug apic=debug (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds (XEN) EDID info not retrieved because no DDC retrieval method detected (XEN) Disc information: (XEN) Found 3 MBR signatures (XEN) Found 4 EDD information structures (XEN) Xen-e820 RAM map: (XEN) - 0009e800 (usable) (XEN) 0009e800 - 000a (reserved) (XEN) 000e - 0010 (reserved) (XEN) 0010 - b9bec000 (usable) (XEN) b9bec000 - ba761000 (reserved) (XEN) ba761000 - ba7af000 (ACPI data) (XEN) ba7af000 - bc9b3000 (ACPI NVS) (XEN) bc9b3000 - bd078000 (reserved) (XEN) bd078000 - bd079000 (usable) (XEN) bd079000 - bd0ff000 (reserved) (XEN) bd0ff000 - bd318000 (usable) (XEN) bd318000 - bdff9000 (reserved) (XEN) bdff9000 - be00 (usable) (XEN) e000 - f000 (reserved) (XEN) fed1c000 - fed2 (reserved) (XEN) ff00 - 0001 (reserved) (XEN) 0001 - 00084000 (usable) (XEN) ACPI: RSDP 000F0540, 0024 (r2 ALASKA) (XEN) ACPI: XSDT BA770098, 00A4 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FACP BA79F6E8, 010C (r5 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: DSDT BA7701D0, 2F515 (r2 ALASKA A M I 1072009 INTL 20091013) (XEN) ACPI: FACS BC9AFF80, 0040 (XEN) ACPI: APIC BA79F7F8, 0100 (r3 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FPDT BA79F8F8, 0044 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FIDT BA79F940, 009C (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: MCFG BA79F9E0, 003C (r1 ALASKAA M I 1072009 MSFT 97) (XEN) ACPI: ASF! BA7AE218, 00A0 (r32 INTEL HCG1 TFSMF4240) (XEN) ACPI: UEFI BA79FA78, 0042 (r1 ALASKA A M I 1072009 0) (XEN) ACPI: BDAT BA79FAC0, 0030 (r1 ALASKA A M I 0 INTL 20091013) (XEN) ACPI: HPET BA79FAF0, 0038 (r1 ALASKA A M I 1 INTL 20091013) (XEN) ACPI: MSCT BA79FB28, 0090 (r1 ALASKA A M I 1 INTL 20091013) (XEN) ACPI: PMCT BA79FBB8, 0064 (r1 ALASKA A M I 0 INTL 20091013) (XEN) ACPI: SLIT BA79FC20, 002D (r1 ALASKA A M I 1 INTL 20091013) (XEN) ACPI: SRAT BA79FC50, 0E58 (r3 ALASKA A M I 1 INTL 20091013) (XEN) ACPI: WDDT BA7A0AA8, 0040 (r1 ALASKA A M I 0 INTL 20091013) (XEN) ACPI: SSDT BA7A0AE8, D647 (r1 ALASKAPmMgt1 INTL 20120913) (XEN) ACPI: DMAR BA7AE130, 00E4 (r1 ALASKA A M I 1 INTL 20091013) (XEN) System RAM: 32669MB (33453740kB) (XEN) SRAT: PXM 0 -> APIC 0 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 2 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 4 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 6 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 8 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 10 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 1 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 3 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 5 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 7 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 9 -> Node 0 (XEN) SRAT: PXM 0 -> APIC 11 -> Node 0 (XEN) SRAT: Node 0 PXM 0 0-c000 (XEN) SRAT: Node 0 PXM 0 1-84000 (XEN) NUMA: Using 20 for the hash shift. (XEN) Domain heap initialised (XEN) found SMP MP-table at 000fd8b0 (XEN) DMI 2.7 present. (XEN) APIC boot state is 'xapic' (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0x408 (XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0] (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:404,1:0], pm1x_evt[1:400,1:0] (XEN) ACPI: 32/64X FACS address mismatch in FADT - bc9aff80/, using 32 (XEN) ACPI: wakeup_vec[bc9aff8c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee0 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) (XEN) Processor #2 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) (XEN) Processor #4 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) (XEN) Processor #6 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) (XEN) Processor #8 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled) (XEN) Processor #10 7:15 APIC version 21 (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) (XEN)
Re: [Xen-devel] Asus X99-A VT-d problems
>>> On 08.09.15 at 15:12, wrote: > Sorry about that, I'm using gmail and misclicked reply instead of > reply-all. No Idea how to fix the line wrapping though. > > Any ideas how to enable IOMMU in vanilla kernel? I checked from kernel > documentation that the parameters are intel_iommu=on and iommu=force and > they were passed as a boot parameters. Yes, that's how it ought to work (and how it works for me). I suppose you checked that your kernel configuration has CONFIG_INTEL_IOMMU=y? In which case the most likely reason for the missing messages is that your log level doesn't allow them to go to wherever you collect the log. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
Hi, Sorry about that, I'm using gmail and misclicked reply instead of reply-all. No Idea how to fix the line wrapping though. Any ideas how to enable IOMMU in vanilla kernel? I checked from kernel documentation that the parameters are intel_iommu=on and iommu=force and they were passed as a boot parameters. 2015-09-08 15:58 GMT+03:00 Jan Beulich : > >>> On 08.09.15 at 14:18, wrote: > > Well, getting a boot log with Xen seems to be really hard. Meanwhile, > > here's a boot log from native 4.2.0 kernel with IOMMU enabled. > > There are a couple of issues here: > > - you sent this to me privately, instead of to the list (so others can also > look at it) > - your mailer line wrapped the log, which makes it hard to read/grep > (better to attach the file if you can't fix your mail client) > - there's no sign of any IOMMU initialization (there ought to be the > result of > > pr_info("Intel(R) Virtualization Technology for Directed I/O\n"); > > as well as a number of other info level messages (not sure whether > that's a sign of your log level not being high enough, or some other > setup failing). > > Jan > > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
>>> On 07.09.15 at 20:28, wrote: > On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote: >>I'm happy to help to debug this further if needed, but so far I'm not able >>to boot the system at all without iommu=0 boot parameter. >> > > It'd be helpful if you could use a serial console (PCI serial card perhaps? > Or if your motherboard has a built-in IPMI/BMC/AMT management processor with > Serial-Over-LAN port that works aswell..) and capture a logfile from the full > Xen + Linux boot.. with all the verbose/debugging options enabled. Indeed - without seeing a log of what goes wrong I don't think there's much we can do. (One thing that might help is to post a native Linux boot log with the IOMMU fully enabled, provided of course that one boots up. This might provide hints at workarounds they have an place but we don't.) Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Asus X99-A VT-d problems
On Mon, Sep 07, 2015 at 08:45:20PM +0300, Valtteri Kiviniemi wrote: >Hi, > Hi, >I'm using a Asus X99-A motherboard with latest bios which in paper (and in >BIOS) supports VT-d. However, with Xen 4.5.1 and Xen unstable the VT-d >fails completely on SATA AHCI with "failed to IDENTIFY". > >I also tried to disable the SATA controller completely from BIOS and try >to boot from USB stick (to debug further). However, also the USB-ports >fail when using VT-d with error message "Fatal error, HC died" and then >the kernel panics. > >I googled around and tried the pci-phantom boot parameter for both the >SATA and USB controllers and also the iommu=workaround_bios_bug but with >no success. > >The SATA controller that my system is using is: > >00:1f.2 SATA controller: Intel Corporation Wellsburg 6-Port SATA >Controller [AHCI mode] (rev 05) > >And the USB controllers: > >00:14.0 USB controller: Intel Corporation Wellsburg USB xHCI Host >Controller (rev 05) >00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host >Controller #1 (rev 05) >07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host >Controller > >And lastly: I installed ESXI 6.0 and it installed successfully with VT-d >enabled. So I suspect that this might be a combination of BIOS/Xen bug >that VMware has been able to solve (or then they do something >differently). > >I'm happy to help to debug this further if needed, but so far I'm not able >to boot the system at all without iommu=0 boot parameter. > It'd be helpful if you could use a serial console (PCI serial card perhaps? Or if your motherboard has a built-in IPMI/BMC/AMT management processor with Serial-Over-LAN port that works aswell..) and capture a logfile from the full Xen + Linux boot.. with all the verbose/debugging options enabled. >Best regards, >Valtteri Kiviniemi -- Pasi ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel