Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)
On Wed, 2015-11-04 at 16:06 -0500, Konrad Rzeszutek Wilk wrote: > On Wed, Nov 04, 2015 at 02:49:11AM +0200, M. Ivanov wrote: > > Hello, > > > > I've experimented with my X10SAE and I think the problem with being > > unable to resume after suspending to RAM has something to do with the > > PCI Bridge violating the spec by trying to DMA from another address, > > since I got a DMAR error about DMA access on address 05:00.0(but in the > > bios event log it says Bus06), also the Tundra PCI Bridge is on address > > 04:00.0. (so like your case with address 7 and 8, but mine's 4 and 5), > > btw adding a PCI-E vga seems to change the addresses. > > > > > > When I disable IOMMU+Xen it works fine, so I mostly sure it's that. > > Though I've tried running just Linux with the iommu param on and I > > didn't get an error when sleeping/resuming. But I haven't tried doing a > > pass-through with it > > > > > > I've read in a previous thread about a patch of yours for the X10SAE > > problem. Which version of Xen can I use it on?(I am currently tinkering > > with 4.4.3-RELEASE). > > Oh my I can't remember. http://lists.xenproject.org/archives/html/xen-devel/2014-02/msg02037.html I've succeeded in compiling Xen 4.4.3-RELEASE with that patch and i've also loaded the kernel module you've provided(hack.c) but I've tried changing the address to 5 instead of 7, since my Tundra is on 4. But maybe I made some mistake since it didn't make a difference, still couldn't resume from sleep. > > > > Also I take it - I need to use hack.c to tell xen to create the fake > > device,(in my case 05:00.0) and to link it with 04:00.0? But how do I > > get that file to compile? Since I don't have a makefile/etc. for it. > > So.. that motherboard is a pain to work with. I found after numerous > emails to their technical support that the PCI chipset is not capable > of dealing with VT-d. That is PCI passhtrough of any PCI devices - nada. Well, as of my new bios 3.0 I haven't tried passing through devices. But it probably works, since I could do so in 2.0a(ethernet controllers and usb controller). The only problem I've experienced is not being able to resume after suspend. > > > > Also, can't I just disable the PCI Tundra bridge somehow? And what about > > phantom pci and the pciback-hide? Can they help? > > That would be nice. If all else fails, maybe I will try modifying the BIOS with a hex editor. > > > > I've read about problems regarding the Asmedia controller, so I've > > disabled it from the bios, but that didn't help at all. > > Lets take one problem at a time. The current issue you are seeing > is suspend/resume right? That is you just booted Xen + Linux > and ran 'pm-suspend'. And the motherboard did not resume from there? > > But it works OK with Linux? > > Is AMT enabled on your setup? Hm, I will try disabling it now(since it says Enabled in the bios), but AMT doesn't work anyway since I've updated my BIOS to 3.0 and it says in the Supermicro support docs that 3.0 doesn't support AMT even though the option is in the BIOS. Will buy a Supermicro serial cable these days. Annoying that I can't use a regular one. In the meantime, I don't get it why the Xen log isn't working. I'm using the log_lvl=all and guest_loglvl=all, yet I ain't getting anything in /var/log/xen. It did work once on Fedora 22 + Xen 4.5, atm on Fedora 23 + Xen 4.5 - doesn't work, nor on Fedora 21 + Xen 4.4. > > > > > > As for my Xen crash log > > > > DMAR:[DMA Write] Request device [05:00.0] fault addr Can't remember it > > DMAR:[fault reason 02] Present bit in context entry is clear > > > > Here is my tree: > > > > 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM > > Controller (rev 06) > > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core > > Processor PCI Express x16 Controller (rev 06) > > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 > > Processor Integrated Graphics Controller (rev 06) > > 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core > > Processor HD Audio Controller (rev 06) > > 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset > > Family USB xHCI (rev 05) > > 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series > > Chipset Family MEI Controller #1 (rev 04) > > 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series > > Chipset Family KT Controller (rev 04) > > 00:19.0 Ethernet
Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)
Hello, I've experimented with my X10SAE and I think the problem with being unable to resume after suspending to RAM has something to do with the PCI Bridge violating the spec by trying to DMA from another address, since I got a DMAR error about DMA access on address 05:00.0(but in the bios event log it says Bus06), also the Tundra PCI Bridge is on address 04:00.0. (so like your case with address 7 and 8, but mine's 4 and 5), btw adding a PCI-E vga seems to change the addresses. When I disable IOMMU+Xen it works fine, so I mostly sure it's that. Though I've tried running just Linux with the iommu param on and I didn't get an error when sleeping/resuming. But I haven't tried doing a pass-through with it I've read in a previous thread about a patch of yours for the X10SAE problem. Which version of Xen can I use it on?(I am currently tinkering with 4.4.3-RELEASE). Also I take it - I need to use hack.c to tell xen to create the fake device,(in my case 05:00.0) and to link it with 04:00.0? But how do I get that file to compile? Since I don't have a makefile/etc. for it. Also, can't I just disable the PCI Tundra bridge somehow? And what about phantom pci and the pciback-hide? Can they help? I've read about problems regarding the Asmedia controller, so I've disabled it from the bios, but that didn't help at all. As for my Xen crash log DMAR:[DMA Write] Request device [05:00.0] fault addr Can't remember it DMAR:[fault reason 02] Present bit in context entry is clear Here is my tree: 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) 00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 04:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) 05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] 06:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) signature.asc Description: This is a digitally signed message part ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)
On Fri, 2015-08-14 at 12:15 -0400, Konrad Rzeszutek Wilk wrote: > On Fri, Aug 14, 2015 at 05:08:32PM +0300, M. Ivanov wrote: > > On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote: > > > On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote: > > > > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote: > > > > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote: > > > > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote: > > > > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote: > > > > > > > > Hello, > > > > > > > > > > > > > > > > excuse me for bothering you, but I've read an old thread on a > > > > > > > > mailing > > > > > > > > list about X10SAE compatibility. > > > > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html > > > > > > > > > > > > > > CC-ing Xen devel. > > > > > > > > > > > > > > > > Currently I own this board and am trying to use it with Xen and > > > > > > > > be able > > > > > > > > to suspend and resume. > > > > > > > > > > > > > > > > But I am getting errors from the USB 3 Renesas controller about > > > > > > > > parity > > > > > > > > in my bios event log, and my system hangs on resume, > > > > > > > > so I was wondering if that is connected to the bridge(tundra) > > > > > > > > you've > > > > > > > > mentioned. > > > > > > > > > > > > > > Did you update the BIOS to the latest version? > > > > > > Will updating to version 3 solve my issue? > > > > > > Can you do a suspend/resume on your X10SAE? > > > > > > > > > > It did work at some point. I will find out when I am at home later > > > > > today. > > > > > > > > > Looking forward to your reply and am really thankful for your time, > > > > so far I've tried changing many of the settings in the bios, > > > > fiddling with Xen's kernel params, > > > > blacklisting the xhci driver, doing a xl detach. > > > > > > > > The only thing I haven't done yet is updating the bios, > > > > but Supermicro's support couldn't give me a changelog: > > > > > > > > "The primary objective for ver3.0 BIOS release is to support Intel > > > > Broadwell CPUs" > > > > "We do not know if BIOS update will fix the issue you are seeing as we > > > > never tested it with Xen." > > > > > > I did test it remotely and it did something very odd. It suspended and > > > then > > > immediately resumed with tons of VT-d errors!? > > > > > > I will try again but be actually right by it. > > Thanks for your effort, > > Can you suggest a way for me to log what is happening? > > I usually have an serial cable attached to it and log that. > Cranking up the debug on everything gives me some idea. Reasonable enough, will do. > > > > Since my machine just hangs up and I don't get a picture on the screen. > > Or when I do(sometimes) - it's just a cursor,(on a black screen with > > nothing else,no errors shown) and the machine doesn't react to any key > > combinations. > > Yeah. That is frustrating. > > > > On a side note: > > > > I did try updating the bios, > > but got some really strange result. > > The first time I got checksums about everything OK, > > (erase,flash,verify), > > but then it said - FDT is locked! > > And I am at least 90% sure I've enabled reflash in > > the BIOS setup prior to flashing. > > So I've restarted but didn't clear the CMOS(through the jumper on the > > board). And it said Bios v 3.0 in the setup, then I also did the > > Reset to optmized defaults. > > > > Tried suspending and couldn't resume like always. > > But perhaps currently my BIOS is in a broken/corrupted state. > > > > After that I've tried reflashing the bios again. But this time - > > after the messages about Erasing,Flashing,Verifying: > > it just froze. I didn't get any message about FDT, restarting or > > whatsoever. > > > > So
Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)
On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote: > On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote: > > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote: > > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote: > > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote: > > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote: > > > > > > Hello, > > > > > > > > > > > > excuse me for bothering you, but I've read an old thread on a > > > > > > mailing > > > > > > list about X10SAE compatibility. > > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html > > > > > > > > > > CC-ing Xen devel. > > > > > > > > > > > > Currently I own this board and am trying to use it with Xen and be > > > > > > able > > > > > > to suspend and resume. > > > > > > > > > > > > But I am getting errors from the USB 3 Renesas controller about > > > > > > parity > > > > > > in my bios event log, and my system hangs on resume, > > > > > > so I was wondering if that is connected to the bridge(tundra) you've > > > > > > mentioned. > > > > > > > > > > Did you update the BIOS to the latest version? > > > > Will updating to version 3 solve my issue? > > > > Can you do a suspend/resume on your X10SAE? > > > > > > It did work at some point. I will find out when I am at home later today. > > > > > Looking forward to your reply and am really thankful for your time, > > so far I've tried changing many of the settings in the bios, > > fiddling with Xen's kernel params, > > blacklisting the xhci driver, doing a xl detach. > > > > The only thing I haven't done yet is updating the bios, > > but Supermicro's support couldn't give me a changelog: > > > > "The primary objective for ver3.0 BIOS release is to support Intel > > Broadwell CPUs" > > "We do not know if BIOS update will fix the issue you are seeing as we > > never tested it with Xen." > > I did test it remotely and it did something very odd. It suspended and then > immediately resumed with tons of VT-d errors!? > > I will try again but be actually right by it. Thanks for your effort, Can you suggest a way for me to log what is happening? Since my machine just hangs up and I don't get a picture on the screen. Or when I do(sometimes) - it's just a cursor,(on a black screen with nothing else,no errors shown) and the machine doesn't react to any key combinations. On a side note: I did try updating the bios, but got some really strange result. The first time I got checksums about everything OK, (erase,flash,verify), but then it said - FDT is locked! And I am at least 90% sure I've enabled reflash in the BIOS setup prior to flashing. So I've restarted but didn't clear the CMOS(through the jumper on the board). And it said Bios v 3.0 in the setup, then I also did the Reset to optmized defaults. Tried suspending and couldn't resume like always. But perhaps currently my BIOS is in a broken/corrupted state. After that I've tried reflashing the bios again. But this time - after the messages about Erasing,Flashing,Verifying: it just froze. I didn't get any message about FDT, restarting or whatsoever. So I rebooted and it seems to work like before, says version 3.0 in the BIOS setup. I wonder if I should try clearing the CMOS and flashing again. > > > > > > > > > > > > I will be very glad if you could share any information regarding > > > > > > this > > > > > > matter. > > > > > > > > > > > > Best regards, > > > > > > M. Ivanov > > > > > > > > > > > > > > > > > > > > > > > > signature.asc Description: This is a digitally signed message part ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard
On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote: > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote: > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote: > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote: > > > > Hello, > > > > > > > > excuse me for bothering you, but I've read an old thread on a mailing > > > > list about X10SAE compatibility. > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html > > > > > > CC-ing Xen devel. > > > > > > > > Currently I own this board and am trying to use it with Xen and be able > > > > to suspend and resume. > > > > > > > > But I am getting errors from the USB 3 Renesas controller about parity > > > > in my bios event log, and my system hangs on resume, > > > > so I was wondering if that is connected to the bridge(tundra) you've > > > > mentioned. > > > > > > Did you update the BIOS to the latest version? > > Will updating to version 3 solve my issue? > > Can you do a suspend/resume on your X10SAE? > > It did work at some point. I will find out when I am at home later today. > Looking forward to your reply and am really thankful for your time, so far I've tried changing many of the settings in the bios, fiddling with Xen's kernel params, blacklisting the xhci driver, doing a xl detach. The only thing I haven't done yet is updating the bios, but Supermicro's support couldn't give me a changelog: "The primary objective for ver3.0 BIOS release is to support Intel Broadwell CPUs" "We do not know if BIOS update will fix the issue you are seeing as we never tested it with Xen." > > > > > > > > I will be very glad if you could share any information regarding this > > > > matter. > > > > > > > > Best regards, > > > > M. Ivanov > > > > > > > > > > signature.asc Description: This is a digitally signed message part ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] About Xen bridged pci devices and suspend/resume for the X10SAE motherboard
On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote: > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote: > > Hello, > > > > excuse me for bothering you, but I've read an old thread on a mailing > > list about X10SAE compatibility. > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html > > CC-ing Xen devel. > > > > Currently I own this board and am trying to use it with Xen and be able > > to suspend and resume. > > > > But I am getting errors from the USB 3 Renesas controller about parity > > in my bios event log, and my system hangs on resume, > > so I was wondering if that is connected to the bridge(tundra) you've > > mentioned. > > Did you update the BIOS to the latest version? Will updating to version 3 solve my issue? Can you do a suspend/resume on your X10SAE? > > > > I will be very glad if you could share any information regarding this > > matter. > > > > Best regards, > > M. Ivanov > > signature.asc Description: This is a digitally signed message part ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
