Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 02:01:15PM -0600, Venu Busireddy wrote: > On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote: > > On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: > > > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk > > > wrote: > > > > . snip.. > > > >> >> > Given this commit is pretty old, I'm also curious why it's only > > > >> >> > reported > > > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 > > > >> >> > happens > > > >> >> > to be the one upon which you first tried iommu=0 on a platform > > > >> >> > supporting > > > >> >> > interrupt remapping? > > > >> >> > > > >> >> It just happens to be the one I tried. VT-d was spamming my console > > > >> >> with faults like this: > > > >> >> > > > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault > > > >> >> addr > > > >> >> 277e28000, iommu reg = 82c000201000 > > > 82c000201000 > > > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > > > >> >> > > > >> >> So I figured I can just turn it off to clean up my console. I still > > > >> >> don't know what these VT-d faults are about.. > > > >> > > > > >> > What is the 0:02.0 device? Is it being passed in to your guest? > > > >> > > > >> Nope, there is no pass-through to any guests. The device is: > > > >> > > > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core > > > >> Processor Family Integrated Graphics Controller (rev 09) > > > > > > > > Let me guess, you have an SandyBridge motherboard and were using > > > > Intel AMT? > > > > > > Correct. > > > > > > > > > > > I see those all the time on that box (and I think my Haswell > > > > one too). The best I could narrow it down was that the card decided that > > > > certain area should be in the RMRR regions. With Venu/Elen's patch > > > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able > > > > to provide an rmrr paramater to include these addresses. > > > > > > Some more information on how to use that flag would be valuable.. I > > > tried "rmrr=27000-28000=00:02.0" to no avail and I really have > > > no idea what I'm doing here =) > > > > CC-ing Elena and Venu who I hope can help you with this. > > > > You may want to provide the full 'xl dmesg' as the 'rmrr' should have > > outputted some details.. > > Tamas, > > While 'xl dmesg' and the output from the serial console will certainly > help, it appears that you may have already provided the info needed. There > may be other devices than 0:02.0 that have problems, but let us assume > that it is the only device. > > Your change to xen command line is incorrect. The correct option to add > to the xen command line is: > > rmrr=0x277e28=0:00:02.0 Could you please improve the documentation by mentioning that 'start' and 'end' should be frame numbers instead of full memory addresses? I also stumbled upon this and had to go look at the code in order to figure out what I should pass to rmrr. Also a couple of examples about it's utilization wouldn't hurt IMHO. Roger. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
> From: Jan Beulich [mailto:jbeul...@suse.com] > Sent: Friday, February 17, 2017 4:21 PM > > I think the commit description is pretty clear. Especially this part > > "This could have the nice side effect of allowing to use "iommu=off" > even when x2APIC was pre-enabled by the BIOS (in which case interrupt > remapping is a requirement, but DMA translation [obviously] isn't), but > that doesn't currently work (and hence x2apic_bsp_setup() forces the > IOMMU on rather than just interrupt remapping)." > > is quite relevant in the context here. Just like Linux, we really ought > to have a way to run with interrupt remapping, but without DMA > remapping. So I think this old commit was a half-hearted step into > that direction, not recognizing that it would break some other case. > > So I think the only reasonable way forward to address Tamas's > issue is to fully disentangle DMA and interrupt remapping setup, > which I'm sure is going to take some time. As a minimal adjustment, > though, I wonder whether that old commit's adjustment to apic.c > shouldn't rather have set force_intremap. Tamas - could you check > whether > > --- a/xen/arch/x86/apic.c > +++ b/xen/arch/x86/apic.c > @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void) > goto restore_out; > } > > -force_iommu = 1; > +force_intremap = 1; > > genapic = apic_x2apic_probe(); > printk("Switched to APIC driver %s.\n", genapic->name); > > makes things any better? > > Jan Looks there were some conflicting commits moving the policy back-and-forth. For example, your another commit in 2013 changed back to the old way that iommu_intremap depends on iommu_enable: if ( iommu_enable ) { rc = iommu_hardware_setup(); iommu_enabled = (rc == 0); } (added by fae03721) if ( !iommu_enabled ) iommu_intremap = 0; if ( (force_iommu && !iommu_enabled) || (force_intremap && !iommu_intremap) ) panic("Couldn't enable %s and iommu=required/force", !iommu_enabled ? "IOMMU" : "Interrupt Remapping"); But I cannot understand why Tamas doesn't hit above panic again... Thanks Kevin ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 2:31 PM, Andrew Cooper wrote: > On 17/02/17 21:28, Tamas K Lengyel wrote: >> On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy >> wrote: >>> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote: On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk > wrote: >> . snip.. >> Given this commit is pretty old, I'm also curious why it's only >> reported >> on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens >> to be the one upon which you first tried iommu=0 on a platform >> supporting >> interrupt remapping? > It just happens to be the one I tried. VT-d was spamming my console > with faults like this: > > (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr > 277e28000, iommu reg = 82c000201000 > 82c000201000 > (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > > So I figured I can just turn it off to clean up my console. I still > don't know what these VT-d faults are about.. What is the 0:02.0 device? Is it being passed in to your guest? >>> Nope, there is no pass-through to any guests. The device is: >>> >>> 00:02.0 Display controller: Intel Corporation 2nd Generation Core >>> Processor Family Integrated Graphics Controller (rev 09) >> Let me guess, you have an SandyBridge motherboard and were using >> Intel AMT? > Correct. > >> I see those all the time on that box (and I think my Haswell >> one too). The best I could narrow it down was that the card decided that >> certain area should be in the RMRR regions. With Venu/Elen's patch >> in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able >> to provide an rmrr paramater to include these addresses. > Some more information on how to use that flag would be valuable.. I > tried "rmrr=27000-28000=00:02.0" to no avail and I really have > no idea what I'm doing here =) CC-ing Elena and Venu who I hope can help you with this. You may want to provide the full 'xl dmesg' as the 'rmrr' should have outputted some details.. >>> Tamas, >>> >>> While 'xl dmesg' and the output from the serial console will certainly >>> help, it appears that you may have already provided the info needed. There >>> may be other devices than 0:02.0 that have problems, but let us assume >>> that it is the only device. >>> >>> Your change to xen command line is incorrect. The correct option to add >>> to the xen command line is: >>> >>> rmrr=0x277e28=0:00:02.0 >> I see, so it has to be a page and not an address.. OTOH the problem >> with specifying 0x277e28 is that the fault address seem to change >> every time I reboot the machine. I'm not sure where that range is >> coming from but it always seems to be between 27000-28000. > > Looking at the parsing code, ranges work fine but you need to use 0x for > hex addresses, or the address will be interpreted as decimal. > Ah, yes, makes sense =) Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On 17/02/17 21:28, Tamas K Lengyel wrote: > On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy > wrote: >> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote: >>> On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk wrote: > . snip.. > Given this commit is pretty old, I'm also curious why it's only > reported > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > to be the one upon which you first tried iommu=0 on a platform > supporting > interrupt remapping? It just happens to be the one I tried. VT-d was spamming my console with faults like this: (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr 277e28000, iommu reg = 82c000201000 82c000201000 (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set So I figured I can just turn it off to clean up my console. I still don't know what these VT-d faults are about.. >>> What is the 0:02.0 device? Is it being passed in to your guest? >> Nope, there is no pass-through to any guests. The device is: >> >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core >> Processor Family Integrated Graphics Controller (rev 09) > Let me guess, you have an SandyBridge motherboard and were using > Intel AMT? Correct. > I see those all the time on that box (and I think my Haswell > one too). The best I could narrow it down was that the card decided that > certain area should be in the RMRR regions. With Venu/Elen's patch > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able > to provide an rmrr paramater to include these addresses. Some more information on how to use that flag would be valuable.. I tried "rmrr=27000-28000=00:02.0" to no avail and I really have no idea what I'm doing here =) >>> CC-ing Elena and Venu who I hope can help you with this. >>> >>> You may want to provide the full 'xl dmesg' as the 'rmrr' should have >>> outputted some details.. >> Tamas, >> >> While 'xl dmesg' and the output from the serial console will certainly >> help, it appears that you may have already provided the info needed. There >> may be other devices than 0:02.0 that have problems, but let us assume >> that it is the only device. >> >> Your change to xen command line is incorrect. The correct option to add >> to the xen command line is: >> >> rmrr=0x277e28=0:00:02.0 > I see, so it has to be a page and not an address.. OTOH the problem > with specifying 0x277e28 is that the fault address seem to change > every time I reboot the machine. I'm not sure where that range is > coming from but it always seems to be between 27000-28000. Looking at the parsing code, ranges work fine but you need to use 0x for hex addresses, or the address will be interpreted as decimal. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy wrote: > On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote: >> On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: >> > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk >> > wrote: >> > > . snip.. >> > >> >> > Given this commit is pretty old, I'm also curious why it's only >> > >> >> > reported >> > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens >> > >> >> > to be the one upon which you first tried iommu=0 on a platform >> > >> >> > supporting >> > >> >> > interrupt remapping? >> > >> >> >> > >> >> It just happens to be the one I tried. VT-d was spamming my console >> > >> >> with faults like this: >> > >> >> >> > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr >> > >> >> 277e28000, iommu reg = 82c000201000 >> > 82c000201000 >> > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set >> > >> >> >> > >> >> So I figured I can just turn it off to clean up my console. I still >> > >> >> don't know what these VT-d faults are about.. >> > >> > >> > >> > What is the 0:02.0 device? Is it being passed in to your guest? >> > >> >> > >> Nope, there is no pass-through to any guests. The device is: >> > >> >> > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core >> > >> Processor Family Integrated Graphics Controller (rev 09) >> > > >> > > Let me guess, you have an SandyBridge motherboard and were using >> > > Intel AMT? >> > >> > Correct. >> > >> > > >> > > I see those all the time on that box (and I think my Haswell >> > > one too). The best I could narrow it down was that the card decided that >> > > certain area should be in the RMRR regions. With Venu/Elen's patch >> > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able >> > > to provide an rmrr paramater to include these addresses. >> > >> > Some more information on how to use that flag would be valuable.. I >> > tried "rmrr=27000-28000=00:02.0" to no avail and I really have >> > no idea what I'm doing here =) >> >> CC-ing Elena and Venu who I hope can help you with this. >> >> You may want to provide the full 'xl dmesg' as the 'rmrr' should have >> outputted some details.. > > Tamas, > > While 'xl dmesg' and the output from the serial console will certainly > help, it appears that you may have already provided the info needed. There > may be other devices than 0:02.0 that have problems, but let us assume > that it is the only device. > > Your change to xen command line is incorrect. The correct option to add > to the xen command line is: > > rmrr=0x277e28=0:00:02.0 I see, so it has to be a page and not an address.. OTOH the problem with specifying 0x277e28 is that the fault address seem to change every time I reboot the machine. I'm not sure where that range is coming from but it always seems to be between 27000-28000. > > For the RMRR specification to work, apart from changing the Xen command > line appropriately, you will also need the corresponding patch applied to > your Xen image. The patch has been pushed upstream, but is only available > in 4.9. Did you backport that change and rebuilt your Xen image with > it? Unless you do that, the Xen command line changes won't work. Ah, no, I thought it was already part of Xen 4.8.. I'll check with 4.9 unstable in a bit. Thanks! Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote: > On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: > > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk > > wrote: > > > . snip.. > > >> >> > Given this commit is pretty old, I'm also curious why it's only > > >> >> > reported > > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > > >> >> > to be the one upon which you first tried iommu=0 on a platform > > >> >> > supporting > > >> >> > interrupt remapping? > > >> >> > > >> >> It just happens to be the one I tried. VT-d was spamming my console > > >> >> with faults like this: > > >> >> > > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr > > >> >> 277e28000, iommu reg = 82c000201000 > > 82c000201000 > > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > > >> >> > > >> >> So I figured I can just turn it off to clean up my console. I still > > >> >> don't know what these VT-d faults are about.. > > >> > > > >> > What is the 0:02.0 device? Is it being passed in to your guest? > > >> > > >> Nope, there is no pass-through to any guests. The device is: > > >> > > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core > > >> Processor Family Integrated Graphics Controller (rev 09) > > > > > > Let me guess, you have an SandyBridge motherboard and were using > > > Intel AMT? > > > > Correct. > > > > > > > > I see those all the time on that box (and I think my Haswell > > > one too). The best I could narrow it down was that the card decided that > > > certain area should be in the RMRR regions. With Venu/Elen's patch > > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able > > > to provide an rmrr paramater to include these addresses. > > > > Some more information on how to use that flag would be valuable.. I > > tried "rmrr=27000-28000=00:02.0" to no avail and I really have > > no idea what I'm doing here =) > > CC-ing Elena and Venu who I hope can help you with this. > > You may want to provide the full 'xl dmesg' as the 'rmrr' should have > outputted some details.. Tamas, While 'xl dmesg' and the output from the serial console will certainly help, it appears that you may have already provided the info needed. There may be other devices than 0:02.0 that have problems, but let us assume that it is the only device. Your change to xen command line is incorrect. The correct option to add to the xen command line is: rmrr=0x277e28=0:00:02.0 For the RMRR specification to work, apart from changing the Xen command line appropriately, you will also need the corresponding patch applied to your Xen image. The patch has been pushed upstream, but is only available in 4.9. Did you backport that change and rebuilt your Xen image with it? Unless you do that, the Xen command line changes won't work. Good luck! Venu ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote: > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk > wrote: > > . snip.. > >> >> > Given this commit is pretty old, I'm also curious why it's only > >> >> > reported > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > >> >> > to be the one upon which you first tried iommu=0 on a platform > >> >> > supporting > >> >> > interrupt remapping? > >> >> > >> >> It just happens to be the one I tried. VT-d was spamming my console > >> >> with faults like this: > >> >> > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr > >> >> 277e28000, iommu reg = 82c000201000 > 82c000201000 > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > >> >> > >> >> So I figured I can just turn it off to clean up my console. I still > >> >> don't know what these VT-d faults are about.. > >> > > >> > What is the 0:02.0 device? Is it being passed in to your guest? > >> > >> Nope, there is no pass-through to any guests. The device is: > >> > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core > >> Processor Family Integrated Graphics Controller (rev 09) > > > > Let me guess, you have an SandyBridge motherboard and were using > > Intel AMT? > > Correct. > > > > > I see those all the time on that box (and I think my Haswell > > one too). The best I could narrow it down was that the card decided that > > certain area should be in the RMRR regions. With Venu/Elen's patch > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able > > to provide an rmrr paramater to include these addresses. > > Some more information on how to use that flag would be valuable.. I > tried "rmrr=27000-28000=00:02.0" to no avail and I really have > no idea what I'm doing here =) CC-ing Elena and Venu who I hope can help you with this. You may want to provide the full 'xl dmesg' as the 'rmrr' should have outputted some details.. > > Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk wrote: > . snip.. >> >> > Given this commit is pretty old, I'm also curious why it's only reported >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens >> >> > to be the one upon which you first tried iommu=0 on a platform >> >> > supporting >> >> > interrupt remapping? >> >> >> >> It just happens to be the one I tried. VT-d was spamming my console >> >> with faults like this: >> >> >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr >> >> 277e28000, iommu reg = 82c000201000 82c000201000 >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set >> >> >> >> So I figured I can just turn it off to clean up my console. I still >> >> don't know what these VT-d faults are about.. >> > >> > What is the 0:02.0 device? Is it being passed in to your guest? >> >> Nope, there is no pass-through to any guests. The device is: >> >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core >> Processor Family Integrated Graphics Controller (rev 09) > > Let me guess, you have an SandyBridge motherboard and were using > Intel AMT? Correct. > > I see those all the time on that box (and I think my Haswell > one too). The best I could narrow it down was that the card decided that > certain area should be in the RMRR regions. With Venu/Elen's patch > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able > to provide an rmrr paramater to include these addresses. Some more information on how to use that flag would be valuable.. I tried "rmrr=27000-28000=00:02.0" to no avail and I really have no idea what I'm doing here =) Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
. snip.. > >> > Given this commit is pretty old, I'm also curious why it's only reported > >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > >> > to be the one upon which you first tried iommu=0 on a platform supporting > >> > interrupt remapping? > >> > >> It just happens to be the one I tried. VT-d was spamming my console > >> with faults like this: > >> > >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr > >> 277e28000, iommu reg = 82c000201000 > >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > >> > >> So I figured I can just turn it off to clean up my console. I still > >> don't know what these VT-d faults are about.. > > > > What is the 0:02.0 device? Is it being passed in to your guest? > > Nope, there is no pass-through to any guests. The device is: > > 00:02.0 Display controller: Intel Corporation 2nd Generation Core > Processor Family Integrated Graphics Controller (rev 09) Let me guess, you have an SandyBridge motherboard and were using Intel AMT? I see those all the time on that box (and I think my Haswell one too). The best I could narrow it down was that the card decided that certain area should be in the RMRR regions. With Venu/Elen's patch in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able to provide an rmrr paramater to include these addresses. I see those > > Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 11:48 AM, Konrad Rzeszutek Wilk wrote: > On Fri, Feb 17, 2017 at 11:42:20AM -0700, Tamas K Lengyel wrote: >> On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin wrote: >> >> From: Tian, Kevin >> >> Sent: Friday, February 17, 2017 11:35 AM >> >> > >> >> >> > >> Or wait - do you have the same issue if you use >> >> > >> "iommu=no,no-intremap"? In which case the problem would be >> >> > >> that "iommu=no" should clear more than just "iommu_enable", or >> >> > >> code checking iommu_intremap early (before iommu_setup() >> >> > >> manages to clear it in the case here) would need to made look at >> >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if >> >> > >> both variables are clear, which suggests to me that >> >> > >> iommu_enable is intended to have two different meanings in >> >> > >> different contexts (master flag vs. controlling just DMA >> >> > >> remapping). Kevin, Feng - any thoughts here? >> >> > > >> >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" >> >> > >> >> > Thanks for confirming. >> >> > >> >> > Kevin, Feng, we now depend on your input regarding the intentions >> >> > with the two variables. >> >> > >> >> >> >> Feng just left Intel. Let me take a look at code to understand the >> >> rationale behind. >> >> >> > >> > Jan, looks it's caused by your change back to 2012: >> > >> > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f >> > Author: Jan Beulich >> > Date: Fri Nov 2 17:15:30 2012 +0100 >> > >> > Before that iommu_enable was the master flag consistently. I'm still >> > trying to understand the background and you may help elaborate if >> > still something in your memory. >> > >> > I agree we should stick to one meaning after clearing up above issue. >> > >> > Given this commit is pretty old, I'm also curious why it's only reported >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens >> > to be the one upon which you first tried iommu=0 on a platform supporting >> > interrupt remapping? >> >> It just happens to be the one I tried. VT-d was spamming my console >> with faults like this: >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr >> 277e28000, iommu reg = 82c000201000 >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set >> >> So I figured I can just turn it off to clean up my console. I still >> don't know what these VT-d faults are about.. > > What is the 0:02.0 device? Is it being passed in to your guest? Nope, there is no pass-through to any guests. The device is: 00:02.0 Display controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 11:42:20AM -0700, Tamas K Lengyel wrote: > On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin wrote: > >> From: Tian, Kevin > >> Sent: Friday, February 17, 2017 11:35 AM > >> > >> > >> > >> Or wait - do you have the same issue if you use > >> > >> "iommu=no,no-intremap"? In which case the problem would be > >> > >> that "iommu=no" should clear more than just "iommu_enable", or > >> > >> code checking iommu_intremap early (before iommu_setup() > >> > >> manages to clear it in the case here) would need to made look at > >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if > >> > >> both variables are clear, which suggests to me that > >> > >> iommu_enable is intended to have two different meanings in > >> > >> different contexts (master flag vs. controlling just DMA > >> > >> remapping). Kevin, Feng - any thoughts here? > >> > > > >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" > >> > > >> > Thanks for confirming. > >> > > >> > Kevin, Feng, we now depend on your input regarding the intentions > >> > with the two variables. > >> > > >> > >> Feng just left Intel. Let me take a look at code to understand the > >> rationale behind. > >> > > > > Jan, looks it's caused by your change back to 2012: > > > > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f > > Author: Jan Beulich > > Date: Fri Nov 2 17:15:30 2012 +0100 > > > > Before that iommu_enable was the master flag consistently. I'm still > > trying to understand the background and you may help elaborate if > > still something in your memory. > > > > I agree we should stick to one meaning after clearing up above issue. > > > > Given this commit is pretty old, I'm also curious why it's only reported > > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > > to be the one upon which you first tried iommu=0 on a platform supporting > > interrupt remapping? > > It just happens to be the one I tried. VT-d was spamming my console > with faults like this: > > (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr > 277e28000, iommu reg = 82c000201000 > (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set > > So I figured I can just turn it off to clean up my console. I still > don't know what these VT-d faults are about.. What is the 0:02.0 device? Is it being passed in to your guest? > > Tamas > > ___ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin wrote: >> From: Tian, Kevin >> Sent: Friday, February 17, 2017 11:35 AM >> > >> >> > >> Or wait - do you have the same issue if you use >> > >> "iommu=no,no-intremap"? In which case the problem would be >> > >> that "iommu=no" should clear more than just "iommu_enable", or >> > >> code checking iommu_intremap early (before iommu_setup() >> > >> manages to clear it in the case here) would need to made look at >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if >> > >> both variables are clear, which suggests to me that >> > >> iommu_enable is intended to have two different meanings in >> > >> different contexts (master flag vs. controlling just DMA >> > >> remapping). Kevin, Feng - any thoughts here? >> > > >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" >> > >> > Thanks for confirming. >> > >> > Kevin, Feng, we now depend on your input regarding the intentions >> > with the two variables. >> > >> >> Feng just left Intel. Let me take a look at code to understand the >> rationale behind. >> > > Jan, looks it's caused by your change back to 2012: > > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f > Author: Jan Beulich > Date: Fri Nov 2 17:15:30 2012 +0100 > > Before that iommu_enable was the master flag consistently. I'm still > trying to understand the background and you may help elaborate if > still something in your memory. > > I agree we should stick to one meaning after clearing up above issue. > > Given this commit is pretty old, I'm also curious why it's only reported > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens > to be the one upon which you first tried iommu=0 on a platform supporting > interrupt remapping? It just happens to be the one I tried. VT-d was spamming my console with faults like this: (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr 277e28000, iommu reg = 82c000201000 (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set So I figured I can just turn it off to clean up my console. I still don't know what these VT-d faults are about.. Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Fri, Feb 17, 2017 at 1:20 AM, Jan Beulich wrote: On 17.02.17 at 07:53, wrote: >>> From: Tian, Kevin >>> Sent: Friday, February 17, 2017 11:35 AM >>> > >> >>> > >> Or wait - do you have the same issue if you use >>> > >> "iommu=no,no-intremap"? In which case the problem would be >>> > >> that "iommu=no" should clear more than just "iommu_enable", or >>> > >> code checking iommu_intremap early (before iommu_setup() >>> > >> manages to clear it in the case here) would need to made look at >>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if >>> > >> both variables are clear, which suggests to me that >>> > >> iommu_enable is intended to have two different meanings in >>> > >> different contexts (master flag vs. controlling just DMA >>> > >> remapping). Kevin, Feng - any thoughts here? >>> > > >>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" >>> > >>> > Thanks for confirming. >>> > >>> > Kevin, Feng, we now depend on your input regarding the intentions >>> > with the two variables. >>> > >>> >>> Feng just left Intel. Let me take a look at code to understand the >>> rationale behind. >>> >> >> Jan, looks it's caused by your change back to 2012: >> >> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f >> Author: Jan Beulich >> Date: Fri Nov 2 17:15:30 2012 +0100 > > Oh, I see. That's been a while; I'm sorry for not having remembered > and bugging you. > >> Before that iommu_enable was the master flag consistently. I'm still >> trying to understand the background and you may help elaborate if >> still something in your memory. > > I think the commit description is pretty clear. Especially this part > > "This could have the nice side effect of allowing to use "iommu=off" > even when x2APIC was pre-enabled by the BIOS (in which case interrupt > remapping is a requirement, but DMA translation [obviously] isn't), but > that doesn't currently work (and hence x2apic_bsp_setup() forces the > IOMMU on rather than just interrupt remapping)." > > is quite relevant in the context here. Just like Linux, we really ought > to have a way to run with interrupt remapping, but without DMA > remapping. So I think this old commit was a half-hearted step into > that direction, not recognizing that it would break some other case. > > So I think the only reasonable way forward to address Tamas's > issue is to fully disentangle DMA and interrupt remapping setup, > which I'm sure is going to take some time. As a minimal adjustment, > though, I wonder whether that old commit's adjustment to apic.c > shouldn't rather have set force_intremap. Tamas - could you check > whether > > --- a/xen/arch/x86/apic.c > +++ b/xen/arch/x86/apic.c > @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void) > goto restore_out; > } > > -force_iommu = 1; > +force_intremap = 1; > > genapic = apic_x2apic_probe(); > printk("Switched to APIC driver %s.\n", genapic->name); > > makes things any better? > > Jan Just making this change won't compile: apic.c: In function ‘x2apic_bsp_setup’: apic.c:947:5: error: ‘force_intremap’ undeclared (first use in this function) force_intremap = 1; ^ Had to also add diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index d793f5de1a..a45578608b 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -52,6 +52,7 @@ custom_param("iommu", parse_iommu_param); bool_t __initdata iommu_enable = 1; bool_t __read_mostly iommu_enabled; bool_t __read_mostly force_iommu; +bool_t __read_mostly force_intremap; bool_t __hwdom_initdata iommu_dom0_strict; bool_t __read_mostly iommu_verbose; bool_t __read_mostly iommu_workaround_bios_bug; @@ -364,7 +365,7 @@ int iommu_iotlb_flush_all(struct domain *d) int __init iommu_setup(void) { int rc = -ENODEV; -bool_t force_intremap = force_iommu && iommu_intremap; +force_intremap = force_iommu && iommu_intremap; if ( iommu_dom0_strict ) iommu_passthrough = 0; diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 5803e3f95b..770e4f6282 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -29,6 +29,7 @@ extern bool_t iommu_enable, iommu_enabled; extern bool_t force_iommu, iommu_verbose; +extern bool_t force_intremap; extern bool_t iommu_workaround_bios_bug, iommu_igfx, iommu_passthrough; extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost; extern bool_t iommu_hap_pt_share; With these changes in place the system starts to boot but dom0 fails to assemble the mdraid disk and find root: Begin: Running /scripts/local-block ... mdadm: No devices listed in conf file were found. WARNING: Failed to connect to lvmetad. Falling back to device scanning. Volume group "vg" not found Cannot process volume group vg done. done. Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?)
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
>>> On 17.02.17 at 07:53, wrote: >> From: Tian, Kevin >> Sent: Friday, February 17, 2017 11:35 AM >> > >> >> > >> Or wait - do you have the same issue if you use >> > >> "iommu=no,no-intremap"? In which case the problem would be >> > >> that "iommu=no" should clear more than just "iommu_enable", or >> > >> code checking iommu_intremap early (before iommu_setup() >> > >> manages to clear it in the case here) would need to made look at >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if >> > >> both variables are clear, which suggests to me that >> > >> iommu_enable is intended to have two different meanings in >> > >> different contexts (master flag vs. controlling just DMA >> > >> remapping). Kevin, Feng - any thoughts here? >> > > >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" >> > >> > Thanks for confirming. >> > >> > Kevin, Feng, we now depend on your input regarding the intentions >> > with the two variables. >> > >> >> Feng just left Intel. Let me take a look at code to understand the >> rationale behind. >> > > Jan, looks it's caused by your change back to 2012: > > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f > Author: Jan Beulich > Date: Fri Nov 2 17:15:30 2012 +0100 Oh, I see. That's been a while; I'm sorry for not having remembered and bugging you. > Before that iommu_enable was the master flag consistently. I'm still > trying to understand the background and you may help elaborate if > still something in your memory. I think the commit description is pretty clear. Especially this part "This could have the nice side effect of allowing to use "iommu=off" even when x2APIC was pre-enabled by the BIOS (in which case interrupt remapping is a requirement, but DMA translation [obviously] isn't), but that doesn't currently work (and hence x2apic_bsp_setup() forces the IOMMU on rather than just interrupt remapping)." is quite relevant in the context here. Just like Linux, we really ought to have a way to run with interrupt remapping, but without DMA remapping. So I think this old commit was a half-hearted step into that direction, not recognizing that it would break some other case. So I think the only reasonable way forward to address Tamas's issue is to fully disentangle DMA and interrupt remapping setup, which I'm sure is going to take some time. As a minimal adjustment, though, I wonder whether that old commit's adjustment to apic.c shouldn't rather have set force_intremap. Tamas - could you check whether --- a/xen/arch/x86/apic.c +++ b/xen/arch/x86/apic.c @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void) goto restore_out; } -force_iommu = 1; +force_intremap = 1; genapic = apic_x2apic_probe(); printk("Switched to APIC driver %s.\n", genapic->name); makes things any better? Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
>>> On 17.02.17 at 04:35, wrote: > Feng just left Intel. Let me take a look at code to understand the > rationale behind. Do you know if he means to stay VT-d maintainer? If yes, do you have a new email address of him? If not, would you mind sending an update to ./MAINTAINERS removing him? Thanks, Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
> From: Tian, Kevin > Sent: Friday, February 17, 2017 11:35 AM > > >> > > >> Or wait - do you have the same issue if you use > > >> "iommu=no,no-intremap"? In which case the problem would be > > >> that "iommu=no" should clear more than just "iommu_enable", or > > >> code checking iommu_intremap early (before iommu_setup() > > >> manages to clear it in the case here) would need to made look at > > >> both variables. Oddly enough acpi_parse_dmar() only bails if > > >> both variables are clear, which suggests to me that > > >> iommu_enable is intended to have two different meanings in > > >> different contexts (master flag vs. controlling just DMA > > >> remapping). Kevin, Feng - any thoughts here? > > > > > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" > > > > Thanks for confirming. > > > > Kevin, Feng, we now depend on your input regarding the intentions > > with the two variables. > > > > Feng just left Intel. Let me take a look at code to understand the > rationale behind. > Jan, looks it's caused by your change back to 2012: commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f Author: Jan Beulich Date: Fri Nov 2 17:15:30 2012 +0100 Before that iommu_enable was the master flag consistently. I'm still trying to understand the background and you may help elaborate if still something in your memory. I agree we should stick to one meaning after clearing up above issue. Given this commit is pretty old, I'm also curious why it's only reported on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens to be the one upon which you first tried iommu=0 on a platform supporting interrupt remapping? Thanks Kevin ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
> From: Jan Beulich [mailto:jbeul...@suse.com] > Sent: Thursday, February 16, 2017 5:32 PM > > >>> On 15.02.17 at 20:30, wrote: > > On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich wrote: > > On 15.02.17 at 00:21, wrote: > >>> On 14/02/2017 22:47, Tamas K Lengyel wrote: > (XEN) Switched to APIC driver x2apic_cluster. > (XEN) XSM Framework v1.0.0 initialized > (XEN) Flask: 128 avtab hash slots, 394 rules. > (XEN) Flask: 128 avtab hash slots, 394 rules. > (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools > (XEN) Flask: 12 classes, 394 rules > (XEN) Flask: Starting in enforcing mode. > (XEN) xstate: size: 0x340 and states: 0x7 > (XEN) Intel machine check reporting enabled > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Platform timer is 14.318MHz HPET > (XEN) Detected 3392.326 MHz processor. > (XEN) Initing memory sharing. > (XEN) alt table 82d0802d3f38 -> 82d0802d5564 > (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f > (XEN) PCI: Not using MCFG for segment bus 00-3f > (XEN) > (XEN) > (XEN) Panic on CPU 0: > (XEN) Couldn't enable IOMMU and iommu=required/force > (XEN) > (XEN) > (XEN) Reboot in five seconds... > (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. > > As seen in the command line iommu is not set to required or forced. > Yet Xen thinks it is set to required/force. Has anyone else run into > this issue? > >>> > >>> This area is a rats nest :( > >>> > >>> The problem is that the APIC setup has chosen to use the x2apic_cluster > >>> driver, which in the Xen code depends on x2APIC, which depends on > >>> interrupt remapping, which depends on an IOMMU. > >>> > >>> (I could have sworn we fixed this before), but the bug is that the APIC > >>> setup shouldn't choose any of the x2apic modes if there isn't an > >>> interrupt remapping capable IOMMU. > >> > >> And from going over the code I can't see how this would happen, > >> so Tamas, it would be nice if you could add some verbosity to the > >> functions involved. In particular x2apic_bsp_setup() must be > >> getting success (zero) from iommu_enable_x2apic_IR(), yet that > >> function calls iommu_supports_eim(), which ought to produce > >> false through its very first exit path in your case. > >> > >> Or wait - do you have the same issue if you use > >> "iommu=no,no-intremap"? In which case the problem would be > >> that "iommu=no" should clear more than just "iommu_enable", or > >> code checking iommu_intremap early (before iommu_setup() > >> manages to clear it in the case here) would need to made look at > >> both variables. Oddly enough acpi_parse_dmar() only bails if > >> both variables are clear, which suggests to me that > >> iommu_enable is intended to have two different meanings in > >> different contexts (master flag vs. controlling just DMA > >> remapping). Kevin, Feng - any thoughts here? > > > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" > > Thanks for confirming. > > Kevin, Feng, we now depend on your input regarding the intentions > with the two variables. > Feng just left Intel. Let me take a look at code to understand the rationale behind. Thanks Kevin ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
>>> On 15.02.17 at 20:30, wrote: > On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich wrote: > On 15.02.17 at 00:21, wrote: >>> On 14/02/2017 22:47, Tamas K Lengyel wrote: (XEN) Switched to APIC driver x2apic_cluster. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: 128 avtab hash slots, 394 rules. (XEN) Flask: 128 avtab hash slots, 394 rules. (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools (XEN) Flask: 12 classes, 394 rules (XEN) Flask: Starting in enforcing mode. (XEN) xstate: size: 0x340 and states: 0x7 (XEN) Intel machine check reporting enabled (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Platform timer is 14.318MHz HPET (XEN) Detected 3392.326 MHz processor. (XEN) Initing memory sharing. (XEN) alt table 82d0802d3f38 -> 82d0802d5564 (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f (XEN) PCI: Not using MCFG for segment bus 00-3f (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) Couldn't enable IOMMU and iommu=required/force (XEN) (XEN) (XEN) Reboot in five seconds... (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. As seen in the command line iommu is not set to required or forced. Yet Xen thinks it is set to required/force. Has anyone else run into this issue? >>> >>> This area is a rats nest :( >>> >>> The problem is that the APIC setup has chosen to use the x2apic_cluster >>> driver, which in the Xen code depends on x2APIC, which depends on >>> interrupt remapping, which depends on an IOMMU. >>> >>> (I could have sworn we fixed this before), but the bug is that the APIC >>> setup shouldn't choose any of the x2apic modes if there isn't an >>> interrupt remapping capable IOMMU. >> >> And from going over the code I can't see how this would happen, >> so Tamas, it would be nice if you could add some verbosity to the >> functions involved. In particular x2apic_bsp_setup() must be >> getting success (zero) from iommu_enable_x2apic_IR(), yet that >> function calls iommu_supports_eim(), which ought to produce >> false through its very first exit path in your case. >> >> Or wait - do you have the same issue if you use >> "iommu=no,no-intremap"? In which case the problem would be >> that "iommu=no" should clear more than just "iommu_enable", or >> code checking iommu_intremap early (before iommu_setup() >> manages to clear it in the case here) would need to made look at >> both variables. Oddly enough acpi_parse_dmar() only bails if >> both variables are clear, which suggests to me that >> iommu_enable is intended to have two different meanings in >> different contexts (master flag vs. controlling just DMA >> remapping). Kevin, Feng - any thoughts here? > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" Thanks for confirming. Kevin, Feng, we now depend on your input regarding the intentions with the two variables. Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Wed, Feb 15, 2017 at 12:30 PM, Tamas K Lengyel wrote: > On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich wrote: > On 15.02.17 at 00:21, wrote: >>> On 14/02/2017 22:47, Tamas K Lengyel wrote: (XEN) Switched to APIC driver x2apic_cluster. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: 128 avtab hash slots, 394 rules. (XEN) Flask: 128 avtab hash slots, 394 rules. (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools (XEN) Flask: 12 classes, 394 rules (XEN) Flask: Starting in enforcing mode. (XEN) xstate: size: 0x340 and states: 0x7 (XEN) Intel machine check reporting enabled (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Platform timer is 14.318MHz HPET (XEN) Detected 3392.326 MHz processor. (XEN) Initing memory sharing. (XEN) alt table 82d0802d3f38 -> 82d0802d5564 (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f (XEN) PCI: Not using MCFG for segment bus 00-3f (XEN) (XEN) (XEN) Panic on CPU 0: (XEN) Couldn't enable IOMMU and iommu=required/force (XEN) (XEN) (XEN) Reboot in five seconds... (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. As seen in the command line iommu is not set to required or forced. Yet Xen thinks it is set to required/force. Has anyone else run into this issue? >>> >>> This area is a rats nest :( >>> >>> The problem is that the APIC setup has chosen to use the x2apic_cluster >>> driver, which in the Xen code depends on x2APIC, which depends on >>> interrupt remapping, which depends on an IOMMU. >>> >>> (I could have sworn we fixed this before), but the bug is that the APIC >>> setup shouldn't choose any of the x2apic modes if there isn't an >>> interrupt remapping capable IOMMU. >> >> And from going over the code I can't see how this would happen, >> so Tamas, it would be nice if you could add some verbosity to the >> functions involved. In particular x2apic_bsp_setup() must be >> getting success (zero) from iommu_enable_x2apic_IR(), yet that >> function calls iommu_supports_eim(), which ought to produce >> false through its very first exit path in your case. >> >> Or wait - do you have the same issue if you use >> "iommu=no,no-intremap"? In which case the problem would be >> that "iommu=no" should clear more than just "iommu_enable", or >> code checking iommu_intremap early (before iommu_setup() >> manages to clear it in the case here) would need to made look at >> both variables. Oddly enough acpi_parse_dmar() only bails if >> both variables are clear, which suggests to me that >> iommu_enable is intended to have two different meanings in >> different contexts (master flag vs. controlling just DMA >> remapping). Kevin, Feng - any thoughts here? > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" And just to clarify, x2apic is disabled in this case "(XEN) Not enabling x2APIC: depends on iommu_supports_eim." Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich wrote: On 15.02.17 at 00:21, wrote: >> On 14/02/2017 22:47, Tamas K Lengyel wrote: >>> (XEN) Switched to APIC driver x2apic_cluster. >>> (XEN) XSM Framework v1.0.0 initialized >>> (XEN) Flask: 128 avtab hash slots, 394 rules. >>> (XEN) Flask: 128 avtab hash slots, 394 rules. >>> (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools >>> (XEN) Flask: 12 classes, 394 rules >>> (XEN) Flask: Starting in enforcing mode. >>> (XEN) xstate: size: 0x340 and states: 0x7 >>> (XEN) Intel machine check reporting enabled >>> (XEN) Using scheduler: SMP Credit Scheduler (credit) >>> (XEN) Platform timer is 14.318MHz HPET >>> (XEN) Detected 3392.326 MHz processor. >>> (XEN) Initing memory sharing. >>> (XEN) alt table 82d0802d3f38 -> 82d0802d5564 >>> (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f >>> (XEN) PCI: Not using MCFG for segment bus 00-3f >>> (XEN) >>> (XEN) >>> (XEN) Panic on CPU 0: >>> (XEN) Couldn't enable IOMMU and iommu=required/force >>> (XEN) >>> (XEN) >>> (XEN) Reboot in five seconds... >>> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. >>> >>> As seen in the command line iommu is not set to required or forced. >>> Yet Xen thinks it is set to required/force. Has anyone else run into >>> this issue? >> >> This area is a rats nest :( >> >> The problem is that the APIC setup has chosen to use the x2apic_cluster >> driver, which in the Xen code depends on x2APIC, which depends on >> interrupt remapping, which depends on an IOMMU. >> >> (I could have sworn we fixed this before), but the bug is that the APIC >> setup shouldn't choose any of the x2apic modes if there isn't an >> interrupt remapping capable IOMMU. > > And from going over the code I can't see how this would happen, > so Tamas, it would be nice if you could add some verbosity to the > functions involved. In particular x2apic_bsp_setup() must be > getting success (zero) from iommu_enable_x2apic_IR(), yet that > function calls iommu_supports_eim(), which ought to produce > false through its very first exit path in your case. > > Or wait - do you have the same issue if you use > "iommu=no,no-intremap"? In which case the problem would be > that "iommu=no" should clear more than just "iommu_enable", or > code checking iommu_intremap early (before iommu_setup() > manages to clear it in the case here) would need to made look at > both variables. Oddly enough acpi_parse_dmar() only bails if > both variables are clear, which suggests to me that > iommu_enable is intended to have two different meanings in > different contexts (master flag vs. controlling just DMA > remapping). Kevin, Feng - any thoughts here? iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
>>> On 15.02.17 at 00:21, wrote: > On 14/02/2017 22:47, Tamas K Lengyel wrote: >> (XEN) Switched to APIC driver x2apic_cluster. >> (XEN) XSM Framework v1.0.0 initialized >> (XEN) Flask: 128 avtab hash slots, 394 rules. >> (XEN) Flask: 128 avtab hash slots, 394 rules. >> (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools >> (XEN) Flask: 12 classes, 394 rules >> (XEN) Flask: Starting in enforcing mode. >> (XEN) xstate: size: 0x340 and states: 0x7 >> (XEN) Intel machine check reporting enabled >> (XEN) Using scheduler: SMP Credit Scheduler (credit) >> (XEN) Platform timer is 14.318MHz HPET >> (XEN) Detected 3392.326 MHz processor. >> (XEN) Initing memory sharing. >> (XEN) alt table 82d0802d3f38 -> 82d0802d5564 >> (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f >> (XEN) PCI: Not using MCFG for segment bus 00-3f >> (XEN) >> (XEN) >> (XEN) Panic on CPU 0: >> (XEN) Couldn't enable IOMMU and iommu=required/force >> (XEN) >> (XEN) >> (XEN) Reboot in five seconds... >> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. >> >> As seen in the command line iommu is not set to required or forced. >> Yet Xen thinks it is set to required/force. Has anyone else run into >> this issue? > > This area is a rats nest :( > > The problem is that the APIC setup has chosen to use the x2apic_cluster > driver, which in the Xen code depends on x2APIC, which depends on > interrupt remapping, which depends on an IOMMU. > > (I could have sworn we fixed this before), but the bug is that the APIC > setup shouldn't choose any of the x2apic modes if there isn't an > interrupt remapping capable IOMMU. And from going over the code I can't see how this would happen, so Tamas, it would be nice if you could add some verbosity to the functions involved. In particular x2apic_bsp_setup() must be getting success (zero) from iommu_enable_x2apic_IR(), yet that function calls iommu_supports_eim(), which ought to produce false through its very first exit path in your case. Or wait - do you have the same issue if you use "iommu=no,no-intremap"? In which case the problem would be that "iommu=no" should clear more than just "iommu_enable", or code checking iommu_intremap early (before iommu_setup() manages to clear it in the case here) would need to made look at both variables. Oddly enough acpi_parse_dmar() only bails if both variables are clear, which suggests to me that iommu_enable is intended to have two different meanings in different contexts (master flag vs. controlling just DMA remapping). Kevin, Feng - any thoughts here? Jan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0
On 14/02/2017 22:47, Tamas K Lengyel wrote: > (XEN) Switched to APIC driver x2apic_cluster. > (XEN) XSM Framework v1.0.0 initialized > (XEN) Flask: 128 avtab hash slots, 394 rules. > (XEN) Flask: 128 avtab hash slots, 394 rules. > (XEN) Flask: 5 users, 3 roles, 43 types, 2 bools > (XEN) Flask: 12 classes, 394 rules > (XEN) Flask: Starting in enforcing mode. > (XEN) xstate: size: 0x340 and states: 0x7 > (XEN) Intel machine check reporting enabled > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Platform timer is 14.318MHz HPET > (XEN) Detected 3392.326 MHz processor. > (XEN) Initing memory sharing. > (XEN) alt table 82d0802d3f38 -> 82d0802d5564 > (XEN) PCI: MCFG configuration 0: base e000 segment buses 00 - 3f > (XEN) PCI: Not using MCFG for segment bus 00-3f > (XEN) > (XEN) > (XEN) Panic on CPU 0: > (XEN) Couldn't enable IOMMU and iommu=required/force > (XEN) > (XEN) > (XEN) Reboot in five seconds... > (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. > > As seen in the command line iommu is not set to required or forced. > Yet Xen thinks it is set to required/force. Has anyone else run into > this issue? This area is a rats nest :( The problem is that the APIC setup has chosen to use the x2apic_cluster driver, which in the Xen code depends on x2APIC, which depends on interrupt remapping, which depends on an IOMMU. (I could have sworn we fixed this before), but the bug is that the APIC setup shouldn't choose any of the x2apic modes if there isn't an interrupt remapping capable IOMMU. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel