Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Roger Pau Monné]

On Fri, Feb 17, 2017 at 02:01:15PM -0600, Venu Busireddy wrote:
> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
> > > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
> > >  wrote:
> > > > . snip..
> > > >> >> > Given this commit is pretty old, I'm also curious why it's only 
> > > >> >> > reported
> > > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 
> > > >> >> > happens
> > > >> >> > to be the one upon which you first tried iommu=0 on a platform 
> > > >> >> > supporting
> > > >> >> > interrupt remapping?
> > > >> >>
> > > >> >> It just happens to be the one I tried. VT-d was spamming my console
> > > >> >> with faults like this:
> > > >> >>
> > > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault 
> > > >> >> addr
> > > >> >> 277e28000, iommu reg = 82c000201000
> > > 82c000201000
> > > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> > > >> >>
> > > >> >> So I figured I can just turn it off to clean up my console. I still
> > > >> >> don't know what these VT-d faults are about..
> > > >> >
> > > >> > What is the 0:02.0 device? Is it being passed in to your guest?
> > > >>
> > > >> Nope, there is no pass-through to any guests. The device is:
> > > >>
> > > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
> > > >> Processor Family Integrated Graphics Controller (rev 09)
> > > >
> > > > Let me guess, you have an SandyBridge motherboard and were using
> > > > Intel AMT?
> > > 
> > > Correct.
> > > 
> > > >
> > > > I see those all the time on that box (and I think my Haswell
> > > > one too). The best I could narrow it down was that the card decided that
> > > > certain area should be in the RMRR regions. With Venu/Elen's patch
> > > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
> > > > to provide an rmrr paramater to include these addresses.
> > > 
> > > Some more information on how to use that flag would be valuable.. I
> > > tried "rmrr=27000-28000=00:02.0" to no avail and I really have
> > > no idea what I'm doing here =)
> > 
> > CC-ing Elena and Venu who I hope can help you with this.
> > 
> > You may want to provide the full 'xl dmesg' as the 'rmrr' should have
> > outputted some details..
> 
> Tamas,
> 
> While 'xl dmesg' and the output from the serial console will certainly
> help, it appears that you may have already provided the info needed. There
> may be other devices than 0:02.0 that have problems, but let us assume
> that it is the only device.
> 
> Your change to xen command line is incorrect. The correct option to add
> to the xen command line is:
> 
> rmrr=0x277e28=0:00:02.0

Could you please improve the documentation by mentioning that 'start' and 'end'
should be frame numbers instead of full memory addresses?

I also stumbled upon this and had to go look at the code in order to figure out
what I should pass to rmrr. Also a couple of examples about it's utilization
wouldn't hurt IMHO.

Roger.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tian, Kevin]

> From: Jan Beulich [mailto:jbeul...@suse.com]
> Sent: Friday, February 17, 2017 4:21 PM
> 
> I think the commit description is pretty clear. Especially this part
> 
> "This could have the nice side effect of allowing to use "iommu=off"
>  even when x2APIC was pre-enabled by the BIOS (in which case interrupt
>  remapping is a requirement, but DMA translation [obviously] isn't), but
>  that doesn't currently work (and hence x2apic_bsp_setup() forces the
>  IOMMU on rather than just interrupt remapping)."
> 
> is quite relevant in the context here. Just like Linux, we really ought
> to have a way to run with interrupt remapping, but without DMA
> remapping. So I think this old commit was a half-hearted step into
> that direction, not recognizing that it would break some other case.
> 
> So I think the only reasonable way forward to address Tamas's
> issue is to fully disentangle DMA and interrupt remapping setup,
> which I'm sure is going to take some time. As a minimal adjustment,
> though, I wonder whether that old commit's adjustment to apic.c
> shouldn't rather have set force_intremap. Tamas - could you check
> whether
> 
> --- a/xen/arch/x86/apic.c
> +++ b/xen/arch/x86/apic.c
> @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void)
>  goto restore_out;
>  }
> 
> -force_iommu = 1;
> +force_intremap = 1;
> 
>  genapic = apic_x2apic_probe();
>  printk("Switched to APIC driver %s.\n", genapic->name);
> 
> makes things any better?
> 
> Jan

Looks there were some conflicting commits moving the policy 
back-and-forth. For example, your another commit in 2013 
changed back to the old way that iommu_intremap depends 
on iommu_enable:

if ( iommu_enable )
{
rc = iommu_hardware_setup();
iommu_enabled = (rc == 0);
}
 (added by fae03721)
if ( !iommu_enabled )
iommu_intremap = 0;

if ( (force_iommu && !iommu_enabled) ||
 (force_intremap && !iommu_intremap) )
panic("Couldn't enable %s and iommu=required/force",
  !iommu_enabled ? "IOMMU" : "Interrupt Remapping");

But I cannot understand why Tamas doesn't hit above panic
again...

Thanks
Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Fri, Feb 17, 2017 at 2:31 PM, Andrew Cooper
 wrote:
> On 17/02/17 21:28, Tamas K Lengyel wrote:
>> On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy
>>  wrote:
>>> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote:
 On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
> On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
>  wrote:
>> . snip..
>> Given this commit is pretty old, I'm also curious why it's only 
>> reported
>> on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
>> to be the one upon which you first tried iommu=0 on a platform 
>> supporting
>> interrupt remapping?
> It just happens to be the one I tried. VT-d was spamming my console
> with faults like this:
>
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 277e28000, iommu reg = 82c000201000
> 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>
> So I figured I can just turn it off to clean up my console. I still
> don't know what these VT-d faults are about..
 What is the 0:02.0 device? Is it being passed in to your guest?
>>> Nope, there is no pass-through to any guests. The device is:
>>>
>>> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
>>> Processor Family Integrated Graphics Controller (rev 09)
>> Let me guess, you have an SandyBridge motherboard and were using
>> Intel AMT?
> Correct.
>
>> I see those all the time on that box (and I think my Haswell
>> one too). The best I could narrow it down was that the card decided that
>> certain area should be in the RMRR regions. With Venu/Elen's patch
>> in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
>> to provide an rmrr paramater to include these addresses.
> Some more information on how to use that flag would be valuable.. I
> tried "rmrr=27000-28000=00:02.0" to no avail and I really have
> no idea what I'm doing here =)
 CC-ing Elena and Venu who I hope can help you with this.

 You may want to provide the full 'xl dmesg' as the 'rmrr' should have
 outputted some details..
>>> Tamas,
>>>
>>> While 'xl dmesg' and the output from the serial console will certainly
>>> help, it appears that you may have already provided the info needed. There
>>> may be other devices than 0:02.0 that have problems, but let us assume
>>> that it is the only device.
>>>
>>> Your change to xen command line is incorrect. The correct option to add
>>> to the xen command line is:
>>>
>>> rmrr=0x277e28=0:00:02.0
>> I see, so it has to be a page and not an address.. OTOH the problem
>> with specifying 0x277e28 is that the fault address seem to change
>> every time I reboot the machine. I'm not sure where that range is
>> coming from but it always seems to be between 27000-28000.
>
> Looking at the parsing code, ranges work fine but you need to use 0x for
> hex addresses, or the address will be interpreted as decimal.
>

Ah, yes, makes sense =)

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Andrew Cooper]

On 17/02/17 21:28, Tamas K Lengyel wrote:
> On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy
>  wrote:
>> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote:
>>> On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
 On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
  wrote:
> . snip..
> Given this commit is pretty old, I'm also curious why it's only 
> reported
> on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> to be the one upon which you first tried iommu=0 on a platform 
> supporting
> interrupt remapping?
 It just happens to be the one I tried. VT-d was spamming my console
 with faults like this:

 (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
 277e28000, iommu reg = 82c000201000
 82c000201000
 (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

 So I figured I can just turn it off to clean up my console. I still
 don't know what these VT-d faults are about..
>>> What is the 0:02.0 device? Is it being passed in to your guest?
>> Nope, there is no pass-through to any guests. The device is:
>>
>> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
>> Processor Family Integrated Graphics Controller (rev 09)
> Let me guess, you have an SandyBridge motherboard and were using
> Intel AMT?
 Correct.

> I see those all the time on that box (and I think my Haswell
> one too). The best I could narrow it down was that the card decided that
> certain area should be in the RMRR regions. With Venu/Elen's patch
> in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
> to provide an rmrr paramater to include these addresses.
 Some more information on how to use that flag would be valuable.. I
 tried "rmrr=27000-28000=00:02.0" to no avail and I really have
 no idea what I'm doing here =)
>>> CC-ing Elena and Venu who I hope can help you with this.
>>>
>>> You may want to provide the full 'xl dmesg' as the 'rmrr' should have
>>> outputted some details..
>> Tamas,
>>
>> While 'xl dmesg' and the output from the serial console will certainly
>> help, it appears that you may have already provided the info needed. There
>> may be other devices than 0:02.0 that have problems, but let us assume
>> that it is the only device.
>>
>> Your change to xen command line is incorrect. The correct option to add
>> to the xen command line is:
>>
>> rmrr=0x277e28=0:00:02.0
> I see, so it has to be a page and not an address.. OTOH the problem
> with specifying 0x277e28 is that the fault address seem to change
> every time I reboot the machine. I'm not sure where that range is
> coming from but it always seems to be between 27000-28000.

Looking at the parsing code, ranges work fine but you need to use 0x for
hex addresses, or the address will be interpreted as decimal.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Fri, Feb 17, 2017 at 1:01 PM, Venu Busireddy
 wrote:
> On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote:
>> On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
>> > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
>> >  wrote:
>> > > . snip..
>> > >> >> > Given this commit is pretty old, I'm also curious why it's only 
>> > >> >> > reported
>> > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
>> > >> >> > to be the one upon which you first tried iommu=0 on a platform 
>> > >> >> > supporting
>> > >> >> > interrupt remapping?
>> > >> >>
>> > >> >> It just happens to be the one I tried. VT-d was spamming my console
>> > >> >> with faults like this:
>> > >> >>
>> > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
>> > >> >> 277e28000, iommu reg = 82c000201000
>> > 82c000201000
>> > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>> > >> >>
>> > >> >> So I figured I can just turn it off to clean up my console. I still
>> > >> >> don't know what these VT-d faults are about..
>> > >> >
>> > >> > What is the 0:02.0 device? Is it being passed in to your guest?
>> > >>
>> > >> Nope, there is no pass-through to any guests. The device is:
>> > >>
>> > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
>> > >> Processor Family Integrated Graphics Controller (rev 09)
>> > >
>> > > Let me guess, you have an SandyBridge motherboard and were using
>> > > Intel AMT?
>> >
>> > Correct.
>> >
>> > >
>> > > I see those all the time on that box (and I think my Haswell
>> > > one too). The best I could narrow it down was that the card decided that
>> > > certain area should be in the RMRR regions. With Venu/Elen's patch
>> > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
>> > > to provide an rmrr paramater to include these addresses.
>> >
>> > Some more information on how to use that flag would be valuable.. I
>> > tried "rmrr=27000-28000=00:02.0" to no avail and I really have
>> > no idea what I'm doing here =)
>>
>> CC-ing Elena and Venu who I hope can help you with this.
>>
>> You may want to provide the full 'xl dmesg' as the 'rmrr' should have
>> outputted some details..
>
> Tamas,
>
> While 'xl dmesg' and the output from the serial console will certainly
> help, it appears that you may have already provided the info needed. There
> may be other devices than 0:02.0 that have problems, but let us assume
> that it is the only device.
>
> Your change to xen command line is incorrect. The correct option to add
> to the xen command line is:
>
> rmrr=0x277e28=0:00:02.0

I see, so it has to be a page and not an address.. OTOH the problem
with specifying 0x277e28 is that the fault address seem to change
every time I reboot the machine. I'm not sure where that range is
coming from but it always seems to be between 27000-28000.

>
> For the RMRR specification to work, apart from changing the Xen command
> line appropriately, you will also need the corresponding patch applied to
> your Xen image. The patch has been pushed upstream, but is only available
> in 4.9. Did you backport that change and rebuilt your Xen image with
> it? Unless you do that, the Xen command line changes won't work.

Ah, no, I thought it was already part of Xen 4.8.. I'll check with 4.9
unstable in a bit.

Thanks!
Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Venu Busireddy]

On Fri, Feb 17, 2017 at 02:29:32PM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
> > On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
> >  wrote:
> > > . snip..
> > >> >> > Given this commit is pretty old, I'm also curious why it's only 
> > >> >> > reported
> > >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> > >> >> > to be the one upon which you first tried iommu=0 on a platform 
> > >> >> > supporting
> > >> >> > interrupt remapping?
> > >> >>
> > >> >> It just happens to be the one I tried. VT-d was spamming my console
> > >> >> with faults like this:
> > >> >>
> > >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> > >> >> 277e28000, iommu reg = 82c000201000
> > 82c000201000
> > >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> > >> >>
> > >> >> So I figured I can just turn it off to clean up my console. I still
> > >> >> don't know what these VT-d faults are about..
> > >> >
> > >> > What is the 0:02.0 device? Is it being passed in to your guest?
> > >>
> > >> Nope, there is no pass-through to any guests. The device is:
> > >>
> > >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
> > >> Processor Family Integrated Graphics Controller (rev 09)
> > >
> > > Let me guess, you have an SandyBridge motherboard and were using
> > > Intel AMT?
> > 
> > Correct.
> > 
> > >
> > > I see those all the time on that box (and I think my Haswell
> > > one too). The best I could narrow it down was that the card decided that
> > > certain area should be in the RMRR regions. With Venu/Elen's patch
> > > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
> > > to provide an rmrr paramater to include these addresses.
> > 
> > Some more information on how to use that flag would be valuable.. I
> > tried "rmrr=27000-28000=00:02.0" to no avail and I really have
> > no idea what I'm doing here =)
> 
> CC-ing Elena and Venu who I hope can help you with this.
> 
> You may want to provide the full 'xl dmesg' as the 'rmrr' should have
> outputted some details..

Tamas,

While 'xl dmesg' and the output from the serial console will certainly
help, it appears that you may have already provided the info needed. There
may be other devices than 0:02.0 that have problems, but let us assume
that it is the only device.

Your change to xen command line is incorrect. The correct option to add
to the xen command line is:

rmrr=0x277e28=0:00:02.0

For the RMRR specification to work, apart from changing the Xen command
line appropriately, you will also need the corresponding patch applied to
your Xen image. The patch has been pushed upstream, but is only available
in 4.9. Did you backport that change and rebuilt your Xen image with
it? Unless you do that, the Xen command line changes won't work.

Good luck!

Venu


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Konrad Rzeszutek Wilk]

On Fri, Feb 17, 2017 at 12:27:25PM -0700, Tamas K Lengyel wrote:
> On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
>  wrote:
> > . snip..
> >> >> > Given this commit is pretty old, I'm also curious why it's only 
> >> >> > reported
> >> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> >> >> > to be the one upon which you first tried iommu=0 on a platform 
> >> >> > supporting
> >> >> > interrupt remapping?
> >> >>
> >> >> It just happens to be the one I tried. VT-d was spamming my console
> >> >> with faults like this:
> >> >>
> >> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> >> >> 277e28000, iommu reg = 82c000201000
> 82c000201000
> >> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> >> >>
> >> >> So I figured I can just turn it off to clean up my console. I still
> >> >> don't know what these VT-d faults are about..
> >> >
> >> > What is the 0:02.0 device? Is it being passed in to your guest?
> >>
> >> Nope, there is no pass-through to any guests. The device is:
> >>
> >> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
> >> Processor Family Integrated Graphics Controller (rev 09)
> >
> > Let me guess, you have an SandyBridge motherboard and were using
> > Intel AMT?
> 
> Correct.
> 
> >
> > I see those all the time on that box (and I think my Haswell
> > one too). The best I could narrow it down was that the card decided that
> > certain area should be in the RMRR regions. With Venu/Elen's patch
> > in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
> > to provide an rmrr paramater to include these addresses.
> 
> Some more information on how to use that flag would be valuable.. I
> tried "rmrr=27000-28000=00:02.0" to no avail and I really have
> no idea what I'm doing here =)

CC-ing Elena and Venu who I hope can help you with this.

You may want to provide the full 'xl dmesg' as the 'rmrr' should have
outputted some details..
> 
> Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Fri, Feb 17, 2017 at 11:56 AM, Konrad Rzeszutek Wilk
 wrote:
> . snip..
>> >> > Given this commit is pretty old, I'm also curious why it's only reported
>> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
>> >> > to be the one upon which you first tried iommu=0 on a platform 
>> >> > supporting
>> >> > interrupt remapping?
>> >>
>> >> It just happens to be the one I tried. VT-d was spamming my console
>> >> with faults like this:
>> >>
>> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
>> >> 277e28000, iommu reg = 82c000201000
82c000201000
>> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>> >>
>> >> So I figured I can just turn it off to clean up my console. I still
>> >> don't know what these VT-d faults are about..
>> >
>> > What is the 0:02.0 device? Is it being passed in to your guest?
>>
>> Nope, there is no pass-through to any guests. The device is:
>>
>> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
>> Processor Family Integrated Graphics Controller (rev 09)
>
> Let me guess, you have an SandyBridge motherboard and were using
> Intel AMT?

Correct.

>
> I see those all the time on that box (and I think my Haswell
> one too). The best I could narrow it down was that the card decided that
> certain area should be in the RMRR regions. With Venu/Elen's patch
> in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
> to provide an rmrr paramater to include these addresses.

Some more information on how to use that flag would be valuable.. I
tried "rmrr=27000-28000=00:02.0" to no avail and I really have
no idea what I'm doing here =)

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Konrad Rzeszutek Wilk]

. snip..
> >> > Given this commit is pretty old, I'm also curious why it's only reported
> >> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> >> > to be the one upon which you first tried iommu=0 on a platform supporting
> >> > interrupt remapping?
> >>
> >> It just happens to be the one I tried. VT-d was spamming my console
> >> with faults like this:
> >>
> >> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> >> 277e28000, iommu reg = 82c000201000
> >> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> >>
> >> So I figured I can just turn it off to clean up my console. I still
> >> don't know what these VT-d faults are about..
> >
> > What is the 0:02.0 device? Is it being passed in to your guest?
> 
> Nope, there is no pass-through to any guests. The device is:
> 
> 00:02.0 Display controller: Intel Corporation 2nd Generation Core
> Processor Family Integrated Graphics Controller (rev 09)

Let me guess, you have an SandyBridge motherboard and were using
Intel AMT?

I see those all the time on that box (and I think my Haswell
one too). The best I could narrow it down was that the card decided that
certain area should be in the RMRR regions. With Venu/Elen's patch
in (see 431685e8deb660976d8e986c41a647944e410c6c) you should be able
to provide an rmrr paramater to include these addresses.



I see those
> 
> Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Fri, Feb 17, 2017 at 11:48 AM, Konrad Rzeszutek Wilk
 wrote:
> On Fri, Feb 17, 2017 at 11:42:20AM -0700, Tamas K Lengyel wrote:
>> On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin  wrote:
>> >> From: Tian, Kevin
>> >> Sent: Friday, February 17, 2017 11:35 AM
>> >> > >>
>> >> > >> Or wait - do you have the same issue if you use
>> >> > >> "iommu=no,no-intremap"? In which case the problem would be
>> >> > >> that "iommu=no" should clear more than just "iommu_enable", or
>> >> > >> code checking iommu_intremap early (before iommu_setup()
>> >> > >> manages to clear it in the case here) would need to made look at
>> >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
>> >> > >> both variables are clear, which suggests to me that
>> >> > >> iommu_enable is intended to have two different meanings in
>> >> > >> different contexts (master flag vs. controlling just DMA
>> >> > >> remapping). Kevin, Feng - any thoughts here?
>> >> > >
>> >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
>> >> >
>> >> > Thanks for confirming.
>> >> >
>> >> > Kevin, Feng, we now depend on your input regarding the intentions
>> >> > with the two variables.
>> >> >
>> >>
>> >> Feng just left Intel. Let me take a look at code to understand the
>> >> rationale behind.
>> >>
>> >
>> > Jan, looks it's caused by your change back to 2012:
>> >
>> > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
>> > Author: Jan Beulich 
>> > Date:   Fri Nov 2 17:15:30 2012 +0100
>> >
>> > Before that iommu_enable was the master flag consistently. I'm still
>> > trying to understand the background and you may help elaborate if
>> > still something in your memory.
>> >
>> > I agree we should stick to one meaning after clearing up above issue.
>> >
>> > Given this commit is pretty old, I'm also curious why it's only reported
>> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
>> > to be the one upon which you first tried iommu=0 on a platform supporting
>> > interrupt remapping?
>>
>> It just happens to be the one I tried. VT-d was spamming my console
>> with faults like this:
>>
>> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
>> 277e28000, iommu reg = 82c000201000
>> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
>>
>> So I figured I can just turn it off to clean up my console. I still
>> don't know what these VT-d faults are about..
>
> What is the 0:02.0 device? Is it being passed in to your guest?

Nope, there is no pass-through to any guests. The device is:

00:02.0 Display controller: Intel Corporation 2nd Generation Core
Processor Family Integrated Graphics Controller (rev 09)

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Konrad Rzeszutek Wilk]

On Fri, Feb 17, 2017 at 11:42:20AM -0700, Tamas K Lengyel wrote:
> On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin  wrote:
> >> From: Tian, Kevin
> >> Sent: Friday, February 17, 2017 11:35 AM
> >> > >>
> >> > >> Or wait - do you have the same issue if you use
> >> > >> "iommu=no,no-intremap"? In which case the problem would be
> >> > >> that "iommu=no" should clear more than just "iommu_enable", or
> >> > >> code checking iommu_intremap early (before iommu_setup()
> >> > >> manages to clear it in the case here) would need to made look at
> >> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
> >> > >> both variables are clear, which suggests to me that
> >> > >> iommu_enable is intended to have two different meanings in
> >> > >> different contexts (master flag vs. controlling just DMA
> >> > >> remapping). Kevin, Feng - any thoughts here?
> >> > >
> >> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
> >> >
> >> > Thanks for confirming.
> >> >
> >> > Kevin, Feng, we now depend on your input regarding the intentions
> >> > with the two variables.
> >> >
> >>
> >> Feng just left Intel. Let me take a look at code to understand the
> >> rationale behind.
> >>
> >
> > Jan, looks it's caused by your change back to 2012:
> >
> > commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
> > Author: Jan Beulich 
> > Date:   Fri Nov 2 17:15:30 2012 +0100
> >
> > Before that iommu_enable was the master flag consistently. I'm still
> > trying to understand the background and you may help elaborate if
> > still something in your memory.
> >
> > I agree we should stick to one meaning after clearing up above issue.
> >
> > Given this commit is pretty old, I'm also curious why it's only reported
> > on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> > to be the one upon which you first tried iommu=0 on a platform supporting
> > interrupt remapping?
> 
> It just happens to be the one I tried. VT-d was spamming my console
> with faults like this:
> 
> (XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
> 277e28000, iommu reg = 82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> 
> So I figured I can just turn it off to clean up my console. I still
> don't know what these VT-d faults are about..

What is the 0:02.0 device? Is it being passed in to your guest?
> 
> Tamas
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Thu, Feb 16, 2017 at 11:53 PM, Tian, Kevin  wrote:
>> From: Tian, Kevin
>> Sent: Friday, February 17, 2017 11:35 AM
>> > >>
>> > >> Or wait - do you have the same issue if you use
>> > >> "iommu=no,no-intremap"? In which case the problem would be
>> > >> that "iommu=no" should clear more than just "iommu_enable", or
>> > >> code checking iommu_intremap early (before iommu_setup()
>> > >> manages to clear it in the case here) would need to made look at
>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
>> > >> both variables are clear, which suggests to me that
>> > >> iommu_enable is intended to have two different meanings in
>> > >> different contexts (master flag vs. controlling just DMA
>> > >> remapping). Kevin, Feng - any thoughts here?
>> > >
>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
>> >
>> > Thanks for confirming.
>> >
>> > Kevin, Feng, we now depend on your input regarding the intentions
>> > with the two variables.
>> >
>>
>> Feng just left Intel. Let me take a look at code to understand the
>> rationale behind.
>>
>
> Jan, looks it's caused by your change back to 2012:
>
> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
> Author: Jan Beulich 
> Date:   Fri Nov 2 17:15:30 2012 +0100
>
> Before that iommu_enable was the master flag consistently. I'm still
> trying to understand the background and you may help elaborate if
> still something in your memory.
>
> I agree we should stick to one meaning after clearing up above issue.
>
> Given this commit is pretty old, I'm also curious why it's only reported
> on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
> to be the one upon which you first tried iommu=0 on a platform supporting
> interrupt remapping?

It just happens to be the one I tried. VT-d was spamming my console
with faults like this:

(XEN) [VT-D]DMAR:[DMA Write] Request device [:00:02.0] fault addr
277e28000, iommu reg = 82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

So I figured I can just turn it off to clean up my console. I still
don't know what these VT-d faults are about..

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Fri, Feb 17, 2017 at 1:20 AM, Jan Beulich  wrote:
 On 17.02.17 at 07:53,  wrote:
>>>  From: Tian, Kevin
>>> Sent: Friday, February 17, 2017 11:35 AM
>>> > >>
>>> > >> Or wait - do you have the same issue if you use
>>> > >> "iommu=no,no-intremap"? In which case the problem would be
>>> > >> that "iommu=no" should clear more than just "iommu_enable", or
>>> > >> code checking iommu_intremap early (before iommu_setup()
>>> > >> manages to clear it in the case here) would need to made look at
>>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
>>> > >> both variables are clear, which suggests to me that
>>> > >> iommu_enable is intended to have two different meanings in
>>> > >> different contexts (master flag vs. controlling just DMA
>>> > >> remapping). Kevin, Feng - any thoughts here?
>>> > >
>>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
>>> >
>>> > Thanks for confirming.
>>> >
>>> > Kevin, Feng, we now depend on your input regarding the intentions
>>> > with the two variables.
>>> >
>>>
>>> Feng just left Intel. Let me take a look at code to understand the
>>> rationale behind.
>>>
>>
>> Jan, looks it's caused by your change back to 2012:
>>
>> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
>> Author: Jan Beulich 
>> Date:   Fri Nov 2 17:15:30 2012 +0100
>
> Oh, I see. That's been a while; I'm sorry for not having remembered
> and bugging you.
>
>> Before that iommu_enable was the master flag consistently. I'm still
>> trying to understand the background and you may help elaborate if
>> still something in your memory.
>
> I think the commit description is pretty clear. Especially this part
>
> "This could have the nice side effect of allowing to use "iommu=off"
>  even when x2APIC was pre-enabled by the BIOS (in which case interrupt
>  remapping is a requirement, but DMA translation [obviously] isn't), but
>  that doesn't currently work (and hence x2apic_bsp_setup() forces the
>  IOMMU on rather than just interrupt remapping)."
>
> is quite relevant in the context here. Just like Linux, we really ought
> to have a way to run with interrupt remapping, but without DMA
> remapping. So I think this old commit was a half-hearted step into
> that direction, not recognizing that it would break some other case.
>
> So I think the only reasonable way forward to address Tamas's
> issue is to fully disentangle DMA and interrupt remapping setup,
> which I'm sure is going to take some time. As a minimal adjustment,
> though, I wonder whether that old commit's adjustment to apic.c
> shouldn't rather have set force_intremap. Tamas - could you check
> whether
>
> --- a/xen/arch/x86/apic.c
> +++ b/xen/arch/x86/apic.c
> @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void)
>  goto restore_out;
>  }
>
> -force_iommu = 1;
> +force_intremap = 1;
>
>  genapic = apic_x2apic_probe();
>  printk("Switched to APIC driver %s.\n", genapic->name);
>
> makes things any better?
>
> Jan

Just making this change won't compile:

apic.c: In function ‘x2apic_bsp_setup’:
apic.c:947:5: error: ‘force_intremap’ undeclared (first use in this function)
 force_intremap = 1;
 ^

Had to also add

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index d793f5de1a..a45578608b 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -52,6 +52,7 @@ custom_param("iommu", parse_iommu_param);
 bool_t __initdata iommu_enable = 1;
 bool_t __read_mostly iommu_enabled;
 bool_t __read_mostly force_iommu;
+bool_t __read_mostly force_intremap;
 bool_t __hwdom_initdata iommu_dom0_strict;
 bool_t __read_mostly iommu_verbose;
 bool_t __read_mostly iommu_workaround_bios_bug;
@@ -364,7 +365,7 @@ int iommu_iotlb_flush_all(struct domain *d)
 int __init iommu_setup(void)
 {
 int rc = -ENODEV;
-bool_t force_intremap = force_iommu && iommu_intremap;
+force_intremap = force_iommu && iommu_intremap;

 if ( iommu_dom0_strict )
 iommu_passthrough = 0;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 5803e3f95b..770e4f6282 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -29,6 +29,7 @@

 extern bool_t iommu_enable, iommu_enabled;
 extern bool_t force_iommu, iommu_verbose;
+extern bool_t force_intremap;
 extern bool_t iommu_workaround_bios_bug, iommu_igfx, iommu_passthrough;
 extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
 extern bool_t iommu_hap_pt_share;

With these changes in place the system starts to boot but dom0 fails
to assemble the mdraid disk and find root:

Begin: Running /scripts/local-block ... mdadm: No devices listed in
conf file were found.
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  Volume group "vg" not found
  Cannot process volume group vg
done.
done.
Gave up waiting for root device.  Common problems:
 - Boot args (cat 

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Jan Beulich]

>>> On 17.02.17 at 07:53,  wrote:
>>  From: Tian, Kevin
>> Sent: Friday, February 17, 2017 11:35 AM
>> > >>
>> > >> Or wait - do you have the same issue if you use
>> > >> "iommu=no,no-intremap"? In which case the problem would be
>> > >> that "iommu=no" should clear more than just "iommu_enable", or
>> > >> code checking iommu_intremap early (before iommu_setup()
>> > >> manages to clear it in the case here) would need to made look at
>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
>> > >> both variables are clear, which suggests to me that
>> > >> iommu_enable is intended to have two different meanings in
>> > >> different contexts (master flag vs. controlling just DMA
>> > >> remapping). Kevin, Feng - any thoughts here?
>> > >
>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
>> >
>> > Thanks for confirming.
>> >
>> > Kevin, Feng, we now depend on your input regarding the intentions
>> > with the two variables.
>> >
>> 
>> Feng just left Intel. Let me take a look at code to understand the
>> rationale behind.
>> 
> 
> Jan, looks it's caused by your change back to 2012:
> 
> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
> Author: Jan Beulich 
> Date:   Fri Nov 2 17:15:30 2012 +0100

Oh, I see. That's been a while; I'm sorry for not having remembered
and bugging you.

> Before that iommu_enable was the master flag consistently. I'm still
> trying to understand the background and you may help elaborate if
> still something in your memory.

I think the commit description is pretty clear. Especially this part

"This could have the nice side effect of allowing to use "iommu=off"
 even when x2APIC was pre-enabled by the BIOS (in which case interrupt
 remapping is a requirement, but DMA translation [obviously] isn't), but
 that doesn't currently work (and hence x2apic_bsp_setup() forces the
 IOMMU on rather than just interrupt remapping)."

is quite relevant in the context here. Just like Linux, we really ought
to have a way to run with interrupt remapping, but without DMA
remapping. So I think this old commit was a half-hearted step into
that direction, not recognizing that it would break some other case.

So I think the only reasonable way forward to address Tamas's
issue is to fully disentangle DMA and interrupt remapping setup,
which I'm sure is going to take some time. As a minimal adjustment,
though, I wonder whether that old commit's adjustment to apic.c
shouldn't rather have set force_intremap. Tamas - could you check
whether

--- a/xen/arch/x86/apic.c
+++ b/xen/arch/x86/apic.c
@@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void)
 goto restore_out;
 }
 
-force_iommu = 1;
+force_intremap = 1;
 
 genapic = apic_x2apic_probe();
 printk("Switched to APIC driver %s.\n", genapic->name);

makes things any better?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Jan Beulich]

>>> On 17.02.17 at 04:35,  wrote:
> Feng just left Intel. Let me take a look at code to understand the
> rationale behind.

Do you know if he means to stay VT-d maintainer? If yes, do you
have a new email address of him? If not, would you mind sending
an update to ./MAINTAINERS removing him?

Thanks, Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Friday, February 17, 2017 11:35 AM
> > >>
> > >> Or wait - do you have the same issue if you use
> > >> "iommu=no,no-intremap"? In which case the problem would be
> > >> that "iommu=no" should clear more than just "iommu_enable", or
> > >> code checking iommu_intremap early (before iommu_setup()
> > >> manages to clear it in the case here) would need to made look at
> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
> > >> both variables are clear, which suggests to me that
> > >> iommu_enable is intended to have two different meanings in
> > >> different contexts (master flag vs. controlling just DMA
> > >> remapping). Kevin, Feng - any thoughts here?
> > >
> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
> >
> > Thanks for confirming.
> >
> > Kevin, Feng, we now depend on your input regarding the intentions
> > with the two variables.
> >
> 
> Feng just left Intel. Let me take a look at code to understand the
> rationale behind.
> 

Jan, looks it's caused by your change back to 2012:

commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
Author: Jan Beulich 
Date:   Fri Nov 2 17:15:30 2012 +0100

Before that iommu_enable was the master flag consistently. I'm still
trying to understand the background and you may help elaborate if
still something in your memory.

I agree we should stick to one meaning after clearing up above issue.

Given this commit is pretty old, I'm also curious why it's only reported
on 4.8. Tamas, did you succeed with iommu=0 pre 4.8, or 4.8 happens
to be the one upon which you first tried iommu=0 on a platform supporting
interrupt remapping?

Thanks
Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tian, Kevin]

> From: Jan Beulich [mailto:jbeul...@suse.com]
> Sent: Thursday, February 16, 2017 5:32 PM
> 
> >>> On 15.02.17 at 20:30,  wrote:
> > On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich  wrote:
> > On 15.02.17 at 00:21,  wrote:
> >>> On 14/02/2017 22:47, Tamas K Lengyel wrote:
>  (XEN) Switched to APIC driver x2apic_cluster.
>  (XEN) XSM Framework v1.0.0 initialized
>  (XEN) Flask: 128 avtab hash slots, 394 rules.
>  (XEN) Flask: 128 avtab hash slots, 394 rules.
>  (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
>  (XEN) Flask:  12 classes, 394 rules
>  (XEN) Flask:  Starting in enforcing mode.
>  (XEN) xstate: size: 0x340 and states: 0x7
>  (XEN) Intel machine check reporting enabled
>  (XEN) Using scheduler: SMP Credit Scheduler (credit)
>  (XEN) Platform timer is 14.318MHz HPET
>  (XEN) Detected 3392.326 MHz processor.
>  (XEN) Initing memory sharing.
>  (XEN) alt table 82d0802d3f38 -> 82d0802d5564
>  (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
>  (XEN) PCI: Not using MCFG for segment  bus 00-3f
>  (XEN)
>  (XEN) 
>  (XEN) Panic on CPU 0:
>  (XEN) Couldn't enable IOMMU and iommu=required/force
>  (XEN) 
>  (XEN)
>  (XEN) Reboot in five seconds...
>  (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
> 
>  As seen in the command line iommu is not set to required or forced.
>  Yet Xen thinks it is set to required/force. Has anyone else run into
>  this issue?
> >>>
> >>> This area is a rats nest :(
> >>>
> >>> The problem is that the APIC setup has chosen to use the x2apic_cluster
> >>> driver, which in the Xen code depends on x2APIC, which depends on
> >>> interrupt remapping, which depends on an IOMMU.
> >>>
> >>> (I could have sworn we fixed this before), but the bug is that the APIC
> >>> setup shouldn't choose any of the x2apic modes if there isn't an
> >>> interrupt remapping capable IOMMU.
> >>
> >> And from going over the code I can't see how this would happen,
> >> so Tamas, it would be nice if you could add some verbosity to the
> >> functions involved. In particular x2apic_bsp_setup() must be
> >> getting success (zero) from iommu_enable_x2apic_IR(), yet that
> >> function calls iommu_supports_eim(), which ought to produce
> >> false through its very first exit path in your case.
> >>
> >> Or wait - do you have the same issue if you use
> >> "iommu=no,no-intremap"? In which case the problem would be
> >> that "iommu=no" should clear more than just "iommu_enable", or
> >> code checking iommu_intremap early (before iommu_setup()
> >> manages to clear it in the case here) would need to made look at
> >> both variables. Oddly enough acpi_parse_dmar() only bails if
> >> both variables are clear, which suggests to me that
> >> iommu_enable is intended to have two different meanings in
> >> different contexts (master flag vs. controlling just DMA
> >> remapping). Kevin, Feng - any thoughts here?
> >
> > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
> 
> Thanks for confirming.
> 
> Kevin, Feng, we now depend on your input regarding the intentions
> with the two variables.
> 

Feng just left Intel. Let me take a look at code to understand the
rationale behind.

Thanks
Kevin

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Jan Beulich]

>>> On 15.02.17 at 20:30,  wrote:
> On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich  wrote:
> On 15.02.17 at 00:21,  wrote:
>>> On 14/02/2017 22:47, Tamas K Lengyel wrote:
 (XEN) Switched to APIC driver x2apic_cluster.
 (XEN) XSM Framework v1.0.0 initialized
 (XEN) Flask: 128 avtab hash slots, 394 rules.
 (XEN) Flask: 128 avtab hash slots, 394 rules.
 (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
 (XEN) Flask:  12 classes, 394 rules
 (XEN) Flask:  Starting in enforcing mode.
 (XEN) xstate: size: 0x340 and states: 0x7
 (XEN) Intel machine check reporting enabled
 (XEN) Using scheduler: SMP Credit Scheduler (credit)
 (XEN) Platform timer is 14.318MHz HPET
 (XEN) Detected 3392.326 MHz processor.
 (XEN) Initing memory sharing.
 (XEN) alt table 82d0802d3f38 -> 82d0802d5564
 (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
 (XEN) PCI: Not using MCFG for segment  bus 00-3f
 (XEN)
 (XEN) 
 (XEN) Panic on CPU 0:
 (XEN) Couldn't enable IOMMU and iommu=required/force
 (XEN) 
 (XEN)
 (XEN) Reboot in five seconds...
 (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

 As seen in the command line iommu is not set to required or forced.
 Yet Xen thinks it is set to required/force. Has anyone else run into
 this issue?
>>>
>>> This area is a rats nest :(
>>>
>>> The problem is that the APIC setup has chosen to use the x2apic_cluster
>>> driver, which in the Xen code depends on x2APIC, which depends on
>>> interrupt remapping, which depends on an IOMMU.
>>>
>>> (I could have sworn we fixed this before), but the bug is that the APIC
>>> setup shouldn't choose any of the x2apic modes if there isn't an
>>> interrupt remapping capable IOMMU.
>>
>> And from going over the code I can't see how this would happen,
>> so Tamas, it would be nice if you could add some verbosity to the
>> functions involved. In particular x2apic_bsp_setup() must be
>> getting success (zero) from iommu_enable_x2apic_IR(), yet that
>> function calls iommu_supports_eim(), which ought to produce
>> false through its very first exit path in your case.
>>
>> Or wait - do you have the same issue if you use
>> "iommu=no,no-intremap"? In which case the problem would be
>> that "iommu=no" should clear more than just "iommu_enable", or
>> code checking iommu_intremap early (before iommu_setup()
>> manages to clear it in the case here) would need to made look at
>> both variables. Oddly enough acpi_parse_dmar() only bails if
>> both variables are clear, which suggests to me that
>> iommu_enable is intended to have two different meanings in
>> different contexts (master flag vs. controlling just DMA
>> remapping). Kevin, Feng - any thoughts here?
> 
> iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"

Thanks for confirming.

Kevin, Feng, we now depend on your input regarding the intentions
with the two variables.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Wed, Feb 15, 2017 at 12:30 PM, Tamas K Lengyel
 wrote:
> On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich  wrote:
> On 15.02.17 at 00:21,  wrote:
>>> On 14/02/2017 22:47, Tamas K Lengyel wrote:
 (XEN) Switched to APIC driver x2apic_cluster.
 (XEN) XSM Framework v1.0.0 initialized
 (XEN) Flask: 128 avtab hash slots, 394 rules.
 (XEN) Flask: 128 avtab hash slots, 394 rules.
 (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
 (XEN) Flask:  12 classes, 394 rules
 (XEN) Flask:  Starting in enforcing mode.
 (XEN) xstate: size: 0x340 and states: 0x7
 (XEN) Intel machine check reporting enabled
 (XEN) Using scheduler: SMP Credit Scheduler (credit)
 (XEN) Platform timer is 14.318MHz HPET
 (XEN) Detected 3392.326 MHz processor.
 (XEN) Initing memory sharing.
 (XEN) alt table 82d0802d3f38 -> 82d0802d5564
 (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
 (XEN) PCI: Not using MCFG for segment  bus 00-3f
 (XEN)
 (XEN) 
 (XEN) Panic on CPU 0:
 (XEN) Couldn't enable IOMMU and iommu=required/force
 (XEN) 
 (XEN)
 (XEN) Reboot in five seconds...
 (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

 As seen in the command line iommu is not set to required or forced.
 Yet Xen thinks it is set to required/force. Has anyone else run into
 this issue?
>>>
>>> This area is a rats nest :(
>>>
>>> The problem is that the APIC setup has chosen to use the x2apic_cluster
>>> driver, which in the Xen code depends on x2APIC, which depends on
>>> interrupt remapping, which depends on an IOMMU.
>>>
>>> (I could have sworn we fixed this before), but the bug is that the APIC
>>> setup shouldn't choose any of the x2apic modes if there isn't an
>>> interrupt remapping capable IOMMU.
>>
>> And from going over the code I can't see how this would happen,
>> so Tamas, it would be nice if you could add some verbosity to the
>> functions involved. In particular x2apic_bsp_setup() must be
>> getting success (zero) from iommu_enable_x2apic_IR(), yet that
>> function calls iommu_supports_eim(), which ought to produce
>> false through its very first exit path in your case.
>>
>> Or wait - do you have the same issue if you use
>> "iommu=no,no-intremap"? In which case the problem would be
>> that "iommu=no" should clear more than just "iommu_enable", or
>> code checking iommu_intremap early (before iommu_setup()
>> manages to clear it in the case here) would need to made look at
>> both variables. Oddly enough acpi_parse_dmar() only bails if
>> both variables are clear, which suggests to me that
>> iommu_enable is intended to have two different meanings in
>> different contexts (master flag vs. controlling just DMA
>> remapping). Kevin, Feng - any thoughts here?
>
> iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"

And just to clarify, x2apic is disabled in this case "(XEN) Not
enabling x2APIC: depends on iommu_supports_eim."

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Tamas K Lengyel]

On Wed, Feb 15, 2017 at 3:03 AM, Jan Beulich  wrote:
 On 15.02.17 at 00:21,  wrote:
>> On 14/02/2017 22:47, Tamas K Lengyel wrote:
>>> (XEN) Switched to APIC driver x2apic_cluster.
>>> (XEN) XSM Framework v1.0.0 initialized
>>> (XEN) Flask: 128 avtab hash slots, 394 rules.
>>> (XEN) Flask: 128 avtab hash slots, 394 rules.
>>> (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
>>> (XEN) Flask:  12 classes, 394 rules
>>> (XEN) Flask:  Starting in enforcing mode.
>>> (XEN) xstate: size: 0x340 and states: 0x7
>>> (XEN) Intel machine check reporting enabled
>>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>>> (XEN) Platform timer is 14.318MHz HPET
>>> (XEN) Detected 3392.326 MHz processor.
>>> (XEN) Initing memory sharing.
>>> (XEN) alt table 82d0802d3f38 -> 82d0802d5564
>>> (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
>>> (XEN) PCI: Not using MCFG for segment  bus 00-3f
>>> (XEN)
>>> (XEN) 
>>> (XEN) Panic on CPU 0:
>>> (XEN) Couldn't enable IOMMU and iommu=required/force
>>> (XEN) 
>>> (XEN)
>>> (XEN) Reboot in five seconds...
>>> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
>>>
>>> As seen in the command line iommu is not set to required or forced.
>>> Yet Xen thinks it is set to required/force. Has anyone else run into
>>> this issue?
>>
>> This area is a rats nest :(
>>
>> The problem is that the APIC setup has chosen to use the x2apic_cluster
>> driver, which in the Xen code depends on x2APIC, which depends on
>> interrupt remapping, which depends on an IOMMU.
>>
>> (I could have sworn we fixed this before), but the bug is that the APIC
>> setup shouldn't choose any of the x2apic modes if there isn't an
>> interrupt remapping capable IOMMU.
>
> And from going over the code I can't see how this would happen,
> so Tamas, it would be nice if you could add some verbosity to the
> functions involved. In particular x2apic_bsp_setup() must be
> getting success (zero) from iommu_enable_x2apic_IR(), yet that
> function calls iommu_supports_eim(), which ought to produce
> false through its very first exit path in your case.
>
> Or wait - do you have the same issue if you use
> "iommu=no,no-intremap"? In which case the problem would be
> that "iommu=no" should clear more than just "iommu_enable", or
> code checking iommu_intremap early (before iommu_setup()
> manages to clear it in the case here) would need to made look at
> both variables. Oddly enough acpi_parse_dmar() only bails if
> both variables are clear, which suggests to me that
> iommu_enable is intended to have two different meanings in
> different contexts (master flag vs. controlling just DMA
> remapping). Kevin, Feng - any thoughts here?

iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"

Tamas

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Jan Beulich]

>>> On 15.02.17 at 00:21,  wrote:
> On 14/02/2017 22:47, Tamas K Lengyel wrote:
>> (XEN) Switched to APIC driver x2apic_cluster.
>> (XEN) XSM Framework v1.0.0 initialized
>> (XEN) Flask: 128 avtab hash slots, 394 rules.
>> (XEN) Flask: 128 avtab hash slots, 394 rules.
>> (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
>> (XEN) Flask:  12 classes, 394 rules
>> (XEN) Flask:  Starting in enforcing mode.
>> (XEN) xstate: size: 0x340 and states: 0x7
>> (XEN) Intel machine check reporting enabled
>> (XEN) Using scheduler: SMP Credit Scheduler (credit)
>> (XEN) Platform timer is 14.318MHz HPET
>> (XEN) Detected 3392.326 MHz processor.
>> (XEN) Initing memory sharing.
>> (XEN) alt table 82d0802d3f38 -> 82d0802d5564
>> (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
>> (XEN) PCI: Not using MCFG for segment  bus 00-3f
>> (XEN)
>> (XEN) 
>> (XEN) Panic on CPU 0:
>> (XEN) Couldn't enable IOMMU and iommu=required/force
>> (XEN) 
>> (XEN)
>> (XEN) Reboot in five seconds...
>> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
>>
>> As seen in the command line iommu is not set to required or forced.
>> Yet Xen thinks it is set to required/force. Has anyone else run into
>> this issue?
> 
> This area is a rats nest :(
> 
> The problem is that the APIC setup has chosen to use the x2apic_cluster
> driver, which in the Xen code depends on x2APIC, which depends on
> interrupt remapping, which depends on an IOMMU.
> 
> (I could have sworn we fixed this before), but the bug is that the APIC
> setup shouldn't choose any of the x2apic modes if there isn't an
> interrupt remapping capable IOMMU.

And from going over the code I can't see how this would happen,
so Tamas, it would be nice if you could add some verbosity to the
functions involved. In particular x2apic_bsp_setup() must be
getting success (zero) from iommu_enable_x2apic_IR(), yet that
function calls iommu_supports_eim(), which ought to produce
false through its very first exit path in your case.

Or wait - do you have the same issue if you use
"iommu=no,no-intremap"? In which case the problem would be
that "iommu=no" should clear more than just "iommu_enable", or
code checking iommu_intremap early (before iommu_setup()
manages to clear it in the case here) would need to made look at
both variables. Oddly enough acpi_parse_dmar() only bails if
both variables are clear, which suggests to me that
iommu_enable is intended to have two different meanings in
different contexts (master flag vs. controlling just DMA
remapping). Kevin, Feng - any thoughts here?

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0

[Andrew Cooper]

On 14/02/2017 22:47, Tamas K Lengyel wrote:
> (XEN) Switched to APIC driver x2apic_cluster.
> (XEN) XSM Framework v1.0.0 initialized
> (XEN) Flask: 128 avtab hash slots, 394 rules.
> (XEN) Flask: 128 avtab hash slots, 394 rules.
> (XEN) Flask:  5 users, 3 roles, 43 types, 2 bools
> (XEN) Flask:  12 classes, 394 rules
> (XEN) Flask:  Starting in enforcing mode.
> (XEN) xstate: size: 0x340 and states: 0x7
> (XEN) Intel machine check reporting enabled
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Detected 3392.326 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) alt table 82d0802d3f38 -> 82d0802d5564
> (XEN) PCI: MCFG configuration 0: base e000 segment  buses 00 - 3f
> (XEN) PCI: Not using MCFG for segment  bus 00-3f
> (XEN)
> (XEN) 
> (XEN) Panic on CPU 0:
> (XEN) Couldn't enable IOMMU and iommu=required/force
> (XEN) 
> (XEN)
> (XEN) Reboot in five seconds...
> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
>
> As seen in the command line iommu is not set to required or forced.
> Yet Xen thinks it is set to required/force. Has anyone else run into
> this issue?

This area is a rats nest :(

The problem is that the APIC setup has chosen to use the x2apic_cluster
driver, which in the Xen code depends on x2APIC, which depends on
interrupt remapping, which depends on an IOMMU.

(I could have sworn we fixed this before), but the bug is that the APIC
setup shouldn't choose any of the x2apic modes if there isn't an
interrupt remapping capable IOMMU.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel