Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[M. Vefa Bicakci]

On 11/02/2016 08:23 PM, Sebastian Andrzej Siewior wrote:
> On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
>> Hello Sebastian,
>
> Hi,
> 
>> The patch fixes the kernel oops for me.
>>
>> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
>> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
>> the allocation of package identifiers for the virtual CPUs.
>>
>> Prior to your patch, my Xen-based virtual machines would intermittently
>> crash most of the time at boot-up with the backtrace reported by Charles.
>> Due to this, I was under the impression that this is a subtle race
>> condition.
> 
> how hard is it to get such a xen setup up and running?

Hello Sebastian,

Sorry about my late reply!

The set-up I use is a bit involved/complicated. To replicate it, you
would need to install Qubes OS R3.2 (assuming that you have compatible
hardware with a lot of RAM) and then build a custom 4.8.y-based kernel
with a set of cherry-picked commits. After installing this kernel in
dom0 with dnf or rpm, you would need to run:

  # Generate a domU initrd and copy the kernel image and the generated
  # initrd to Qubes OS's domU kernel directory (/var/lib/qubes/...)
  $ sudo /usr/sbin/qubes-prepare-vm-kernel 

  # From now on, use kernel_version when starting domU instances.
  $ qubes-prefs -s default-kernel 

Afterwards, starting a domU (i.e., AppVM in Qubes OS terminology) should
exhibit the issue in question related to RAPL:

  $ sudo truncate -s0 /var/log/xen/console/guest-.log
  $ qvm-start --debug 
  $ cat /var/log/xen/console/guest-.log

As you may appreciate, the set-up is a bit involved. Nevertheless, in
case you would like to replicate my set-up, I can try to publish my
linux-4.8.y-based git branch so that you can build a similar kernel as
the one I use.

Vefa

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[M. Vefa Bicakci]

On 11/02/2016 08:23 PM, Sebastian Andrzej Siewior wrote:
> On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
>> Hello Sebastian,
>
> Hi,
> 
>> The patch fixes the kernel oops for me.
>>
>> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
>> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
>> the allocation of package identifiers for the virtual CPUs.
>>
>> Prior to your patch, my Xen-based virtual machines would intermittently
>> crash most of the time at boot-up with the backtrace reported by Charles.
>> Due to this, I was under the impression that this is a subtle race
>> condition.
> 
> how hard is it to get such a xen setup up and running?

Hello Sebastian,

Sorry about my late reply!

The set-up I use is a bit involved/complicated. To replicate it, you
would need to install Qubes OS R3.2 (assuming that you have compatible
hardware with a lot of RAM) and then build a custom 4.8.y-based kernel
with a set of cherry-picked commits. After installing this kernel in
dom0 with dnf or rpm, you would need to run:

  # Generate a domU initrd and copy the kernel image and the generated
  # initrd to Qubes OS's domU kernel directory (/var/lib/qubes/...)
  $ sudo /usr/sbin/qubes-prepare-vm-kernel 

  # From now on, use kernel_version when starting domU instances.
  $ qubes-prefs -s default-kernel 

Afterwards, starting a domU (i.e., AppVM in Qubes OS terminology) should
exhibit the issue in question related to RAPL:

  $ sudo truncate -s0 /var/log/xen/console/guest-.log
  $ qvm-start --debug 
  $ cat /var/log/xen/console/guest-.log

As you may appreciate, the set-up is a bit involved. Nevertheless, in
case you would like to replicate my set-up, I can try to publish my
linux-4.8.y-based git branch so that you can build a similar kernel as
the one I use.

Vefa

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
> Hello Sebastian,
Hi,

> The patch fixes the kernel oops for me.
> 
> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
> the allocation of package identifiers for the virtual CPUs.
> 
> Prior to your patch, my Xen-based virtual machines would intermittently
> crash most of the time at boot-up with the backtrace reported by Charles.
> Due to this, I was under the impression that this is a subtle race
> condition.

how hard is it to get such a xen setup up and running?

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-11-01 13:15:53 [+0300], M. Vefa Bicakci wrote:
> Hello Sebastian,
Hi,

> The patch fixes the kernel oops for me.
> 
> I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
> on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
> the allocation of package identifiers for the virtual CPUs.
> 
> Prior to your patch, my Xen-based virtual machines would intermittently
> crash most of the time at boot-up with the backtrace reported by Charles.
> Due to this, I was under the impression that this is a subtle race
> condition.

how hard is it to get such a xen setup up and running?

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-11-02 05:16:03 [-0400], Charles (Chas) Williams wrote:
> Yes, it does prevent RAPL from starting and loading.  From the boot log:
please send the whole bootlog. offlist if you want.

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-11-02 05:16:03 [-0400], Charles (Chas) Williams wrote:
> Yes, it does prevent RAPL from starting and loading.  From the boot log:
please send the whole bootlog. offlist if you want.

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/28/2016 04:03 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:

I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.


But sometimes the topology info is correct and if I get lucky, the
package id could be valid for all the CPU's.  Given the behavior,
I have seen so far it makes me thing the RAPL isn't being emulated.
So even if I did boot onto a "valid" set of cores, would I always be
certain that I will be on those cores?


I don't what vmware does here. Nor do they ship source to check. So if
you have a big HW box with say two packages, it might make sense to give
this information to the guest _if_ the CPUs are pinned and the guest
never migrates.


Yes, I agree _if_.  That's why it simply isn't clear to me that we should
attempt do any RAPL at all for VMWare.  The current behavior doesn't seem
to make sense and I don't expect it to suddenly start acting reasonable.
Since I don't understand why some package id's are valid and others
are not, I would prefer not to trust any of the information as far as
enabling/disabling the RAPL monitoring.




Per your request in your next email:


One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).


Our previous kernel was 4.4, and didn't use the logical package id:

I see.

Did the patch I sent fixed it for you and were you not able to test?


Yes, it does prevent RAPL from starting and loading.  From the boot log:

[2.711481] RAPL PMU: rapl pmu error: max package: 4 but CPU2 belongs to 
65535
[2.711639] rapl pmu error: max package: 4 but CPU2 belongs to 65535

This was consistent across several reboots.  I poked around in the
VM settings.  Apparently this guest is configured for four virtual
sockets with one core per socket.  Testing with two virtual sockets,
one core per socket:

[2.163177] RAPL PMU: rapl pmu error: max package: 2 but CPU1 belongs to 
65535
[2.163304] rapl pmu error: max package: 2 but CPU1 belongs to 65535

Booting with 1 virtual socket, 1 core per socket:

[1.750311] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 
10737418240 ms ovfl timer
[1.750312] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[1.750313] RAPL PMU: hw unit of domain package 2^-0 Joules
[1.750314] RAPL PMU: hw unit of domain dram 2^-0 Joules

Booting with 1 virtual socket, 4 cores per socket:

[3.527298] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 
10737418240 ms ovfl timer
[3.527302] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[3.527304] RAPL PMU: hw unit of domain package 2^-0 Joules
[3.527307] RAPL PMU: hw unit of domain dram 2^-0 Joules

So, it looks like VMWare tends to always get something wrong if you have
more than one virtual socket.  The above behavior was consistent across
several reboots.

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/28/2016 04:03 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:

I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.


But sometimes the topology info is correct and if I get lucky, the
package id could be valid for all the CPU's.  Given the behavior,
I have seen so far it makes me thing the RAPL isn't being emulated.
So even if I did boot onto a "valid" set of cores, would I always be
certain that I will be on those cores?


I don't what vmware does here. Nor do they ship source to check. So if
you have a big HW box with say two packages, it might make sense to give
this information to the guest _if_ the CPUs are pinned and the guest
never migrates.


Yes, I agree _if_.  That's why it simply isn't clear to me that we should
attempt do any RAPL at all for VMWare.  The current behavior doesn't seem
to make sense and I don't expect it to suddenly start acting reasonable.
Since I don't understand why some package id's are valid and others
are not, I would prefer not to trust any of the information as far as
enabling/disabling the RAPL monitoring.




Per your request in your next email:


One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).


Our previous kernel was 4.4, and didn't use the logical package id:

I see.

Did the patch I sent fixed it for you and were you not able to test?


Yes, it does prevent RAPL from starting and loading.  From the boot log:

[2.711481] RAPL PMU: rapl pmu error: max package: 4 but CPU2 belongs to 
65535
[2.711639] rapl pmu error: max package: 4 but CPU2 belongs to 65535

This was consistent across several reboots.  I poked around in the
VM settings.  Apparently this guest is configured for four virtual
sockets with one core per socket.  Testing with two virtual sockets,
one core per socket:

[2.163177] RAPL PMU: rapl pmu error: max package: 2 but CPU1 belongs to 
65535
[2.163304] rapl pmu error: max package: 2 but CPU1 belongs to 65535

Booting with 1 virtual socket, 1 core per socket:

[1.750311] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 
10737418240 ms ovfl timer
[1.750312] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[1.750313] RAPL PMU: hw unit of domain package 2^-0 Joules
[1.750314] RAPL PMU: hw unit of domain dram 2^-0 Joules

Booting with 1 virtual socket, 4 cores per socket:

[3.527298] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 
10737418240 ms ovfl timer
[3.527302] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[3.527304] RAPL PMU: hw unit of domain package 2^-0 Joules
[3.527307] RAPL PMU: hw unit of domain dram 2^-0 Joules

So, it looks like VMWare tends to always get something wrong if you have
more than one virtual socket.  The above behavior was consistent across
several reboots.

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[M. Vefa Bicakci]

> On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
>>
>> [snip]
>>
>> But sometimes the topology info is correct and if I get lucky, the
>> package id could be valid for all the CPU's.  Given the behavior,
>> I have seen so far it makes me thing the RAPL isn't being emulated.
>> So even if I did boot onto a "valid" set of cores, would I always be
>> certain that I will be on those cores?
> 
> I don't what vmware does here. Nor do they ship source to check. So if
> you have a big HW box with say two packages, it might make sense to give
> this information to the guest _if_ the CPUs are pinned and the guest
> never migrates.
> 
>> Per your request in your next email:
>> 
>> > One thing I forgot to ask: Could you please check if you get the same
>> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
>> > rework).
>> 
>> Our previous kernel was 4.4, and didn't use the logical package id:
>
> I see.
> 
> Did the patch I sent fixed it for you and were you not able to test?

Hello Sebastian,

The patch fixes the kernel oops for me.

I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
the allocation of package identifiers for the virtual CPUs.

Prior to your patch, my Xen-based virtual machines would intermittently
crash most of the time at boot-up with the backtrace reported by Charles.
Due to this, I was under the impression that this is a subtle race
condition.

With your patch, the virtual machines boot-up successfully, all the time.
Here are the relevant excerpts from dmesg:

=== 8< ===
[0.263936] RAPL PMU: rapl pmu error: max package: 1 but CPU0 belongs to 
65535
...
[2.213669] intel_rapl: Found RAPL domain package
[2.213689] intel_rapl: Found RAPL domain core
[2.216337] intel_rapl: Found RAPL domain uncore
[2.216370] intel_rapl: RAPL package 0 domain package locked by BIOS
=== >8 ===

Thank you,

Vefa

Please note: I am not subscribed to the Linux kernel mailing list, so
I had to manually construct the headers of this reply with the proper
In-Reply-To and References values (which were extracted from marc.info).
As a result, this e-mail may not show up as a reply to your earlier
conversation with Charles.

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[M. Vefa Bicakci]

> On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
>>
>> [snip]
>>
>> But sometimes the topology info is correct and if I get lucky, the
>> package id could be valid for all the CPU's.  Given the behavior,
>> I have seen so far it makes me thing the RAPL isn't being emulated.
>> So even if I did boot onto a "valid" set of cores, would I always be
>> certain that I will be on those cores?
> 
> I don't what vmware does here. Nor do they ship source to check. So if
> you have a big HW box with say two packages, it might make sense to give
> this information to the guest _if_ the CPUs are pinned and the guest
> never migrates.
> 
>> Per your request in your next email:
>> 
>> > One thing I forgot to ask: Could you please check if you get the same
>> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
>> > rework).
>> 
>> Our previous kernel was 4.4, and didn't use the logical package id:
>
> I see.
> 
> Did the patch I sent fixed it for you and were you not able to test?

Hello Sebastian,

The patch fixes the kernel oops for me.

I am using a custom 4.8.5-based kernel on Qubes OS R3.2, which is based
on Xen 4.6.3. Apparently, Xen also has a similar bug/flaw/quirk regarding
the allocation of package identifiers for the virtual CPUs.

Prior to your patch, my Xen-based virtual machines would intermittently
crash most of the time at boot-up with the backtrace reported by Charles.
Due to this, I was under the impression that this is a subtle race
condition.

With your patch, the virtual machines boot-up successfully, all the time.
Here are the relevant excerpts from dmesg:

=== 8< ===
[0.263936] RAPL PMU: rapl pmu error: max package: 1 but CPU0 belongs to 
65535
...
[2.213669] intel_rapl: Found RAPL domain package
[2.213689] intel_rapl: Found RAPL domain core
[2.216337] intel_rapl: Found RAPL domain uncore
[2.216370] intel_rapl: RAPL package 0 domain package locked by BIOS
=== >8 ===

Thank you,

Vefa

Please note: I am not subscribed to the Linux kernel mailing list, so
I had to manually construct the headers of this reply with the proper
In-Reply-To and References values (which were extracted from marc.info).
As a result, this e-mail may not show up as a reply to your earlier
conversation with Charles.

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
> > I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
> > topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
> > and 3. That -1 comes probably from topology_update_package_map(). Could
> > you please send a complete boot log and try the following patch? This
> > one should fix your boot problem and disable RAPL if the info is
> > invalid.
> 
> But sometimes the topology info is correct and if I get lucky, the
> package id could be valid for all the CPU's.  Given the behavior,
> I have seen so far it makes me thing the RAPL isn't being emulated.
> So even if I did boot onto a "valid" set of cores, would I always be
> certain that I will be on those cores?

I don't what vmware does here. Nor do they ship source to check. So if
you have a big HW box with say two packages, it might make sense to give
this information to the guest _if_ the CPUs are pinned and the guest
never migrates.

> Per your request in your next email:
> 
> > One thing I forgot to ask: Could you please check if you get the same
> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
> > rework).
> 
> Our previous kernel was 4.4, and didn't use the logical package id:
I see.

Did the patch I sent fixed it for you and were you not able to test?

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-27 15:00:32 [-0400], Charles (Chas) Williams wrote:
> > I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
> > topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
> > and 3. That -1 comes probably from topology_update_package_map(). Could
> > you please send a complete boot log and try the following patch? This
> > one should fix your boot problem and disable RAPL if the info is
> > invalid.
> 
> But sometimes the topology info is correct and if I get lucky, the
> package id could be valid for all the CPU's.  Given the behavior,
> I have seen so far it makes me thing the RAPL isn't being emulated.
> So even if I did boot onto a "valid" set of cores, would I always be
> certain that I will be on those cores?

I don't what vmware does here. Nor do they ship source to check. So if
you have a big HW box with say two packages, it might make sense to give
this information to the guest _if_ the CPUs are pinned and the guest
never migrates.

> Per your request in your next email:
> 
> > One thing I forgot to ask: Could you please check if you get the same
> > pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
> > rework).
> 
> Our previous kernel was 4.4, and didn't use the logical package id:
I see.

Did the patch I sent fixed it for you and were you not able to test?

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/25/2016 08:22 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:

[3.107126] init_rapl_pmus: maxpkg 4

there! vmware bug. It probably worked by chance.


Yes, the behavior is a bit random.


I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.


But sometimes the topology info is correct and if I get lucky, the
package id could be valid for all the CPU's.  Given the behavior,
I have seen so far it makes me thing the RAPL isn't being emulated.
So even if I did boot onto a "valid" set of cores, would I always be
certain that I will be on those cores?


diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 0a535cea8ff3..f5d85f2853d7 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
 {
int maxpkg = topology_max_packages();
size_t size;
+   unsigned int cpu;
+
+   for_each_possible_cpu(cpu) {
+   if (topology_logical_package_id(cpu) >= maxpkg) {
+   pr_err("rapl pmu error: max package: %u but CPU%d belongs to 
%u\n",
+  maxpkg, cpu, topology_logical_package_id(cpu));
+   return -EINVAL;
+   }
+   }

size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
rapl_pmus = kzalloc(size, GFP_KERNEL);


Per your request in your next email:


One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).


Our previous kernel was 4.4, and didn't use the logical package id:

/* check if phys_is is already covered */
for_each_cpu(i, _cpu_mask) {
if (phys_id == topology_physical_package_id(i))
return;

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/25/2016 08:22 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:

[3.107126] init_rapl_pmus: maxpkg 4

there! vmware bug. It probably worked by chance.


Yes, the behavior is a bit random.


I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.


But sometimes the topology info is correct and if I get lucky, the
package id could be valid for all the CPU's.  Given the behavior,
I have seen so far it makes me thing the RAPL isn't being emulated.
So even if I did boot onto a "valid" set of cores, would I always be
certain that I will be on those cores?


diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 0a535cea8ff3..f5d85f2853d7 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
 {
int maxpkg = topology_max_packages();
size_t size;
+   unsigned int cpu;
+
+   for_each_possible_cpu(cpu) {
+   if (topology_logical_package_id(cpu) >= maxpkg) {
+   pr_err("rapl pmu error: max package: %u but CPU%d belongs to 
%u\n",
+  maxpkg, cpu, topology_logical_package_id(cpu));
+   return -EINVAL;
+   }
+   }

size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
rapl_pmus = kzalloc(size, GFP_KERNEL);


Per your request in your next email:


One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).


Our previous kernel was 4.4, and didn't use the logical package id:

/* check if phys_is is already covered */
for_each_cpu(i, _cpu_mask) {
if (phys_id == topology_physical_package_id(i))
return;

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-25 14:22:05 [+0200], To Charles (Chas) Williams wrote:
> > [3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
> > [3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
> > [3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
> > 65535
> > [3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
> > [3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
> > [3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
> > [3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535

One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-25 14:22:05 [+0200], To Charles (Chas) Williams wrote:
> > [3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
> > [3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
> > [3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
> > 65535
> > [3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
> > [3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
> > [3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
> > [3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535

One thing I forgot to ask: Could you please check if you get the same
pkgid reported for cpu 0-3 on a pre-v4.8 kernel? (before the hotplug
rework).

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:
> I can't get dedicated access to the specific bare metal since it is
> running as a dedicated hypervisor.  I haven't seen this issue anywhere
> else though with the 4.8 kernel.

That is something :)

> > If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
> > to the starting point (in case of CPU up it would be CPUHP_OFFLINE.
> 
> You'll like this, I just did a little printk debugging because it was
> easier than trying to get a debugger running:
> 
>   [3.107126] init_rapl_pmus: maxpkg 4
there! vmware bug. It probably worked by chance.

>   [3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
>   [3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
>   [3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
> 65535
>   [3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
>   [3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
>   [3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
>   [3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535
> 
> where pkgid is topology_logical_package_id(cpu).
> 
> I can't understand why I don't see a cpu 3 during cpu prepare, when I
> see one later.  

because cpu 2 and 3 share the same package and if your printk is at the
bottom of the function, it will return early.

> The 65535 is a -1 from topology_phys_to_logical_pkg()
> getting assigned to the logical_proc_id apparently.

yes. The topology field is u16.

> So this is pretty puzzling.  Since this is a guest running under VMWare, I
> don't know that there is any particular CPU pinning or emulation of RAPL.

I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.

diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 0a535cea8ff3..f5d85f2853d7 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
 {
int maxpkg = topology_max_packages();
size_t size;
+   unsigned int cpu;
+
+   for_each_possible_cpu(cpu) {
+   if (topology_logical_package_id(cpu) >= maxpkg) {
+   pr_err("rapl pmu error: max package: %u but CPU%d 
belongs to %u\n",
+  maxpkg, cpu, topology_logical_package_id(cpu));
+   return -EINVAL;
+   }
+   }
 
size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
rapl_pmus = kzalloc(size, GFP_KERNEL);

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-21 17:03:56 [-0400], Charles (Chas) Williams wrote:
> I can't get dedicated access to the specific bare metal since it is
> running as a dedicated hypervisor.  I haven't seen this issue anywhere
> else though with the 4.8 kernel.

That is something :)

> > If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
> > to the starting point (in case of CPU up it would be CPUHP_OFFLINE.
> 
> You'll like this, I just did a little printk debugging because it was
> easier than trying to get a debugger running:
> 
>   [3.107126] init_rapl_pmus: maxpkg 4
there! vmware bug. It probably worked by chance.

>   [3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
>   [3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
>   [3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
> 65535
>   [3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
>   [3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
>   [3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
>   [3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535
> 
> where pkgid is topology_logical_package_id(cpu).
> 
> I can't understand why I don't see a cpu 3 during cpu prepare, when I
> see one later.  

because cpu 2 and 3 share the same package and if your printk is at the
bottom of the function, it will return early.

> The 65535 is a -1 from topology_phys_to_logical_pkg()
> getting assigned to the logical_proc_id apparently.

yes. The topology field is u16.

> So this is pretty puzzling.  Since this is a guest running under VMWare, I
> don't know that there is any particular CPU pinning or emulation of RAPL.

I assume "init_rapl_pmus: maxpkg 4" is from init_rapl_pmus() returning
topology_max_packages(). So it says 4 but then returns 65535 for CPU 2
and 3. That -1 comes probably from topology_update_package_map(). Could
you please send a complete boot log and try the following patch? This
one should fix your boot problem and disable RAPL if the info is
invalid.

diff --git a/arch/x86/events/intel/rapl.c b/arch/x86/events/intel/rapl.c
index 0a535cea8ff3..f5d85f2853d7 100644
--- a/arch/x86/events/intel/rapl.c
+++ b/arch/x86/events/intel/rapl.c
@@ -682,6 +682,15 @@ static int __init init_rapl_pmus(void)
 {
int maxpkg = topology_max_packages();
size_t size;
+   unsigned int cpu;
+
+   for_each_possible_cpu(cpu) {
+   if (topology_logical_package_id(cpu) >= maxpkg) {
+   pr_err("rapl pmu error: max package: %u but CPU%d 
belongs to %u\n",
+  maxpkg, cpu, topology_logical_package_id(cpu));
+   return -EINVAL;
+   }
+   }
 
size = sizeof(*rapl_pmus) + maxpkg * sizeof(struct rapl_pmu *);
rapl_pmus = kzalloc(size, GFP_KERNEL);

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/21/2016 06:56 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:

Recent 4.8 kernels have been oopsing when running under VMWare:


can you reproduce this on bare metal?


I can't get dedicated access to the specific bare metal since it is
running as a dedicated hypervisor.  I haven't seen this issue anywhere
else though with the 4.8 kernel.


[2.270203] BUG: unable to handle kernel NULL pointer dereference at 
0408
[2.270325] IP: [] rapl_cpu_online+0x59/0x70


can you check if pmu is NULL?


It's not.  The dereference at 0x408 and pmu->cpu being fairly early in
the struct seems to indicate that pmu wasn't pointing to 0 at the time
(but fairly close).  I should have noticed that earlier.


Is there a particular order guaranteed by the callbacks?  Will
rapl_cpu_prepare() always happen before online/offline?  Additionally,


yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
from CPUHP_OFFLINE till CPUHP_ONLINE.


Yes, I see that now.  Thanks for the pointer!


If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
to the starting point (in case of CPU up it would be CPUHP_OFFLINE.


You'll like this, I just did a little printk debugging because it was
easier than trying to get a debugger running:

[3.107126] init_rapl_pmus: maxpkg 4
[3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
[3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
[3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
65535
[3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
[3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
[3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
[3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535

where pkgid is topology_logical_package_id(cpu).

I can't understand why I don't see a cpu 3 during cpu prepare, when I
see one later.  The 65535 is a -1 from topology_phys_to_logical_pkg()
getting assigned to the logical_proc_id apparently.

So this is pretty puzzling.  Since this is a guest running under VMWare, I
don't know that there is any particular CPU pinning or emulation of RAPL.

It looks there was a proposal to not run in guests:

https://lkml.org/lkml/2015/12/3/559

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Charles (Chas) Williams]

On 10/21/2016 06:56 AM, Sebastian Andrzej Siewior wrote:

On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:

Recent 4.8 kernels have been oopsing when running under VMWare:


can you reproduce this on bare metal?


I can't get dedicated access to the specific bare metal since it is
running as a dedicated hypervisor.  I haven't seen this issue anywhere
else though with the 4.8 kernel.


[2.270203] BUG: unable to handle kernel NULL pointer dereference at 
0408
[2.270325] IP: [] rapl_cpu_online+0x59/0x70


can you check if pmu is NULL?


It's not.  The dereference at 0x408 and pmu->cpu being fairly early in
the struct seems to indicate that pmu wasn't pointing to 0 at the time
(but fairly close).  I should have noticed that earlier.


Is there a particular order guaranteed by the callbacks?  Will
rapl_cpu_prepare() always happen before online/offline?  Additionally,


yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
from CPUHP_OFFLINE till CPUHP_ONLINE.


Yes, I see that now.  Thanks for the pointer!


If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
to the starting point (in case of CPU up it would be CPUHP_OFFLINE.


You'll like this, I just did a little printk debugging because it was
easier than trying to get a debugger running:

[3.107126] init_rapl_pmus: maxpkg 4
[3.107263] rapl_cpu_prepare: pmu 880234faa540  cpu 0  pkgid 0
[3.107400] rapl_cpu_prepare: pmu 880234faa600  cpu 1  pkgid 2
[3.107537] rapl_cpu_prepare: pmu 880234faa6c0  cpu 2  pkgid 
65535
[3.107662] rapl_cpu_online: pmu 880234faa540 cpu 0 pkgid 0
[3.107907] rapl_cpu_online: pmu 880234faa600 cpu 1 pkgid 2
[3.108133] rapl_cpu_online: pmu 880234faa6c0 cpu 2 pkgid 65535
[3.108333] rapl_cpu_online: pmu 880234faa6c0 cpu 3 pkgid 65535

where pkgid is topology_logical_package_id(cpu).

I can't understand why I don't see a cpu 3 during cpu prepare, when I
see one later.  The 65535 is a -1 from topology_phys_to_logical_pkg()
getting assigned to the logical_proc_id apparently.

So this is pretty puzzling.  Since this is a guest running under VMWare, I
don't know that there is any particular CPU pinning or emulation of RAPL.

It looks there was a proposal to not run in guests:

https://lkml.org/lkml/2015/12/3/559

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:
> Recent 4.8 kernels have been oopsing when running under VMWare:

can you reproduce this on bare metal?

> [2.270203] BUG: unable to handle kernel NULL pointer dereference at 
> 0408
> [2.270325] IP: [] rapl_cpu_online+0x59/0x70
…
> 
> gdb tells me:
> 
> (gdb) info line *(rapl_cpu_online+0x59)
> Line 595 of "arch/x86/events/intel/rapl.c" starts at address 
> 0x81012bb9 
>and ends at 0x81012bbe .
> 
> Which is:
> 
> 
> target = cpumask_any_and(_cpu_mask, topology_core_cpumask(cpu));
> if (target < nr_cpu_ids)
> return 0;
> 
> cpumask_set_cpu(cpu, _cpu_mask);
> pmu->cpu = cpu;   <<

can you check if pmu is NULL?

> return 0;
…

> Is there a particular order guaranteed by the callbacks?  Will
> rapl_cpu_prepare() always happen before online/offline?  Additionally,

yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
from CPUHP_OFFLINE till CPUHP_ONLINE.

> rapl_cpu_prepare() can fail to allocate pmu,

error codes callbacks are handled.

…
> But rapl_cpu_online() would have no idea about this.  What should be
> done in this case?

If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
to the starting point (in case of CPU up it would be CPUHP_OFFLINE.

Sebastian

Re: [PREEMPT-RT] Oops in rapl_cpu_prepare()

[Sebastian Andrzej Siewior]

On 2016-10-20 16:27:55 [-0400], Charles (Chas) Williams wrote:
> Recent 4.8 kernels have been oopsing when running under VMWare:

can you reproduce this on bare metal?

> [2.270203] BUG: unable to handle kernel NULL pointer dereference at 
> 0408
> [2.270325] IP: [] rapl_cpu_online+0x59/0x70
…
> 
> gdb tells me:
> 
> (gdb) info line *(rapl_cpu_online+0x59)
> Line 595 of "arch/x86/events/intel/rapl.c" starts at address 
> 0x81012bb9 
>and ends at 0x81012bbe .
> 
> Which is:
> 
> 
> target = cpumask_any_and(_cpu_mask, topology_core_cpumask(cpu));
> if (target < nr_cpu_ids)
> return 0;
> 
> cpumask_set_cpu(cpu, _cpu_mask);
> pmu->cpu = cpu;   <<

can you check if pmu is NULL?

> return 0;
…

> Is there a particular order guaranteed by the callbacks?  Will
> rapl_cpu_prepare() always happen before online/offline?  Additionally,

yes, see include/linux/cpuhotplug.h. On CPU-up the array ids are invoked
from CPUHP_OFFLINE till CPUHP_ONLINE.

> rapl_cpu_prepare() can fail to allocate pmu,

error codes callbacks are handled.

…
> But rapl_cpu_online() would have no idea about this.  What should be
> done in this case?

If a callback (such as CPUHP_PERF_X86_RAPL_PREP) fail then we rollback
to the starting point (in case of CPU up it would be CPUHP_OFFLINE.

Sebastian