On 04/13/2016 02:18 AM, Ingo Molnar wrote:
* Waiman Long<waiman.l...@hpe.com> wrote:
On a large system with many CPUs, using HPET as the clock source can
have a significant impact on the overall system performance because
of the following reasons:
1) There is a single HPET counter shared by all the CPUs.
2) HPET counter reading is a very slow operation.
Using HPET as the default clock source may happen when, for example,
the TSC clock calibration exceeds the allowable tolerance. Something
the performance slowdown can be so severe that the system may crash
because of a NMI watchdog soft lockup, for example.
/*
+ * Reading the HPET counter is a very slow operation. If a large number of
+ * CPUs are trying to access the HPET counter simultaneously, it can cause
+ * massive delay and slow down system performance dramatically. This may
+ * happen when HPET is the default clock source instead of TSC. For a
+ * really large system with hundreds of CPUs, the slowdown may be so
+ * severe that it may actually crash the system because of a NMI watchdog
+ * soft lockup, for example.
+ *
+ * If multiple CPUs are trying to access the HPET counter at the same time,
+ * we don't actually need to read the counter multiple times. Instead, the
+ * other CPUs can use the counter value read by the first CPU in the group.
Hm, weird, so how can this:
static cycle_t read_hpet(struct clocksource *cs)
{
return (cycle_t)hpet_readl(HPET_COUNTER);
}
... cause an actual slowdown of that magnitude? This goes straight to MMIO. So
is
the hardware so terminally broken?
I only know that accessing the HPET counter is VERY slow. Andy said that
it takes at least a few us. I haven't done that measurement myself.
I am not sure what kind of contention will happen when multiple CPUs are
accessing it at the same time. It is not just the clock tick interrupt
handler that need to access time, many system call will also cause the
current time to be accessed. When we have hundred of CPUs in the system,
it is not too hard to cause a soft lockup if hpet is the default clock
source.
How good is the TSC clocksource on the affected system? Could we simply always
use
the TSC (and not use the HPET at all as a clocksource), instead of trying to fix
broken hardware?
Thanks,
Ingo
The TSC clocksource, on the other hand, is per cpu. So there won't be
much contention in accessing it. Normally TSC will be used the default
clock source. However, if there is too much variation in the actual
clock speeds of the individual CPUs, it will cause the TSC calibration
to fail and revert to use hpet as the clock source. During bootup, hpet
will usually be selected as the default clock source first. After a
short time, the TSC will take over as the default clock source. Problem
can happen during that short period of transition time too. In fact, we
have 16-socket Broadwell-EX systems that has this soft lockup problem
once in a few reboot cycles which prompted me to find a solution to fix it.
Cheers,
Longman
