> I still wonder whether the timeout code is the real culprit. My patch
> will slow down entry into do_machine_check by tens of cycles, several
> cachelines, and possibly a couple of TLB misses. Given that the
> timing seemed marginal to me, it's possible (albeit not that likely)
> that it pushed the time needed for synchronization into the range of
> unreliability.
I think we have very conservative timeouts. Here are the significant bits from
mce
First SPINUNIT ... how many nanoseconds to spin "ndelay(SPINUNIT)" before
pounding on an atomic op to see if we are done:
#define SPINUNIT 100 /* 100ns */
Initialization of our timeout - comment above this claims we are aiming for
"one second"
cfg->monarch_timeout = USEC_PER_SEC;
Inside mce_start() - set up our timeout. Math says this will be 1e9
u64 timeout = (u64)mca_cfg.monarch_timeout * NSEC_PER_USEC;
Now the actual loop:
while (atomic_read(&mce_callin) != cpus) {
if (mce_timed_out(&timeout)) {
atomic_set(&global_nwo, 0);
return -1;
}
ndelay(SPINUNIT);
}
And the inside of mce_timed_out() ... we decrement our 1e9 by 100 for each call:
if ((s64)*t < SPINUNIT) {
if (mca_cfg.tolerant <= 1)
mce_panic("Timeout synchronizing machine check over
CPUs",
NULL, NULL);
cpu_missing = 1;
return 1;
}
*t -= SPINUNIT;
Not sure what the worst case is for waiting for other processors to show up.
Perhaps it is
when the other cpu is in some deep C-state and we have to wait for it to power
the core
back on and resync clock. Maybe it is when a cpu has just started a "wbinvd"
instruction
with a cache entirely full of dirty lines that belong on some other processor
so have to
be written back across QPI (may when QPI links are congested with 10gbit
network traffic?
But a full second seems like a lot ... you adding a handful of cache and TLB
misses shouldn't
push us over the edge.
-Tony
