> On Apr 5, 2021, at 09:07, Stuart Henderson <s...@spacehopper.org> wrote: > > I've attached r620-E5_2630v2-2p6c2t.tgz, from Dell PE R620 with E5-2630v2. > This is a machine which has "disabling user TSC (skew=XXX)" reported for > some cores: > > Nov 5 16:08:34 pweb /bsd: cpu11: disabling user TSC (skew=-107) > Nov 5 16:08:34 pweb /bsd: cpu13: disabling user TSC (skew=-101)
This is probably NUMA latency when synchronizing between packages. If we want to account for this we will need to do more complicated stuff. > 2 SMT per core, 6 core per package, 2 packages. > > s-c-p in the below are smt/core/package for each cpu, i.e. > cpu0: smt 0, core 0, package 0 > cpu1: smt 0, core 0, package 1 > cpu2: smt 0, core 1, package 0 > cpu3: smt 0, core 1, package 1 > etc. > > cpu s-c-p Min Max Median Avg Stddev > 1 0-0-1 -65 498 157 152.071 28.014331 > 2 0-1-0 -22 13 -4 -4.91 7.8085128 > 3 0-1-1 -42 507 112 116.39 27.882671 > 4 0-2-0 -14 7 -2 -0.272 4.7497975 > 5 0-2-1 3 380 137 138.336 18.935249 > 6 0-3-0 -25 17 -5 -4.035 6.9143028 > 7 0-3-1 -136 299 86 87.687 22.889101 > 8 0-4-0 -24 4 -10 -8.81 5.0030921 > 9 0-4-1 -355 353 -6 -25.56 69.909611 > 10 0-5-0 7 41 27 26.838 7.2798869 > 11 0-5-1 -313 163 19 7.541 62.852118 > 12 1-0-0 -30 15 -3 -6.58 9.1155193 > 13 1-0-1 -331 439 17.5 0.494 67.932708 > 14 1-1-0 -30 12 -9 -9.583 6.0745379 > 15 1-1-1 -358 432 -15 -28.774 74.476155 > 16 1-2-0 -28 14 -9 -9.666 6.8339701 > 17 1-2-1 -339 459 -9 -31.638 77.072817 > 18 1-3-0 -29 19 -8 -7.896 7.3815765 > 19 1-3-1 -211 289 -101 -59.981 63.882655 > 20 1-4-0 -21 28 -1 -1.064 7.7156617 > 21 1-4-1 -338 417 -24.5 -31.543 77.017062 > 22 1-5-0 -1 39 17 16.359 5.3022857 > 23 1-5-1 -352 447 28 8.155 69.831278 > > Grouping them by package/smt: > > N Min Max Median Avg Stddev > cpu-p0-smt0 5000 -25 41 -3 1.7622 14.367997 > cpu-p0-smt1 6000 -30 39 -5 -3.0716667 11.631154 > cpu-p1-smt0 6000 -355 507 100 79.410833 79.21407 > cpu-p1-smt1 6000 -358 459 4 -23.881167 75.312631 > > CPU has ITSC, I do not have any "clock going backwards" with monotime.c > with kern.timecounter.hardware=tsc. > > <r620-E5_2630v2-2p6c2t.tgz>
