Hi, Mike Thanks for your reply :)
On 03/25/2013 05:22 PM, Mike Galbraith wrote: > On Mon, 2013-03-25 at 13:24 +0800, Michael Wang wrote: >> Recently testing show that wake-affine stuff cause regression on pgbench, the >> hiding rat was finally catched out. >> >> wake-affine stuff is always trying to pull wakee close to waker, by theory, >> this will benefit us if waker's cpu cached hot data for wakee, or the extreme >> ping-pong case. >> >> However, the whole stuff is somewhat blindly, there is no examining on the >> relationship between waker and wakee, and since the stuff itself >> is time-consuming, some workload suffered, pgbench is just the one who >> has been found. >> >> Thus, throttle the wake-affine stuff for such workload is necessary. >> >> This patch introduced a new knob 'sysctl_sched_wake_affine_interval' with the >> default value 1ms, which means wake-affine stuff only effect once per 1ms, >> which >> usually the minimum balance interval (the idea is that the rapid of >> wake-affine >> should lower than the rapid of load-balance at least). >> >> By turning the new knob, those workload who suffered will have the chance to >> stop the regression. > > I wouldn't do it quite this way. Per task, yes (I suggested that too, > better agree;), but for one, using jiffies in the scheduler when we have > a spiffy clock sitting there ready to use seems wrong, Well, I get the approach from load-balance code, this is one existed way to play interval stuff, just try to keep consistent... and secondly, > when you've got a bursty load, not pulling can hurt like hell. Alex > encountered that while working on his patch set. > >> Test: >> Test with 12 cpu X86 server and tip 3.9.0-rc2. >> >> Default 1ms interval bring limited performance improvement(<5%) for >> pgbench, significant improvement start to show when turning the >> knob to 100ms. > > So it seems you'd be better served by an on/off switch for this load. > 100ms in the future for many tasks is akin to a human todo list entry > scheduled for Solar radius >= Earth orbital radius day ;-) Do you mean 1ms interval is still too big? and you prefer to have a 0 option? > >> original 100ms >> >> | db_size | clients | tps | | tps | >> +---------+---------+-------+ +-------+ >> | 21 MB | 1 | 10572 | | 10675 | >> | 21 MB | 2 | 21275 | | 21228 | >> | 21 MB | 4 | 41866 | | 41946 | >> | 21 MB | 8 | 53931 | | 55176 | >> | 21 MB | 12 | 50956 | | 54457 | +6.87% >> | 21 MB | 16 | 49911 | | 55468 | +11.11% >> | 21 MB | 24 | 46046 | | 56446 | +22.59% >> | 21 MB | 32 | 43405 | | 55177 | +27.12% >> | 7483 MB | 1 | 7734 | | 7721 | >> | 7483 MB | 2 | 19375 | | 19277 | >> | 7483 MB | 4 | 37408 | | 37685 | >> | 7483 MB | 8 | 49033 | | 49152 | >> | 7483 MB | 12 | 45525 | | 49241 | +8.16% >> | 7483 MB | 16 | 45731 | | 51425 | +12.45% >> | 7483 MB | 24 | 41533 | | 52349 | +26.04% >> | 7483 MB | 32 | 36370 | | 51022 | +40.28% >> | 15 GB | 1 | 7576 | | 7422 | >> | 15 GB | 2 | 19157 | | 19176 | >> | 15 GB | 4 | 37285 | | 36982 | >> | 15 GB | 8 | 48718 | | 48413 | >> | 15 GB | 12 | 45167 | | 48497 | +7.37% >> | 15 GB | 16 | 45270 | | 51276 | +13.27% >> | 15 GB | 24 | 40984 | | 51628 | +25.97% >> | 15 GB | 32 | 35918 | | 51060 | +42.16% > > The benefit you get with not pulling is two fold at least, first and > foremost it keeps the forked off clients the hell away from the mother > of all work so it can keep the kids fed. Second, you keep the load > spread out, which is the only way the full box sized load can possibly > perform in the first place. The full box benefit seems clear from the > numbers.. hard working server can compete best for its share when it's > competing against the same set of clients, that's likely why you have to > set the knob to 100ms to get the big win. Actually the 10ms will also get around 27% improvement at most, I use 100ms since it looks more significant... I haven't tried the interval between 1 and 10, but I suppose the benefit could be some kind of parabola, it's not a suddenly change, but smoothly. > > With small burst loads of short running tasks, even things like pgbench > will benefit from pulling to local llc more frequently than 100ms, iff > burst does not exceed socket size. That pulling is not completely evil, > it automagically consolidates your mostly idle NUMA box to it's most > efficient task placement for both power saving and throughput, so IMHO, > you can't just let tasks sit cross node over ~extended idle periods > without doing harm. I see, and actually that's the reason for this proposal, it's just try to reserve all the possible benefit of wake-affine, and provide a way to control the rapid. I think your point here is still that we need a 0 option, it that correct? > > OTOH, if the box is mostly busy, or if there's only one llc, per task > pick a smallish number is dirt simple, and should be a general case > improvement over the overly 1:1 buddy centric current behavior. > > Have you tried the patch set by Alex Shi? In my fiddling with them, > they put a very big dent in the evil side of select_idle_sibling() and > affine wakeups in general, and should help pgbench and ilk heaping > truckloads. I haven't tried yet, whatever, a way to throttle the wake-affine is necessary, since we have the proof that it will damage some workload. OTOH, by theory, we have the chance to pull none-related tasks together, and we can't promise this will help the system all the time, don't we? Regards, Michael Wang > > -Mike > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
