On 2015-11-14 09:54, [email protected] wrote: >>> I think 8 is way to high. Isn't the point of batch to run things >>> when the machine is mostly idle? >> The problem is (and we've had this discussion several times before at >> least in misc@), that the system load doesn't really tell us that.
> What's the proper way to calculate amount of work a system can do, for > (then) figuring CPU idle time threshold? > Does this not also include the work load (type) being done and imply > capability to manage the work load distribution? The problem is that there is no proper way, at least not *one* proper way, to do that. It all depends on your particular situation. The problem is that cron has no way to know whether the job it is just about to fire away is going to take a hundred milliseconds or a hundred hours to run, or what kind of resources it will consume. The reason to use loadavg as an indicator for system activity is that while it measures neither high cpu activity nor high i/o-activity directly, it is actually a pretty good hint as to whether the system is "busy", for a very fuzzy definition of busy. The problem is that the value isn't absolute, it is relative to the configuration and load profile of each system. If my SP system shows a load of 14, I can with some certainty say that it's quite busy. If my 12-core dual Opteron server shows 14, it's hardly even breathing heavily, *even* if it's got mostly cpu-bound activity. If it shows 100+, then as a sysadmin I'd start looking for explanations. Remember also that running a job via batch generally is a very kind way to start heavy tasks, because cron runs the job niced. So in case of much cpu-bound activity, it may never even get in more than a few time slices here and there to run, while if there's much i/o going on it may run next to unnoticed even if it's got lots of cpu-bound stuff to do. So the problem isn't even that big, since the system's own scheduler is pretty good at handling various system loads when they actually have begun life as processes. The one time when it is especially unsuitable to run an extra batch job is if we are memory starved, and are swapping, or are close to having to swap. And as it happens, load_avg is conveniently going to start skyrocketing as we are starting to swap. So, if we break it down, load_avg is really not such a bad metric to use in this particular case. It is "just" that the default limit is set way too low for today's standards. With that said, I'm looking at other ways to determine system workload. Maybe there's a set of metrics that give us a more accurate snapshot of the system's current state, that can be averaged over time like load_avg so as to avoid temporary spikes that may give a faulty impression of the system's activity. But that calculation must also be as unaffected as possible by the system's "dimensions". A system with fast i/o can of course handle more of it before becoming saturated. Likewise with cpu speed and number of cores, and system memory. So the best aspects in my mind to start looking at is whether the system does a lot of *waiting* to get its jobs done. Either waiting for disk or network i/o, processes waiting to run or a lock contention to clear, things like that. So, if nobody is waiting for anybody, then by all means go ahead and run one more job! It's not going to harm anything, as long as it doesn't consume all of the idling resources for itself. The problem can be as complex as we want to make it, or as simple, if it isn't a problem in practice. >> My particular problem, and the reason I suggested this patch in the >> first place, is that I often see loads of 20-30-50 or even way more, >> without there even being a problem. The machine is very responsive, and >> everything works great - there are just a lot of processes running or >> waiting for an opportunity to run. > > That's not the general case on 'single/dual' (or less than "your chosen > higher than 4 number" of) CPU systems, and when running fewer processes > that are more CPU intensive. In these cases it may also be easier to > know what's happening on the system. Selecting the offloaded period > (automatically?) where you don't have direct control requires more > understanding than average load numbers (suggestion only). Or a > different approach at task running (e.g. service oriented nodes > assisting general worker ones). > > Better use statistical approach per machine (counters) while factoring > processing capability and duty saturation cycle (human assessment). Or > simply users circadian cycle and not care much as machines just work > while people rest, with potential and overlap between multiple machines > for same role/task. Good points. But taking the human circadian cycle into account, that is, working when the human is not and vice versa, can easily be accommodated already, by using "at". >> Since the system load essentially is a decaying average of the number of >> runnable or running processes, it is not in any way connected to actual >> processor workload as in instructions executed, just to the fact that >> there is much *potentially* going on in the system. > > Obviously, this explains why the average load figure is not 'the' proper > way to quantify processor business, such method gains little adequacy > without a tuning knob and that is after assessment of other factors. > CPU number does correlate but is not solely deterministic, and imagine > the mess from twisting a knob without understanding what it does (sane > limits, sane defaults). > >> That's also why I suggested to base the default on a value relative to >> the number of cores - it made sense from my practical point of view. But >> I understand where Theo's coming from on this. > > Please comment (improved?) method to estimate processor offloaded > periods that reduces average load guess work, or simply a practical > approach at solving the problem of finding offloaded periods > (threshold) without pushing edge case changes. Solving this for real requires some deliberation... however, Todd's question is relevant - who is actually using this, or rather, is it worth designing something that works better if I'm the only one on the planet that's actually using at/batch? :-) (The answer is of course "yes", because there's good hack value in it...) but seriously, finding a good answer to this question can be beneficial in other similar situations, too. There are a number of real-life situations where there is a batch/queue/work dispatcher controlling a number of asynchronous tasks, that might use an improved method of determining available work load. Regards, /Benny
