Dear All, Several years ago we had the case when we run a job without a scheduler it runs faster, the scheduler in question even thought it was adverstided aware of the NUMA architecture, it selected the wrong cpus to run the jobs in an SGI ALTIX machine and so the overhead was measured as the difference between running an interactive job without scheulder intervention and one with a scheduler.
I also find the maui simulation function very useful after collecting real workload data and if you want to change the policies or scheduler configuration, the simulation mode coupled with the maui cluster, and scheduler statistics gives you an idea of what have changed and what affect it will have in your workload. regards Walid 1 14:25, Reuti <re...@staff.uni-marburg.de> wrote: > Am 17.02.2011 um 09:07 schrieb Fritz Ferstl: > > > My 2 cents here (and I'm aware they will not help Eric ... and apologies > in advance for the rant, it's a long-term heartfelt topic ...): > > > > A DRM benchmark would be nice to have. Benchmarking in general is an > almost vain attempt. You have to be very prescriptive of the boundary > conditions to achieve comparable results. And such narrow boundary > conditions almost never can reflect reality. So all benchmarks are up for > interpretation and up for debate. > > > > But, benchmarks are still a useful means to provide at least some > orientation. As Chris has stated, the variability in the use case scenarios > of workload managers is certainly even bigger than in classical performance > benchmarks such as SPEC or Linpack. You also have to be careful what you are > measuring: the underlying HW, network & storage performance? Or the > efficiency of the SW? Or the ability to tune the workload management system > - in itself and in combination with HW & SW underneath? Or the suitability > of the workload management system for a specific application case? > > > > So I guess that probably a suite of benchmarks would be needed, maybe > akin to SPEC, to provide at least a roughly representative picture. And > you'd have to either standardize on the HW, e.g. take 100 Amazon dedicated > servers and run with that, or you'd have to do it like for Linpack and say: > "I don't care what you use and how much of it but report the resulting > throughput vs time numbers on these use cases." I.e. how fast can you > possibly get. In other words something like the Top500 for workload > management environments. > > > > For many companies and institutions workload managers have become the > most central workhorse - the conveyor belt of a data center. If it stops, > all stops. If you can make it run quicker you achieve your results sooner. > If it enables you to do so you can be much more flexible in responding to > changing demands. So it's almost ironic that large computing centers are > benchmarking individual server performance, run something like Linpack to > advertise their peak performance and create their own site-specific > application benchmark suites for selecting new HW. But they often do not > benchmark with the workload management system in the picture which later, in > combination with tuning and the rest of the environment, will define the > efficiency of the data center. > > > > So a benchmark for DRMs would be a highly useful tool. I've always > wondered how to get an initiative started which would lead to such a > benchmark ... > > The question is: what to measure in a queuing system? This could be: > > a) - The load on the qmaster machine to handle the submitted jobs (used > disk, memory, cpu-time). > > b) - Time vasted to switch from one job to the next one on an exechost > (maybe already sending the next job thereto in a "ready" state beforehand > and release it as soon as the former job finished). This would be > interesting for workflows where many short jobs are submitted. > > c) - Time vasted by resource reservation. It was a couple of times on the > former list to have some kind of real-time features in SGE to know exactly > when a jobs starts and will end (assuming you know the necessary execution > times beforehand). In this context also solving a cutting-stock-problem > would fit (to minimize the overall wallclock time): having a bunch of jobs, > is the queuing system capable of reordering them in such a way, that the > resource reservation vasted is at its minimum (possibly zero) and to finish > the complete bunch of jobs in the shortest wallclock time possible? > > d) - Can I tell the scheduler that I have varying resource requirements > over the runtime of the job to lower vasted resources? Can the queuing > system send my job around in the cluster depending on the resources need in > certains steps of a job? All to minimize the unused resources. > > e) - Can c) be combined with some job dependencies in a directed graph (a > job with error shouldn't trigger the next with a job-hold to start) and > decisions which job of two alternatives should be executed at all? (sadly > the project http://wildfire.bii.a-star.edu.sg/ stopped) > > Somehow I have the impression, that there is no real benchmark possible in > a sense of "always faster" and next year "faster again". This would end up > with the criticism you state about a plain Linpack test. > > It's more like a conformance test according to certain set up rules. > Imagine the case you have queuing system A which can reorder jobs and > minimize vasted resources (but has a high impact on the qmaster machine), > and queuing system B which puts nearly no load on the qmaster machine (but > has just a FIFO for handling jobs). Which one would you buy? > > NB: for us in quantum chemistry, where jobs run for weeks or even months, > a) and b) isn't much of a concern. c) would be interesting though. d) would > be hard to phrase in exact times. > > -- Reuti > >
_______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
