Hi! I'm currently trying to wrap my head around the different schedulers available. Running Cloudera 0.18.3, there's both the Fair Scheduler and the Capacity Scheduler for me to play with.
I have a number of jobs that are run nightly. Some of them have tighter deadlines than other, so if there is a job that has a tight deadline, I would like that to get *all* mapper/reducer slots so that it finishes as quickly as possible. Once there is a free mapper slot that's not needed for the job, it should be made available for other jobs. Within the jobs, there are sub-priorities. As an example, assuming we analyse some kind of web access logs, it's more important the daily report gets done before the monthly report. Etc. Also, the tight-deadline's monthly report should be prioritized before the daily report for non-tight-jobs. I think what I'm after is a way to prioritize jobs, but the available number of priorities (VERY HIGH, HIGH, et. al) are not enough. Ideas on how to do this? I hope I've managed to explain what I want. Cheers, \EF -- Erik Forsberg <forsb...@opera.com> Developer, Opera Software - http://www.opera.com/
