I wrote a little bit much, so I put a summary up front. Sorry about that. Summary: 1) Is there any point in time, where one single instance of Hadoop has access to all keys that are to be distributed to the nodes together with corresponding data? Or maybe at least nodes could have Task priorities, killing and rescheduling tasks, if higher priority tasks arrive (key word: Partitioner, TaskScheduler).
2) Blocking running tasks, does not get me anywhere, if they are not suspended allowing other Reducer to take their place as this is taking up Reducer slots, isn't it? The main problem are Reducers waiting for a slot. 3) Why Reducer ordering (could) affect(s) processing. ad 1) 1.1) The only thing the JobTracker would need to do is to look at keys and derive some job internal order of Reduce tasks. At this point it would be necessary for the JobTracker (or _any other_ instance which would be able to do such a thing!) to know about how many Reducers are to start for a specific job, what their keys are or at least about their priority. 1.2) At some point the Partitioner is distributing keys to nodes. Meaning it could at least group high quality with low quality tasks (based on some criterion). Now cannot _for example_ the TaskTrackers themselves decide which task - that was assigned to them by the Partitioner - to execute first? 1.3) And the question is basically, is there some instance that CAN do some prioritizing of Tasks, the way I want it to. Even if it is only in combination with the Partitioner: "Oh, a Task with lower priority is running. So kill it, and restart it later." Maybe this would work using something else but the FairScheduler. I am making wild guesses here, but I think I am drifting towards the TaskScheduler, if it actually does what I think it does. ad 2) Blocking: If I have enough slots for all the Reduce tasks, I have no problem at all. There is no sense at all in starting a task and then blocking it. Why not let it run? It is not like the Reducer have to wait for some other to finish. They could just quit working/not even start, if there output is redundant (see "ad 3)"). ad 3) This is why Reducer ordering affects processing: Preliminaries: * Each Reducer raises a (global! - using ZooKeeper, FileSystem or maybe Counters) threshold. * Each Reducer can estimate if it will ever pass a given threshold. * Output of Reducers that cannot pass the threshold is discarded. * Some Reducers have a higher probability (by Key) to raise the threshold faster. As a result it would make sense to run Reducers with a higher probability to raise the threshold first. Reducers can cease their work or not even start, if they cannot pass the threshold anymore. On Mon, Dec 20, 2010 at 11:58 AM, Harsh J <qwertyman...@gmail.com> wrote: > The JobTracker wouldn't know what your data is going to be is when it > is assigning the Reduce Tasks. > > If you really do need ordering among your reducers, you should > implement a locking mechanism (making sure the dormant reduce tasks > stay alive by sending out some status reports). > > Although, how is ordering going to affect your reducer's processing? :) > > On Mon, Dec 20, 2010 at 2:37 PM, Martin Becker <_martinbec...@web.de> wrote: >> I just reread my first post. Maybe I was not clear enough: >> It is only important to me that the Reduce tasks _start_ in a >> specified order based on their key. That is the only additional >> constraint I need. >> >> On Mon, Dec 20, 2010 at 9:51 AM, Martin Becker <_martinbec...@web.de> wrote: >>> As far as I understood, MapReduce is waiting for all Mappers to finish >>> until it starts running Reduce tasks. Am I mistaken here? If I am not, >>> then I do not see any more synchrony being introduced than there >>> already is (no locks required). Of course I am not aware of all the >>> internals, but MapReduce is working with a single JobTracker, which >>> distributes Reduce tasks to the different nodes (see >>> http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Overview). >>> So the only point where my "theory" would break is, if Reducer start >>> before Mappers finish. Otherwise the JobTracker should be able to >>> schedule Reduce tasks in a specific order. >>> >>> On Mon, Dec 20, 2010 at 4:45 AM, Harsh J <qwertyman...@gmail.com> wrote: >>>> You could use sort of a distributed lock service to achieve this >>>> (ZooKeeper can help). But such things ought to be avoided as David >>>> pointed out above. >>>> >>>> On Sun, Dec 19, 2010 at 9:09 PM, Martin Becker <_martinbec...@web.de> >>>> wrote: >>>>> Hello everybody, >>>>> >>>>> is there a possibility to make sure that certain/all reduce tasks, >>>>> i.e. the reducers to certain keys, are executed in a specified order? >>>>> This is Job internal, so the Job Scheduler is probably the wrong place to >>>>> start? >>>>> Does the order induced by the Comparable interface influence the >>>>> execution order at all? >>>>> >>>>> Thanks in advance, >>>>> Martin >>>>> >>>> >>>> >>>> >>>> -- >>>> Harsh J >>>> www.harshj.com >>>> >>> >> > > > > -- > Harsh J > www.harshj.com >
