Re: What is the class that launches the reducers?

If you really want to do this, I guess you can move the scheduling of reducers to MRClientProtocol so that only when MR AM is notified to proceed by a client, does it continue to run reducers. The scheduling of reducers is currently done in JobImpl.java at SetupCompletedTransition.transition() Not

Re: What is the class that launches the reducers?

Right now the map and reduce task produces digests of the output. This logic is inside the map and reduce functions. I need to pause the execution when all maps finish because there will be an external program that is synchronizing several mapreduce runtimes. When all map tasks finish from the

Re: What is the class that launches the reducers?

How are you intending to verify the map output? It's only partially dumped to disk. None of the intermediate data goes into HDFS. Daniel On Aug 25, 2016 4:10 PM, "xeon Mailinglist" wrote: > But then I need to set identity maps to run the reducers. If I suspend a >

Re: What is the class that launches the reducers?

Mappers and reducers that are all scheduled when the job starts running. Don't think there is a knob to suspend the job once all mappers finish. On Thu, Aug 25, 2016 at 4:10 PM, xeon Mailinglist wrote: > But then I need to set identity maps to run the reducers. If I

Re: What is the class that launches the reducers?

But then I need to set identity maps to run the reducers. If I suspend a job after the maps finish, I don't need to set identity maps up. I want to suspend a job so that I don't run identity maps and get better performance. On Aug 25, 2016 10:12 PM, "Haibo Chen" wrote:

Re: What is the class that launches the reducers?

One thing you can try is to write a map-only job first and then verify the map out. On Thu, Aug 25, 2016 at 1:18 PM, xeon Mailinglist wrote: > I am using Mapreduce v2. > > On Aug 25, 2016 8:18 PM, "xeon Mailinglist" > wrote: > > > I am

Re: What is the class that launches the reducers?

I am using Mapreduce v2. On Aug 25, 2016 8:18 PM, "xeon Mailinglist" wrote: > I am trying to implement a mechanism in MapReduce v2 that allows to > suspend and resume a job. I must suspend a job when all the mappers finish, > and resume the job from that point after

What is the class that launches the reducers?

I am trying to implement a mechanism in MapReduce v2 that allows to suspend and resume a job. I must suspend a job when all the mappers finish, and resume the job from that point after some time. I do this, because I want to verify the integrity of the map output before executing the reducers. I