Ya. I'll be doing that. Meantime, do you know how control flows and reaches assignTasks()?
If so, please do tell me. I'm suspecting a gap in my understanding. On 11-Jul-2013 6:48 PM, "Devaraj k" <[email protected]> wrote: > Could you debug and check the Job Tracker/your scheduler code, then you can > pinpoint the exact issue and fix it. > > Thanks > Devaraj k > > > On Thu, Jul 11, 2013 at 7:58 PM, Sreejith Ramakrishnan < > [email protected]> wrote: > > > @Devaraj K > > > > I tried setting mapred.jobtracker.maxtasks.per.job to 4 in > mapred-site.xml. > > In the subsequent job, I got this in the log which clearly shows the > > configuration has taken effect: > > > > INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: > nMaps=1 > > nReduces=1 max=4 > > > > Yet, the original problem remains. It doesn't go anywhere from 0%. Btw, > > when we get the jobQueue, should we specify a name for the queue we want? > > (eg. default) > > > > INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added > > successfully for user 'sreejith' to queue 'default' > > > > Also, is there a limit to the no.of tasks which can be obtained in a > > heartbeat? > > > > > > On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <[email protected]> wrote: > > > > > You don't need to explicitly call the initTasks(), you can find this > > > invocation in JobTracker.java class. > > > > > > >*Yet the **Cannot create task split for <my-job-id>** is still there*. > > > >About errors relating to the JobID, is this significant? --- INFO > > > >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1 > > > >nReduces=1 max=-1 > > > > > > Here it shows max=-1, which means max configured(i.e default) tasks for > > > the Job is -1. > > > > > > I think you can see this exception from JT log. > > > > > > LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + > numReduceTasks > > > + " max=" + maxTasks); > > > if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) { > > > throw new IOException( > > > "The number of tasks for this job " + > > > (numMapTasks + numReduceTasks) + > > > " exceeds the configured limit " + maxTasks); > > > } > > > > > > Can you try configuring some number for this configuration > > > 'mapred.jobtracker.maxtasks.per.job' and run the same Job? > > > > > > > > > Thanks > > > Devaraj k > > > > > > > > > -----Original Message----- > > > From: Sreejith Ramakrishnan [mailto:[email protected]] > > > Sent: 11 July 2013 13:26 > > > To: [email protected] > > > Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce > > 0% > > > > > > @Devaraj K > > > > > > *I tried explicitly calling incomingJob.initTasks()*and it gave me a > > > pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job > > > job_201307111248_0002 initialized successfully with 1 map tasks and 1 > > > reduce tasks > > > > > > *Yet the **Cannot create task split for <my-job-id>** is still there*. > > > About errors relating to the JobID, is this significant? --- INFO > > > org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1 > > > nReduces=1 max=-1 > > > > > > *P.S: In the log file I shared, anything with "sreejith == " were > logged > > > by me. Not system-generated* > > > > > > > > > On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan < > > > [email protected]> wrote: > > > > > > > I've attached the relevant portion of the log. > > > > > > > > For easy testing, *I've attached the maven project* for the > > > > EDFJobScheduler. Just make a JAR from it and put it in > > > > $HADOOP_HOME/lib and add this property to *mapred-site.xml*: > > > > > > > > *<property> > > > > * > > > > * <name>mapred.jobtracker.taskScheduler</name> > > > > <value>org.apache.hadoop.mapred.EDFJobScheduler</value> > > > > * > > > > *</property>* > > > > > > > > > > > > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan < > > > > [email protected]> wrote: > > > > > > > >> I'm making my own scheduler. Since I had errors, I rewrote the > > > >> scheduler to do just a simple thing. When I give it a job, it > > > >> allocates 1 map() and 1 > > > >> reduce() to it. But that isn't happening. I used a barebones > WordCount > > > pgm. > > > >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see: > > > >> > > > >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task > split > > > >> for <my-job-id> > > > >> > > > >> DETAILED INFO: > > > >> > > > >> Here's some snippets of code. The scheduler logic is inside the > > > >> assignTasks(): > > > >> > > > >> We get a JobInProgressListener in the constructor- > > > >> this.jobQueueJobInProgressListener = new > > > >> JobQueueJobInProgressListener(); > > > >> > > > >> In start() - > > > >> > > > >> > this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres > > > >> sListener); > > > >> > > > >> Then I just take the last element in that queue (I assume its the > > > >> latest submitted job) and assign to incomingJob (of type > > JobInProgress). > > > >> > > > >> Then I obtain a new map task as follows and append this task to a > > > >> list which is the return value of the assignTasks(): > > > >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus, > > > >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts()); > > > >> > > > >> > > > >> I've attached the scheduler. Can any expert point out if there's any > > > >> ignorant mistakes from my side? > > > >> > > > > > > > > > > > > > >
