@Devaraj K I tried setting mapred.jobtracker.maxtasks.per.job to 4 in mapred-site.xml. In the subsequent job, I got this in the log which clearly shows the configuration has taken effect:
INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: nMaps=1 nReduces=1 max=4 Yet, the original problem remains. It doesn't go anywhere from 0%. Btw, when we get the jobQueue, should we specify a name for the queue we want? (eg. default) INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added successfully for user 'sreejith' to queue 'default' Also, is there a limit to the no.of tasks which can be obtained in a heartbeat? On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <[email protected]> wrote: > You don't need to explicitly call the initTasks(), you can find this > invocation in JobTracker.java class. > > >*Yet the **Cannot create task split for <my-job-id>** is still there*. > >About errors relating to the JobID, is this significant? --- INFO > >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1 > >nReduces=1 max=-1 > > Here it shows max=-1, which means max configured(i.e default) tasks for > the Job is -1. > > I think you can see this exception from JT log. > > LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks > + " max=" + maxTasks); > if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) { > throw new IOException( > "The number of tasks for this job " + > (numMapTasks + numReduceTasks) + > " exceeds the configured limit " + maxTasks); > } > > Can you try configuring some number for this configuration > 'mapred.jobtracker.maxtasks.per.job' and run the same Job? > > > Thanks > Devaraj k > > > -----Original Message----- > From: Sreejith Ramakrishnan [mailto:[email protected]] > Sent: 11 July 2013 13:26 > To: [email protected] > Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0% > > @Devaraj K > > *I tried explicitly calling incomingJob.initTasks()*and it gave me a > pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job > job_201307111248_0002 initialized successfully with 1 map tasks and 1 > reduce tasks > > *Yet the **Cannot create task split for <my-job-id>** is still there*. > About errors relating to the JobID, is this significant? --- INFO > org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1 > nReduces=1 max=-1 > > *P.S: In the log file I shared, anything with "sreejith == " were logged > by me. Not system-generated* > > > On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan < > [email protected]> wrote: > > > I've attached the relevant portion of the log. > > > > For easy testing, *I've attached the maven project* for the > > EDFJobScheduler. Just make a JAR from it and put it in > > $HADOOP_HOME/lib and add this property to *mapred-site.xml*: > > > > *<property> > > * > > * <name>mapred.jobtracker.taskScheduler</name> > > <value>org.apache.hadoop.mapred.EDFJobScheduler</value> > > * > > *</property>* > > > > > > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan < > > [email protected]> wrote: > > > >> I'm making my own scheduler. Since I had errors, I rewrote the > >> scheduler to do just a simple thing. When I give it a job, it > >> allocates 1 map() and 1 > >> reduce() to it. But that isn't happening. I used a barebones WordCount > pgm. > >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see: > >> > >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split > >> for <my-job-id> > >> > >> DETAILED INFO: > >> > >> Here's some snippets of code. The scheduler logic is inside the > >> assignTasks(): > >> > >> We get a JobInProgressListener in the constructor- > >> this.jobQueueJobInProgressListener = new > >> JobQueueJobInProgressListener(); > >> > >> In start() - > >> > >> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres > >> sListener); > >> > >> Then I just take the last element in that queue (I assume its the > >> latest submitted job) and assign to incomingJob (of type JobInProgress). > >> > >> Then I obtain a new map task as follows and append this task to a > >> list which is the return value of the assignTasks(): > >> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus, > >> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts()); > >> > >> > >> I've attached the scheduler. Can any expert point out if there's any > >> ignorant mistakes from my side? > >> > > > > >
