You don't need to explicitly call the initTasks(), you can find this invocation
in JobTracker.java class.
>*Yet the **Cannot create task split for <my-job-id>** is still there*.
>About errors relating to the JobID, is this significant? --- INFO
>org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
>nReduces=1 max=-1
Here it shows max=-1, which means max configured(i.e default) tasks for the Job
is -1.
I think you can see this exception from JT log.
LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks + "
max=" + maxTasks);
if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
throw new IOException(
"The number of tasks for this job " +
(numMapTasks + numReduceTasks) +
" exceeds the configured limit " + maxTasks);
}
Can you try configuring some number for this configuration
'mapred.jobtracker.maxtasks.per.job' and run the same Job?
Thanks
Devaraj k
-----Original Message-----
From: Sreejith Ramakrishnan [mailto:[email protected]]
Sent: 11 July 2013 13:26
To: [email protected]
Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
@Devaraj K
*I tried explicitly calling incomingJob.initTasks()*and it gave me a pleasant
INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307111248_0002 initialized successfully with 1 map tasks and 1 reduce
tasks
*Yet the **Cannot create task split for <my-job-id>** is still there*.
About errors relating to the JobID, is this significant? --- INFO
org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
nReduces=1 max=-1
*P.S: In the log file I shared, anything with "sreejith == " were logged by me.
Not system-generated*
On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
[email protected]> wrote:
> I've attached the relevant portion of the log.
>
> For easy testing, *I've attached the maven project* for the
> EDFJobScheduler. Just make a JAR from it and put it in
> $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
>
> *<property>
> *
> * <name>mapred.jobtracker.taskScheduler</name>
> <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> *
> *</property>*
>
>
> On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> [email protected]> wrote:
>
>> I'm making my own scheduler. Since I had errors, I rewrote the
>> scheduler to do just a simple thing. When I give it a job, it
>> allocates 1 map() and 1
>> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
>> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>>
>> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
>> for <my-job-id>
>>
>> DETAILED INFO:
>>
>> Here's some snippets of code. The scheduler logic is inside the
>> assignTasks():
>>
>> We get a JobInProgressListener in the constructor-
>> this.jobQueueJobInProgressListener = new
>> JobQueueJobInProgressListener();
>>
>> In start() -
>>
>> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
>> sListener);
>>
>> Then I just take the last element in that queue (I assume its the
>> latest submitted job) and assign to incomingJob (of type JobInProgress).
>>
>> Then I obtain a new map task as follows and append this task to a
>> list which is the return value of the assignTasks():
>> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
>> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>>
>>
>> I've attached the scheduler. Can any expert point out if there's any
>> ignorant mistakes from my side?
>>
>
>
