[
https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073038#comment-13073038
]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
Thanks for the review, Vinod. That makes sense.
Another question/concern though: I see that in JobSubmitter.java
(submitJobInternal), it originally generates the "submitJobDir" based on a call
to JobSubmissionFiles.getStagingDir & the jobId. (where getStagingDir
ultimately goes off to YARNRunner.getStagingAreaDir(), which goes to the
ResourceMgrDelegate & then MRApps). Anyway, that staging area directory seems
to include the user name and ".staging" as well as the value from the
"yarn.apps.stagingDir". Is that the value we should be using instead of just
getting the setting from conf? If so, maybe it would make sense to just pass
the actual staging directory in instead of the actual Configuration object.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch,
> MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g.
> hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error
> (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty
> string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at
> org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at
> org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at
> org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at
> org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at
> org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
