[
https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085345#comment-13085345
]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
After running some more tests, there are still some problems with the latest
patch (v7).
First, the createFakeJobReport calls in ClientServiceDelegate.getJobStatus seem
to need a valid jobfile since the MRReliability test uses that codepath and
expects a valid jobFile.
Also, I noticed that with adding the ".staging" and user (the way MRApps
generates the stagingAreaDir) to the current setting in yarn-default.xml
(/tmp/hadoop-yarn/${user.name}/staging), we end up with duplicated information
in the path. I get this:
/tmp/hadoop-yarn/<username>/staging/<username>/.staging/job_XXX/job.xml
(Notice the duplicated <username> and the fact that we have both staging and
.staging. I'm not sure how this would work when another user's jobfile is
requested)
It seems like either the default should not include "<username>/staging", or we
shouldn't be adding both later (unless I'm missing something here). For
consistency, shouldn't we have the jobfile path & stagingAreaDir match what
they were in previous versions?
Looking at a log from a mapreduce job on trunk (not MR-279), I see this:
hdfs://<hostname>/user/<username>/.staging/job_XXXX/job.xml
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch,
> MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch,
> MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g.
> hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error
> (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty
> string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at
> org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at
> org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at
> org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at
> org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at
> org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
