[
https://issues.apache.org/jira/browse/MAPREDUCE-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064044#comment-14064044
]
Jason Lowe commented on MAPREDUCE-5957:
---------------------------------------
Thanks for updating the patch. I do think the wrapper makes it a bit clearer
we're calling something within the context of the job classloader.
One of the drawbacks of the wrapper is that it's harder to deal with checked
exceptions. For example, MRAppMaster.isRecoverySupported used to propagate any
IOException thrown by the user-provided committer code, but the wrapper now
transforms this into a runtime exception. Not sure it really makes a whole lot
of difference in practice for that case, but it is a subtle change in behavior.
Do you think it's worth adding a wrapper form that can throw Exception (or
maybe just IOException)? Thinking along similar lines as PrivilegedAction vs.
PrivilegedExceptionAction in the doAs scenarios. I guess callers can catch
YarnRuntimeException with the current patch and check the cause if they
want/need to convert it back to a checked exception themselves, but curious on
your thoughts.
Otherwise I think the patch looks good. Couple of minor additional items:
- Is there a reason the init transition calls MRAppMaster.createJobClassloader
rather than having MRAppMaster.serviceInit do it? Seems more natural to have
the serviceInit method initialize the MRAppMaster data members if possible.
- Not strictly related to this JIRA, but I happened to notice that
MRAppMaster.isRecoverySupported takes a committer argument that it totally
ignores. Since we're in the area, it would be nice to clean that up by either
removing the argument or use it rather than the committer member. Totally OK
with me if you'd rather postpone that minor cleanup to another JIRA.
> AM throws ClassNotFoundException with job classloader enabled if custom
> output format/committer is used
> -------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5957
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5957
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: MAPREDUCE-5957.patch, MAPREDUCE-5957.patch,
> MAPREDUCE-5957.patch, MAPREDUCE-5957.patch
>
>
> With the job classloader enabled, the MR AM throws ClassNotFoundException if
> a custom output format class is specified.
> {noformat}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:473)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:374)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1459)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1456)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException: Class
> com.foo.test.TestOutputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
> ... 10 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
