[jira] [Commented] (MAPREDUCE-6015) Make MR ApplicationMaster disable loading user's jars firstly

[Bing Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084528#comment-14084528
 ] 

Bing Jiang commented on MAPREDUCE-6015:
---

[~sjlee0], Thanks very much.  Agree with you at the point of 
'mapreduce.job.classloader', it can tackle lots of issue of class conflicts. 
However, if the user's jar contains some conflict classes of launching AM. It 
seems that there is no way to finish using that property. e.g., jetty-6.1.26 
(hadoop system jars) is conflicting with jetty-6.1.11.jar (users jar). AM 
failed to launch.
==
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError:
org.mortbay.jetty.webapp.WebAppContext.getUnavailableException()Ljava/lang/Throwable;
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:813)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:141)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1072)
===

And user logic code is dependent upon jetty-6.1.11.jar, so it cannot disable 
mapreduce.user.classpath.first. At this time, it gives me a impression to treat 
AM and normal Tasks separately.  

> Make MR ApplicationMaster disable loading user's jars firstly 
> --
>
> Key: MAPREDUCE-6015
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6015
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 2.4.1
>Reporter: Bing Jiang
>
> In most cases, we want to use -Dmapreduce.user.classpath.first=true to pick 
> user's jars ahead of hadoop system's jars, which can make tasks run based 
> upon the customized environment under the circumstance that hadoop system 
> default library contains the different version of dependent jars.
> However, using -Dmapreduce.user.classpath.first=true will cause 
> ApplicationMaster failure to launch due to conflicting classes.
> In most cases, if users do not customize the ApplicationMaster for MapReduce 
> framework, I believe we can treat MRAppMaster different with 
> MapTask/ReduceTask at the point of loading user's jar in classloader. 
> I believe it can provide  a property of  
> '-Dmapreduce.am.user.classpath.first=false'  to disable the feature of 
> loading user's jars firstly. 
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (MAPREDUCE-6015) Make MR ApplicationMaster disable loading user's jars firstly

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081023#comment-14081023
 ] 

Sangjin Lee commented on MAPREDUCE-6015:


I understand the nature of the issue, but it also makes me wonder how effective 
the proposed solution would be. There are cases like uberized jobs where the 
proposed config probably cannot be used. Even in the case of non-uberized jobs, 
the MR app master loads several user classes and those may still need 
user.classpath.first for their operation.

Curious, have you used mapreduce.job.classloader? To me that should be a much 
more robust solution, including issues in the MR app master (with the fix that 
was recently made in MAPREDUCE-5957).

> Make MR ApplicationMaster disable loading user's jars firstly 
> --
>
> Key: MAPREDUCE-6015
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6015
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 2.4.1
>Reporter: Bing Jiang
>
> In most cases, we want to use -Dmapreduce.user.classpath.first=true to pick 
> user's jars ahead of hadoop system's jars, which can make tasks run based 
> upon the customized environment under the circumstance that hadoop system 
> default library contains the different version of dependent jars.
> However, using -Dmapreduce.user.classpath.first=true will cause 
> ApplicationMaster failure to launch due to conflicting classes.
> In most cases, if users do not customize the ApplicationMaster for MapReduce 
> framework, I believe we can treat MRAppMaster different with 
> MapTask/ReduceTask at the point of loading user's jar in classloader. 
> I believe it can provide  a property of  
> '-Dmapreduce.am.user.classpath.first=false'  to disable the feature of 
> loading user's jars firstly. 
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)