[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936219#comment-13936219 ] Hudson commented on MAPREDUCE-5751: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1727 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1727/]) MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577554) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.4.0 > > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936196#comment-13936196 ] Hudson commented on MAPREDUCE-5751: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1702 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1702/]) MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577554) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.4.0 > > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936147#comment-13936147 ] Hudson commented on MAPREDUCE-5751: --- FAILURE: Integrated in Hadoop-Yarn-trunk #510 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/510/]) MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577554) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.4.0 > > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935324#comment-13935324 ] Sangjin Lee commented on MAPREDUCE-5751: Thanks Jason! Much appreciated. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.4.0 > > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935099#comment-13935099 ] Hudson commented on MAPREDUCE-5751: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5328 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5328/]) MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577554) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.4.0 > > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925809#comment-13925809 ] Jason Lowe commented on MAPREDUCE-5751: --- +1, lgtm. Will wait a few days before committing to give [~tomwhite] a chance to comment. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch, mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924231#comment-13924231 ] Sangjin Lee commented on MAPREDUCE-5751: You have convinced me on this. :) Yes, I agree there is value in allowing users to run their own jetty version (as that is one of the very values that the isolated classloader provides) if they choose to do so. And that probably outweighs proofing against the possible additional action by the app master. I'll remove the changes for the system classes and submit the patch again. Thanks for the discussion. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924217#comment-13924217 ] Jason Lowe commented on MAPREDUCE-5751: --- Yeah, that makes sense. I know we can't make this bulletproof, rather I am thinking along the lines of which default is more likely to work for users. If a user-provided jetty is not likely to break then I'm thinking we shouldn't preclude it by default as that means the user doesn't get the jetty they expect. I thought by initializing the job classloader after we've started services that we should have avoided our jetty instance from trying to load the user-provided jetty classes. Since we are going to start the jetty services before setting the job classloader, I thought those jetty threads won't have the job classloader as their context classloader and therefore not be affected by any user-provided jetty jars. The AM is the only MR framework piece that uses jetty within a job, and we should know exactly what it will do and when it will init relative to user code. I doubt many users will be starting jetty in their OutputCommitter code that runs in the AM, and therefore it seemed likely that mixing the two jetty versions and using the job classloader feature would work as intended. The AM would get Hadoop's jetty version and tasks would get the user's version. If it's likely to work then I don't see a need to preclude it by default and make most users discover and change the system classes property to get what they want in the majority of the cases. Now if the user code starts up jetty inside the AM (i.e.: within their OutputCommitter which seems weird but who knows) or if it's an uber job then we probably have issues there. Anyway I'm far from a classloader or jetty expert. If you guys agree that jetty should be in the system classes by default for MR jobs then that's probably a good thing. It's definitely the safer thing from the framework perspective since we know we'll get what the framework wants (barring user.classpath.first shenanigans). I just didn't want to preclude by default users from using a different jetty version than Hadoop's dependency if it is likely to work for most cases. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jet
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924092#comment-13924092 ] Sangjin Lee commented on MAPREDUCE-5751: IMHO, making classloader isolation work at all times without fail is unfortunately pretty difficult, most because Java allows leaky ways of loading classes and it's difficult to always enforce the rules without breaking user code. One concrete scenario where the problem I mentioned above could happen without making jetty system classes is as follows. You're probably familiar with how it can happen (if so I apologize), but let me put it down still. Suppose we set the job classloader just prior to starting jobs. And suppose the app master decides to start a new jetty-based server *after* that point in time. Normally all the jetty classes would have been loaded (including the configuration class). But suppose it triggers a code path that wasn't exercised yet which uses the *context classloader* (getContextClassLoader()). This would now return the job classloader. If that class calls code like this, you would get a ClassCastException: {code} ClassLoader cl = Thread.currentThread().getContextClassLoader(); // Foo was loaded by the app classloader previously // FooImpl will be loaded by a different classloader // this line will throw a ClassCastException Foo foo = (Foo)Class.forName("org.jetty.foo.impl.FooImpl", cl); // or (Foo)cl.loadClass("org.jetty.foo.impl.FooImpl") {code} For the problem to surface, it would need to meet all of the following conditions: - its classes are loaded through the app classloader (i.e. the code path is outside the user task code path) - it uses the context classloader *after* it has been set to the job classloader - it tries to load another class (of its own) using that context classloader - it tries to cast it to its own type (interface or base class) that's already been loaded by the app classloader It is pretty hard to get into all of these. But it does happen, and the above code pattern for jetty was one. Another library that I know of that uses the context classloader is jaxb (already included in the system classes). So by including jetty in the system classes we gain just a bit more safety. This is bit of trial and error. But we do know what hadoop may do outside the user code, and hopefully this list does not need to grow. My 2 cents... > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:130
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923971#comment-13923971 ] Jason Lowe commented on MAPREDUCE-5751: --- My apologies, I still don't see why jetty needs to be part of the system classes. If that's true then one could argue guava, jackson, jersey, avro, etc. could also have the same issue. Couldn't those classes do some initialization in the hadoop classloader context then later some user code triggers some other internal classes of those dependencies to be loaded and we end up mixing and matching as we did with jetty? Or am I misunderstanding the issue and jetty is doing something none of our other dependencies will do? I want to make sure I understand the criteria for what should be put in the system classes. If I'm understanding the issue properly, it seems like a slippery slope from adding jetty to adding a large chunk of our dependencies which would effectively undermine the whole point of the app classloader. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923645#comment-13923645 ] Sangjin Lee commented on MAPREDUCE-5751: I think it may be prudent to keep the jetty classes in the system classes. The problem pattern was jetty trying to load a configuration class using the thread context classloader as part of initialization. If some jetty classes are loaded through the normal JVM app classloader, but the configuration class is loaded through the hadoop's job classloader, it would cause the ClassCastException, just like the one that's mentioned in the description. It doesn't even need multiple jetty versions to have this issue. Making jetty part of the system classes solves this issue if it should happen later on as part of user code. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923460#comment-13923460 ] Jason Lowe commented on MAPREDUCE-5751: --- I think moving the app classloader setup to just before we start the job makes a lot of sense. There should be no user code invoked up to that point, and it's close to where user code will be called as part of the job setup. As for the org.mortbay. addition to the system classes list, I'm not enough of a jetty expert to know if this is really necessary. Does jetty do things that prevent multiple jetty versions from co-existing if the app classloader is doing its job? It sounds like it worked when you tried leaving it out but still moving the app classloader init. If multiple jetty versions can really coexist in this setup then I think we shouldn't preclude it by default. But again I might be missing something jetty does that would break if we allow multiple versions even with the app classloader in place. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923404#comment-13923404 ] Sangjin Lee commented on MAPREDUCE-5751: That had skipped my mind. We do need the job classloader for that... Let me know if you have any feedback on the patch. Thanks! > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923371#comment-13923371 ] Jason Lowe commented on MAPREDUCE-5751: --- There is user code that runs in the ApplicationMaster, namely the OutputCommitter code. It's responsible for setup, output commit, output cleanup upon job failure, and task recovery if recovery is supported. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923342#comment-13923342 ] Sangjin Lee commented on MAPREDUCE-5751: Revisiting this topic. Stepping back, if the job is not uberized (i.e. !job.isUber()), then no user code runs in the MR app master, correct? Then, we should be able to set the job classloader only if the job is uberized. Thoughts? > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904481#comment-13904481 ] Sangjin Lee commented on MAPREDUCE-5751: Hmm, the condition we want to check as part of a unit test would be whether the context classloader is already set to the job classloader when the parent (CompositeService) service start is called. However, it doesn't seem trivial to mock/spy it so that we intercept the timing of the CompositeService.serviceStart() in a manner that's not too sensitive to the implementation of MRAppMaster.serviceStart(). Any tips are welcome... > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900752#comment-13900752 ] Gera Shegalov commented on MAPREDUCE-5751: -- I think you can easily add a test case to TestMRAppMaster > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899503#comment-13899503 ] Sangjin Lee commented on MAPREDUCE-5751: Meant to add that it was tested with a pseudo-cluster mode (single node), and with or without uberization. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898699#comment-13898699 ] Sangjin Lee commented on MAPREDUCE-5751: It is not straightforward to write a unit test for this due to the way the MRAppMaster code is structured. I confirm with either of these changes (system classes or delaying the call) the specific problem is fixed. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898656#comment-13898656 ] Hadoop QA commented on MAPREDUCE-5751: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628397/mapreduce-5751.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4351//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4351//console This message is automatically generated. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: mapreduce-5751.patch > > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897994#comment-13897994 ] Sangjin Lee commented on MAPREDUCE-5751: Yes, I think org.mortbay. should be added to the system classes by default. In fact, I think I'll do both changes: (1) adding org.mortbay. to the system classes, and (2) delay setting the classloaders until just before startJobs(). I think (2) still has value in case any other library that comes between these points happens to use the thread context classloader. I'll submit a patch shortly. > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897928#comment-13897928 ] Tom White commented on MAPREDUCE-5751: -- > And I think an argument can be made that it is perhaps the right fix. Jetty > is provided primarily by hadoop, and clients should use those jetty classes. I agree. How about adding it to the system classes by default? > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897060#comment-13897060 ] Sangjin Lee commented on MAPREDUCE-5751: Another option may be to delay setting the classloader as much as possible in MRAppMaster. Currently it's being set before calling initAndStartAppMaster(). We could delay it until after super.start() in MRAppMaster.start(), assuming no user code runs until that point. We could set the classloader just before startJobs(). > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[ https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897050#comment-13897050 ] Sangjin Lee commented on MAPREDUCE-5751: Currently we're working around this issue by adding the jetty classes (org.mortbay.) to mapreduce.job.classloader.system.classes. And I think an argument can be made that it is perhaps the right fix. Jetty is provided primarily by hadoop, and clients should use those jetty classes. Thoughts? > MR app master fails to start in some cases if mapreduce.job.classloader is > true > --- > > Key: MAPREDUCE-5751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > If mapreduce.job.classloader is set to true, and the MR client includes a > jetty jar in its libjars or job jar, the MR app master fails to start. A > typical stack trace we get is as follows: > {noformat} > java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration > cannot be cast to org.mortbay.jetty.webapp.Configuration > at > org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890) > at > org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) > at > org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676) > at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208) > at > org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259) > {noformat} > This happens because as part of the MR app master start the jetty classes are > loaded normally through the app classloader, but WebAppContext tries to load > the specific Configuration class via the thread context classloader (which > had been set to the user job classloader). -- This message was sent by Atlassian JIRA (v6.1.5#6160)