[
https://issues.apache.org/jira/browse/YARN-9190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758129#comment-16758129
]
Zhankun Tang commented on YARN-9190:
------------------------------------
[~sunilg], [~billie.rinaldi], [~leftnoteasy] . I just encountered the Submarine
job failure against the trunk branch due to a similar yarn service master
error. And after run "yarn app -enableFastLaunch", the job succeeds.
Paste part of the error of AM container as below:
{code:java}
2019-02-01 16:38:14,098 [main] ERROR service.ServiceMaster - Error starting
service master
java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/api/records/LocalizationState
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at
org.codehaus.jackson.map.introspect.AnnotatedClass._addMemberMethods(AnnotatedClass.java:620)
at
org.codehaus.jackson.map.introspect.AnnotatedClass.resolveMemberMethods(AnnotatedClass.java:408)
at
org.codehaus.jackson.map.introspect.BasicClassIntrospector.classWithCreators(BasicClassIntrospector.java:185)
at
org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:157)
at
org.codehaus.jackson.map.introspect.BasicClassIntrospector.forDeserialization(BasicClassIntrospector.java:108)
at
org.codehaus.jackson.map.introspect.BasicClassIntrospector.forDeserialization(BasicClassIntrospector.java:16)
...
at
org.apache.hadoop.yarn.service.ServiceMaster.loadApplicationJson(ServiceMaster.java:256)
at
org.apache.hadoop.yarn.service.ServiceMaster.serviceInit(ServiceMaster.java:104)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.service.ServiceMaster.main(ServiceMaster.java:345)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.api.records.LocalizationState
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 60 more
2019-02-01 16:38:14,101 [main] INFO util.ExitUtil - Exiting with status 1:
Error starting service master
{code}
I suggest that we do more verification on this and prioritize this issue
because it could block a new user of Submarine.
> [Submarine] Submarine job will fail to run as a first job on a new created
> Hadoop 3.2.0 RC1 cluster
> ---------------------------------------------------------------------------------------------------
>
> Key: YARN-9190
> URL: https://issues.apache.org/jira/browse/YARN-9190
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Zhankun Tang
> Assignee: Sunil Govindan
> Priority: Minor
>
> This issue was found when verifying submarine in Hadoop 3.2.0 RC1 planning.
> The reproduce steps are:
> # Init a new HDFS and YARN (LinuxContainerExecutor and Docker enabled)
> # Before run any other yarn service job, use yarn user to submit a submarine
> job
> The job will fail with below error:
>
> {code:java}
> LogType:serviceam-err.txt
> LogLastModifiedTime:Thu Jan 10 21:15:23 +0800 2019
> LogLength:86
> LogContents:
> Error: Could not find or load main class
> org.apache.hadoop.yarn.service.ServiceMaster
> End of LogType:serviceam-err.txt
> {code}
> This seems because the dependencies are not ready as the service client
> reported:
> {code:java}
> 2019-01-10 21:50:47,380 WARN client.ServiceClient: Property
> yarn.service.framework.path has a value
> /yarn-services/3.2.0/service-dep.tar.gz, but is not a valid file
> 2019-01-10 21:50:47,381 INFO client.ServiceClient: Uploading all dependency
> jars to HDFS. For faster submission of apps, set config property
> yarn.service.framework.path to the dependency tarball location. Dependency
> tarball can be uploaded to any HDFS path directly or by using command: yarn
> app -enableFastLaunch [<Destination Folder>]{code}
>
> When this error happens, I found that there is no “/yarn-services” directory
> created in HDFS.
> But after I run “yarn app -launch my-sleeper sleeper”, the “/yarn-services”
> created in HDFS and then the submarine job can run successfully.
> {code:java}
> yarn@master0-VirtualBox:~/apache-hadoop-install-dir/hadoop-dev-workspace$
> hdfs dfs -ls /yarn-services/3.2.0/*
> -rwxr-xr-x 1 yarn supergroup 93596476 2019-01-11 08:23
> /yarn-services/3.2.0/service-dep.tar.gz{code}
> It seems an issue of yarn service in 3.2.0 RC1 and I files this Jira to track
> it.
>
> And verified that trunk branch doesn't have this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
