[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050216#comment-16050216 ]
Attila Sasvari commented on OOZIE-2944: --------------------------------------- Right now JavaActionExecutor creates jobConf at multiple places: -[createBaseHadoopConf()|https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L249] -[parseJobXmlAndConfiguration() |https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L342] As a result, actionConf and launcherJobConf are JobConf objects in {{createAppSubmissionContext()}}. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > ------------------------------------------------------------------- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Bug > Reporter: Attila Sasvari > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_000001 on 172.30.64.221_55437 > ========================================================================== > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/0000002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback notification trying > http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED > Callback notification to > http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED > succeeded > Callback notification succeeded > LogType:syslog > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1215 > Log Contents: > 2017-06-13 15:58:43,105 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > 2017-06-13 15:58:43,271 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at /0.0.0.0:8030 > 2017-06-13 15:58:43,844 INFO [main] org.apache.hadoop.io.compress.CodecPool: > Got brand-new compressor [.deflate] > 2017-06-13 15:58:43,972 INFO [main] > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Waiting for > application to be successfully unregistered. > 2017-06-13 15:58:44,078 INFO [AMRM Callback Handler Thread] > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted > while waiting for queue > java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) > {code} > I believe this issue is related to > [OOZIE-2918|https://issues.apache.org/jira/browse/OOZIE-2918]. > {{ShellActionExecutor}} extends {{JavaActionExecutor}} whose > {{needToAddMapReduceToClassPath()}} returns {{false}}. > {{LauncherMapper.getMainArguments()}} is called from {{LauncherAM}} via its > {{run()}} method. {{LauncherMapper}} extends > {{org.apache.hadoop.mapred.Mapper}} that is not on the classpath. -- This message was sent by Atlassian JIRA (v6.4.14#64029)