Attila Sasvari created OOZIE-2944: ------------------------------------- Summary: Shell action example does not work with Oozie on Yarn on hadoop 2.6 Key: OOZIE-2944 URL: https://issues.apache.org/jira/browse/OOZIE-2944 Project: Oozie Issue Type: Bug Reporter: Attila Sasvari
I tried to execute example shell action with a pseudo hadoop 2.6.0. The workflow failed with the exception below : {code} $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId application_1497357254962_0005 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Container: container_1497357254962_0005_01_000001 on 172.30.64.221_55437 ========================================================================== LogType:stderr Log Upload Time:13-Jun-2017 15:58:45 LogLength:1397 Log Contents: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Mapper at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 more LogType:stdout Log Upload Time:13-Jun-2017 15:58:45 LogLength:2152 Log Contents: Submitter user is: asasvari log4j: Trying to find [container-log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@6bc7c054. log4j: Using URL [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] for automatic log4j configuration. log4j: Reading configuration from URL jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties log4j: Hierarchy threshold set to [ALL]. log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. log4j: Level token is [INFO]. log4j: Category root set to INFO log4j: Parsing appender named "CLA". log4j: Parsing layout options for "CLA". log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. log4j: End of parsing for "CLA". log4j: Setting property [containerLogDir] to [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001]. log4j: Setting property [totalLogFileSize] to [0]. log4j: setFile called: /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001/syslog, true log4j: setFile ended log4j: Parsed "CLA" options. log4j: Parsing appender named "EventCounter". log4j: Parsed "EventCounter" options. log4j: Finished configuring. Using login user for UGI Launcher AM configuration loaded Starting the execution of prepare actions Completed the execution of prepare actions successfully Failing Oozie Launcher, null Oozie Launcher, uploading action data to HDFS sequence file: hdfs://localhost:9000/user/asasvari/oozie-asas/0000002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq Stopping AM Callback notification attempts left 0 Callback notification trying http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED Callback notification to http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED succeeded Callback notification succeeded LogType:syslog Log Upload Time:13-Jun-2017 15:58:45 LogLength:1215 Log Contents: 2017-06-13 15:58:43,105 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-06-13 15:58:43,271 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 2017-06-13 15:58:43,844 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate] 2017-06-13 15:58:43,972 INFO [main] org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 2017-06-13 15:58:44,078 INFO [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) {code} I believe this issue is related to [OOZIE-2918|https://issues.apache.org/jira/browse/OOZIE-2918]. {{ShellActionExecutor}} extends {{JavaActionExecutor}} whose {{needToAddMapReduceToClassPath()}} returns {{false}}. {{LauncherMapper.getMainArguments()}} is called from {{LauncherAM}} via its {{run()}} method. {{LauncherMapper}} extends {{org.apache.hadoop.mapred.Mapper}} that is not on the classpath. -- This message was sent by Atlassian JIRA (v6.4.14#64029)