[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050875#comment-16050875 ] Hadoop QA commented on OOZIE-2944: -- Testing JIRA OOZIE-2944 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any line longer than 132 .{color:green}+1{color} the patch does adds/modifies 9 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings .{color:red}WARNING{color}: the current HEAD has 6 Javadoc warning(s) {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 1966 .Tests rerun: 79 .Tests failed at first run: org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM, {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} {color:red}. There is at least one warning, please check{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3896/ > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Sub-task >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch, OOZIE-2944-02.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader >
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050696#comment-16050696 ] Peter Cseh commented on OOZIE-2944: --- +1 (pending Jenkins) on the second patch. Thanks for the fix! > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Sub-task >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch, OOZIE-2944-02.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback notification trying >
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050668#comment-16050668 ] Peter Cseh commented on OOZIE-2944: --- failures are due to OOZIE-2794 i guess. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Sub-task >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch, OOZIE-2944-02.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback notification trying >
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050582#comment-16050582 ] Peter Cseh commented on OOZIE-2944: --- Please update the apidoc for the {{createJobConf}} as well and rename it to something along the lines of createConfiguration/createDefaultConfiguration. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Sub-task >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050519#comment-16050519 ] Hadoop QA commented on OOZIE-2944: -- Testing JIRA OOZIE-2944 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any line longer than 132 .{color:green}+1{color} the patch does adds/modifies 4 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings .{color:red}WARNING{color}: the current HEAD has 6 Javadoc warning(s) {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:red}-1 TESTS{color} .Tests run: 1966 .Tests failed: 6 .Tests errors: 2 .The patch failed the following testcases: . testLoadDump(org.apache.oozie.tools.TestDBLoadDump) . testMain(org.apache.oozie.action.hadoop.TestHiveMain) . testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain) . testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain) . testPigScript(org.apache.oozie.action.hadoop.TestPigMain) . testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI) .Tests failing with errors: . testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration) . testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration) {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} {color:red}. There is at least one warning, please check{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3895/ > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Sub-task >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException:
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050307#comment-16050307 ] Attila Sasvari commented on OOZIE-2944: --- [~pbacsko] [~rkanter] [~gezapeti] can you have a look at the patch? Thanks > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari >Assignee: Attila Sasvari > Attachments: OOZIE-2944-01.patch > > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback notification trying >
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050216#comment-16050216 ] Attila Sasvari commented on OOZIE-2944: --- Right now JavaActionExecutor creates jobConf at multiple places: -[createBaseHadoopConf()|https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L249] -[parseJobXmlAndConfiguration() |https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L342] As a result, actionConf and launcherJobConf are JobConf objects in {{createAppSubmissionContext()}}. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048258#comment-16048258 ] Robert Kanter commented on OOZIE-2944: -- {{LauncherAM}} should use {{Configuration}} instead of {{JobConf}}. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property [containerLogDir] to > [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01]. > log4j: Setting property [totalLogFileSize] to [0]. > log4j: setFile called: > /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog, > true > log4j: setFile ended > log4j: Parsed "CLA" options. > log4j: Parsing appender named "EventCounter". > log4j: Parsed "EventCounter" options. > log4j: Finished configuring. > Using login user for UGI > Launcher AM configuration loaded > Starting the execution of prepare actions > Completed the execution of prepare actions successfully > Failing Oozie Launcher, null > Oozie Launcher, uploading action data to HDFS sequence file: > hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq > Stopping AM > Callback notification attempts left 0 > Callback notification trying > http://0.0.0.0:11000/oozie/callback?id=002-170613150915408-oozie-asas-W@shell-node=FAILED > Callback notification to >
[jira] [Commented] (OOZIE-2944) Shell action example does not work with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048160#comment-16048160 ] Attila Sasvari commented on OOZIE-2944: --- I changed a LauncherMapper so that it does not extend the Mapper class and rebuilt Oozie. I submitted the shell example that now failed with another, but similar, reason: {code} Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf at org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 3 more {code} If I override {{needToAddMapReduceToClassPath()}} in {{ShellActionExecutor}}, so that it returns true, shell example workflow passes (verified). >From this, I suppose each ActionExecutor that extends JavaActionExecutor will >fail if they do not override {{needToAddMapReduceToClassPath()}} and MR is not >in the class path. > Shell action example does not work with Oozie on Yarn on hadoop 2.6 > --- > > Key: OOZIE-2944 > URL: https://issues.apache.org/jira/browse/OOZIE-2944 > Project: Oozie > Issue Type: Bug >Reporter: Attila Sasvari > > I tried to execute example shell action with a pseudo hadoop 2.6.0. The > workflow failed with the exception below : > {code} > $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId > application_1497357254962_0005 > 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437 > == > LogType:stderr > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:1397 > Log Contents: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/mapred/Mapper > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540) > at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170) > at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148) > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 15 more > LogType:stdout > Log Upload Time:13-Jun-2017 15:58:45 > LogLength:2152 > Log Contents: > Submitter user is: asasvari > log4j: Trying to find [container-log4j.properties] using context classloader > sun.misc.Launcher$AppClassLoader@6bc7c054. > log4j: Using URL > [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] > for automatic log4j configuration. > log4j: Reading configuration from URL > jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties > log4j: Hierarchy threshold set to [ALL]. > log4j: Parsing for [root] with value=[INFO,CLA, EventCounter]. > log4j: Level token is [INFO]. > log4j: Category root set to INFO > log4j: Parsing appender named "CLA". > log4j: Parsing layout options for "CLA". > log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n]. > log4j: End of parsing for "CLA". > log4j: Setting property