Do those env vars resolve on the NodeManager nodes correctly? On Mon, Jun 12, 2017 at 2:05 PM, Chris Hebert < chris.hebert-...@digitalreasoning.com> wrote:
> Other YARN apps work fine. For example, I just successfully ran the stock > MapReduce wordcount example (and of course, MapReduce is a YARN > application). > > I ran HelloWorld in debug mode earlier and found that yarnClasspath > contains the following: > $HADOOP_CONF_DIR > $HADOOP_COMMON_HOME/* > $HADOOP_COMMON_HOME/lib/* > $HADOOP_HDFS_HOME/* > $HADOOP_HDFS_HOME/lib/* > $HADOOP_MAPRED_HOME/* > $HADOOP_MAPRED_HOME/lib/* > $HADOOP_YARN_HOME/* > $HADOOP_YARN_HOME/lib/* > > I don't know whether these environment variables are supposed to be > resolved to path variables already or if that happens later. I also don't > know if I'm supposed to explicitly declare these environment variables > somewhere, and if so, I do not know where in the Hadoop configuration it is > ideal for me to declare them. I tried declaring them in hadoop_env.sh and > core-site.xml, but I'm not sure I did it right, and at least in my initial > efforts declaring these variables did not seem to prevent the error. If it > is the correct thing for me to set these variables appropriately somewhere, > then I will continue to try to do so. > > On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yul...@dremio.com> wrote: > > > I don't think it is an issue with classpath on the client - since it gets > > to start AppMaster container > > > > Is any other YARN app runs OK? > > > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing > right > > jars > > > > Look at HelloWorld code for yarnClasspath > > > > > > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert < > > chris.hebert-...@digitalreasoning.com> wrote: > > > > > I hate to ask this here, but it won't work, so whatever. > > > > > > I followed the HelloWorld section of the Getting Started guide < > > > http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop > > and > > > Zookeeper set up and functioning properly. > > > > > > git clone https://github.com/apache/twill.git > > > cd twill > > > mvn clean install -DskipTests > > > > > > export > > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0- > > > SNAPSHOT.jar:`hadoop > > > classpath` > > > java -cp $CP org.apache.twill.example.yarn.HelloWorld > > > my.zookeeper.domain:2181 > > > > > > Yes, `hadoop classpath` echoes all the relevant jar directories. > > > > > > The command runs well for a bit with multiple: > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn > application > > > status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED > > > > > > until: > > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service: > > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state > > > STOPPED > > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from > > > cache: org.apache.hadoop.ipc.Client@4d465b11 > > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn > application > > > status for HelloWorldRunnable application_1111111111111_0001: FAILED > > > ... > > > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > > Yarn > > > application completed with failure HelloWorldRunnable... > > > > > > The ResourceManager reveals: > > > Application application_1111111111111_0001 failed 2 times due to AM > > > Container for appattempt_1111111111111_0001_000002 exited with > > exitCode: 1 > > > ... > > > Diagnostics: Exception from container-launch. > > > > > > The corresponding YARN logs for each DataNode reveal: > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > org/apache/hadoop/conf/Configuration > > > at java.lang.Class.getDeclaredMethods0(Native Method) > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) > > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048) > > > at java.lang.Class.getMethod0(Class.java:3018) > > > at java.lang.Class.getMethod(Class.java:1784) > > > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70) > > > Caused by: java.lang.ClassNotFoundException: > > > org.apache.hadoop.conf.Configuration > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > ... 6 more > > > Launch class (org.apache.twill.internal.appmaster. > ApplicationMasterMain) > > > using classloader java.net.URLClassLoader with classpath: [ > > > *A list of several classpaths like > > > "file:/some/path/yarn/local/usercache/my.username/ > appcache/application_ > > > 1111111111111_0001/container_e10_ > > > 1111111111111 > > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0. > > > 12.0-SNAPSHOT.jar" > > > But none of which are paths to any Hadoop jars of the sort that are > > > referenced in $CP* > > > ] > > > > > > What am I missing? > > > > > > I've spent an embarrassingly large amount of time on this fiddling with > > > environment variables and Hadoop configuration. (I'm an intern learning > > > this stuff the hard way, so it's not really embarrassing, just > > > substantial.) > > > > > >
