Other YARN apps work fine. For example, I just successfully ran the stock MapReduce wordcount example (and of course, MapReduce is a YARN application).
I ran HelloWorld in debug mode earlier and found that yarnClasspath contains the following: $HADOOP_CONF_DIR $HADOOP_COMMON_HOME/* $HADOOP_COMMON_HOME/lib/* $HADOOP_HDFS_HOME/* $HADOOP_HDFS_HOME/lib/* $HADOOP_MAPRED_HOME/* $HADOOP_MAPRED_HOME/lib/* $HADOOP_YARN_HOME/* $HADOOP_YARN_HOME/lib/* I don't know whether these environment variables are supposed to be resolved to path variables already or if that happens later. I also don't know if I'm supposed to explicitly declare these environment variables somewhere, and if so, I do not know where in the Hadoop configuration it is ideal for me to declare them. I tried declaring them in hadoop_env.sh and core-site.xml, but I'm not sure I did it right, and at least in my initial efforts declaring these variables did not seem to prevent the error. If it is the correct thing for me to set these variables appropriately somewhere, then I will continue to try to do so. On Mon, Jun 12, 2017 at 3:46 PM, Yuliya Feldman <yul...@dremio.com> wrote: > I don't think it is an issue with classpath on the client - since it gets > to start AppMaster container > > Is any other YARN app runs OK? > > May be YarnConfiguration.YARN_APPLICATION_CLASSPATH is not producing right > jars > > Look at HelloWorld code for yarnClasspath > > > > On Mon, Jun 12, 2017 at 1:22 PM, Chris Hebert < > chris.hebert-...@digitalreasoning.com> wrote: > > > I hate to ask this here, but it won't work, so whatever. > > > > I followed the HelloWorld section of the Getting Started guide < > > http://twill.apache.org/GettingStarted.html> on my cluster with Hadoop > and > > Zookeeper set up and functioning properly. > > > > git clone https://github.com/apache/twill.git > > cd twill > > mvn clean install -DskipTests > > > > export > > CP=twill-examples/yarn/target/twill-examples-yarn-0.12.0- > > SNAPSHOT.jar:`hadoop > > classpath` > > java -cp $CP org.apache.twill.example.yarn.HelloWorld > > my.zookeeper.domain:2181 > > > > Yes, `hadoop classpath` echoes all the relevant jar directories. > > > > The command runs well for a bit with multiple: > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application > > status for HelloWorldRunnable application_1111111111111_0001: ACCEPTED > > > > until: > > [ STARTING] DEBUG o.a.hadoop.service.AbstractService - Service: > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state > > STOPPED > > [ STARTING] DEBUG org.apache.hadoop.ipc.Client - stopping client from > > cache: org.apache.hadoop.ipc.Client@4d465b11 > > [ STARTING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn application > > status for HelloWorldRunnable application_1111111111111_0001: FAILED > > ... > > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Yarn > > application completed with failure HelloWorldRunnable... > > > > The ResourceManager reveals: > > Application application_1111111111111_0001 failed 2 times due to AM > > Container for appattempt_1111111111111_0001_000002 exited with > exitCode: 1 > > ... > > Diagnostics: Exception from container-launch. > > > > The corresponding YARN logs for each DataNode reveal: > > Exception in thread "main" java.lang.NoClassDefFoundError: > > org/apache/hadoop/conf/Configuration > > at java.lang.Class.getDeclaredMethods0(Native Method) > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) > > at java.lang.Class.privateGetMethodRecursive(Class.java:3048) > > at java.lang.Class.getMethod0(Class.java:3018) > > at java.lang.Class.getMethod(Class.java:1784) > > at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:70) > > Caused by: java.lang.ClassNotFoundException: > > org.apache.hadoop.conf.Configuration > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 6 more > > Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain) > > using classloader java.net.URLClassLoader with classpath: [ > > *A list of several classpaths like > > "file:/some/path/yarn/local/usercache/my.username/appcache/application_ > > 1111111111111_0001/container_e10_ > > 1111111111111 > > _0001_02_000001/application.jar/lib/twill-examples-yarn-0. > > 12.0-SNAPSHOT.jar" > > But none of which are paths to any Hadoop jars of the sort that are > > referenced in $CP* > > ] > > > > What am I missing? > > > > I've spent an embarrassingly large amount of time on this fiddling with > > environment variables and Hadoop configuration. (I'm an intern learning > > this stuff the hard way, so it's not really embarrassing, just > > substantial.) > > >
