Hi Guys,
Wanted general help running twill on yarn cluster and class paths
How do you guys generally run the yarn application on a cluster?
e.g. I use
export HADOOP_CLASSPATH=all dependencies for my twill project
yarn jar myJar.yar main.client.class … arguments. (on one node on the cluster)
the yarn command adds all libraries used by hadoop into the class path. This
causes various library version errors. e.g. in my case, my hadoop cluster has
guava-11 which is incompatible with guava 13 used by Twill. Hence I get an error
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.util.concurrent.Service.addListener(Lcom/google/common/util/concurrent/Service$Listener;Ljava/util/concurrent/Executor;)V
at
org.apache.twill.internal.AbstractExecutionServiceController.start(AbstractExecutionServiceController.java:65)
at
org.apache.twill.yarn.YarnTwillPreparer.start(YarnTwillPreparer.java:290)
I would have to believe that there would be an easy way around this? any
suggestions?
Safder
