Hello community,
tried to run storm app on yarn, using cloudera hadoop and spark distro
(from http://archive.cloudera.com/cdh5/cdh/5)
hadoop version: hadoop-2.3.0-cdh5.0.3.tar.gz
spark version: spark-0.9.0-cdh5.0.3.tar.gz
DEFAULT_YARN_APPLICATION_CLASSPATH is part of hadoop-api-yarn jar ...
Somewhere in here, you are not actually running vs Hadoop 2 binaries.
Your cluster is certainly Hadoop 2, but your client is not using the
Hadoop libs you think it is (or your compiled binary is linking
against Hadoop 1, which is the default for Spark -- did you change
it?)
On Wed, Jul 16, 2014
Andrew,
Are you running on a CM-managed cluster? I just checked, and there is a
bug here (fixed in 1.0), but it's avoided by having
yarn.application.classpath defined in your yarn-site.xml.
-Sandy
On Wed, Jul 16, 2014 at 10:02 AM, Sean Owen so...@cloudera.com wrote:
Somewhere in here, you
OK, if you're sure your binary has Hadoop 2 and/or your classpath has
Hadoop 2, that's not it. I'd look at Sandy's suggestion then.
On Wed, Jul 16, 2014 at 6:11 PM, Andrew Milkowski amgm2...@gmail.com wrote:
thanks Sean! so what I did is in project/SparkBuild.scala I made it compile
with
thanks Sandzy, no CM-managed cluster, straight from cloudera tar (
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.3.tar.gz)
trying your suggestion immediate! thanks so much for taking time..
On Wed, Jul 16, 2014 at 1:10 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
Andrew,
Are
Sandy, perfect! you saved me tons of time! added this in yarn-site.xml job
ran to completion
Can you do me (us) a favor and push newest and patched spark/hadoop to cdh5
(tar's) if possible
and thanks again for this (huge time saver)
On Wed, Jul 16, 2014 at 1:10 PM, Sandy Ryza
For others, to solve topic problem: in yarn-site.xml add:
property
nameyarn.application.classpath/name
value$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/share/hadoop/common/*,
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
