Florian Verhein created SPARK-5241:
--------------------------------------
Summary: spark-ec2 spark init scripts do not handle all hadoop (or
tachyon?) dependencies correctly
Key: SPARK-5241
URL: https://issues.apache.org/jira/browse/SPARK-5241
Project: Spark
Issue Type: Bug
Components: Build, EC2
Reporter: Florian Verhein
spark-ec2/spark/init.sh doesn't completely adhere to hadoop dependencies. This
may also be an issue for tachyon dependencies. Related: tachyon appears require
builds against the right version of hadoop also (probably causes this:
SPARK-3185).
Applies to the spark build from git checkout in spark/init.sh (I suspect this
should also be changed to using mvn as that's the reference build according to
the docs?).
May apply to pre-built spark in spark/init.sh as well, but I'm not sure about
this. E.g. I thought that the hadoop2.4 and cdh4.2 builds of spark are
different.
Also note that hadoop native is built from hadoop 2.4.1 on the AMI, and this is
used regardless of HADOOP_MAJOR_VERSION in the *-hdfs modules.
Tachyon is hard coded to 0.4.1 (which is probably built against hadoop1.x?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
