Hi Erik, I am working with TOT branch-0.9 (> 0.9.1) and the following works for me for maven build:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn -Pyarn -Dhadoop.version=2.3.0 -Dyarn.version=2.3.0 -DskipTests clean package And from http://spark.apache.org/docs/latest/running-on-yarn.html, for sbt build, you could try: SPARK_HADOOP_VERSION=2.3.0 SPARK_YARN=true sbt/sbt assembly Thanks, Rahul Singhal From: Erik Freed <erikjfr...@codecision.com<mailto:erikjfr...@codecision.com>> Reply-To: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Date: Friday 4 April 2014 7:58 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Hadoop 2.X Spark Client Jar 0.9.0 problem Hi All, I am not sure if this is a 0.9.0 problem to be fixed in 0.9.1 so perhaps already being addressed, but I am having a devil of a time with a spark 0.9.0 client jar for hadoop 2.X. If I go to the site and download: * Download binaries for Hadoop 2 (HDP2, CDH5): find an Apache mirror <http://www.apache.org/dyn/closer.cgi/incubator/spark/spark-0.9.0-incubating/spark-0.9.0-incubating-bin-hadoop2.tgz> or direct file download<http://d3kbcqa49mib13.cloudfront.net/spark-0.9.0-incubating-bin-hadoop2.tgz> I get a jar with what appears to be hadoop 1.0.4 that fails when using hadoop 2.3.0. I have tried repeatedly to build the source tree with the correct options per the documentation but always seemingly ending up with hadoop 1.0.4. As far as I can tell the reason that the jar available on the web site doesn't have the correct hadoop client in it, is because the build itself is having that problem. I am about to try to troubleshoot the build but wanted to see if anyone out there has encountered the same problem and/or if I am just doing something dumb (!) Anyone else using hadoop 2.X? How do you get the right client jar if so? cheers, Erik -- Erik James Freed CoDecision Software 510.859.3360 erikjfr...@codecision.com<mailto:erikjfr...@codecision.com> 1480 Olympus Avenue Berkeley, CA 94708 179 Maria Lane Orcas, WA 98245
