OK, from scanning the pom.xml, I think you would try: -Pyarn -Dhadoop.version=2.6.0
If it doesn't package or pass tests, then I'd assume it's not supported :( On Fri, Feb 13, 2015 at 7:33 PM, Grandl Robert <rgra...@yahoo.com> wrote: > I am trying to run BlinkDB(https://github.com/sameeragarwal/blinkdb) which > seems to work only with Spark 0.9. However, if I want to access HDFS I need > to compile Spark against Hadoop version which is running on my > cluster(2.6.0). Hence, the versions problem ... > > > > On Friday, February 13, 2015 11:28 AM, Sean Owen <so...@cloudera.com> wrote: > > > Oh right, you said Spark 0.9. Those profiles won't exist back then. I > don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles > were introduced later to fix up some compatibility. Why not use 1.2.1? > > On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert <rgra...@yahoo.com> wrote: >> Thanks Sean for your prompt response. >> >> I was trying to compile as following: >> mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package >> >> but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, >> and >> all hadoop jars are in .m2 repository. >> >> Do you have any idea what might happens ? >> >> Robert >> >> [WARNING] Class com.google.protobuf.Parser not found - continuing with a >> stub. >> [ERROR] error while loading RpcResponseHeaderProto, class file >> >> '/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)' >> is broken >> (class java.lang.NullPointerException/null) >> [WARNING] one warning found >> [ERROR] one error found >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Reactor Summary: >> [INFO] >> [INFO] Spark Project Parent POM .......................... SUCCESS >> [2.537s] >> [INFO] Spark Project Core ................................ FAILURE >> [25.917s] >> [INFO] Spark Project Bagel ............................... SKIPPED >> [INFO] Spark Project GraphX .............................. SKIPPED >> [INFO] Spark Project ML Library .......................... SKIPPED >> [INFO] Spark Project Streaming ........................... SKIPPED >> [INFO] Spark Project Tools ............................... SKIPPED >> [INFO] Spark Project REPL ................................ SKIPPED >> [INFO] Spark Project Assembly ............................ SKIPPED >> [INFO] Spark Project External Twitter .................... SKIPPED >> [INFO] Spark Project External Kafka ...................... SKIPPED >> [INFO] Spark Project External Flume ...................... SKIPPED >> [INFO] Spark Project External ZeroMQ ..................... SKIPPED >> [INFO] Spark Project External MQTT ....................... SKIPPED >> [INFO] Spark Project Examples ............................ SKIPPED >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD FAILURE >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 30.002s >> [INFO] Finished at: Fri Feb 13 11:21:36 PST 2015 >> [INFO] Final Memory: 49M/1226M >> [INFO] >> ------------------------------------------------------------------------ >> [WARNING] The requested profile "hadoop-2.4" could not be activated >> because >> it does not exist. >> [ERROR] Failed to execute goal >> net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) >> on >> project spark-core_2.10: Execution scala-compile-first of goal >> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed >> -> >> [Help 1] >> >> >> >> On Friday, February 13, 2015 11:16 AM, Sean Owen <so...@cloudera.com> >> wrote: >> >> >> If you just need standalone mode, you don't need "-Pyarn". There is no >> "-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set >> -Dhadoop.version=2.6.0. That should be it. >> >> If that still doesn't work, define "doesn't succeed". >> >> On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert >> <rgra...@yahoo.com.invalid> wrote: >>> Hi guys, >>> >>> Probably a dummy question. Do you know how to compile Spark 0.9 to easily >>> integrate with HDFS 2.6.0 ? >>> >>> I was trying >>> sbt/sbt -Pyarn -Phadoop-2.6 assembly >>> or >>> mvn -Dhadoop.version=2.6.0 -DskipTests clean package >>> >>> but none of these approaches succeeded. >>> >>> Thanks, >>> Robert >> >> > > > > > > On Friday, February 13, 2015 11:28 AM, Sean Owen <so...@cloudera.com> wrote: > > > Oh right, you said Spark 0.9. Those profiles won't exist back then. I > don't even know if Hadoop 2.6 will work with 0.9 as-is. The profiles > were introduced later to fix up some compatibility. Why not use 1.2.1? > > On Fri, Feb 13, 2015 at 7:26 PM, Grandl Robert <rgra...@yahoo.com> wrote: >> Thanks Sean for your prompt response. >> >> I was trying to compile as following: >> mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package >> >> but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, >> and >> all hadoop jars are in .m2 repository. >> >> Do you have any idea what might happens ? >> >> Robert >> >> [WARNING] Class com.google.protobuf.Parser not found - continuing with a >> stub. >> [ERROR] error while loading RpcResponseHeaderProto, class file >> >> '/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)' >> is broken >> (class java.lang.NullPointerException/null) >> [WARNING] one warning found >> [ERROR] one error found >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Reactor Summary: >> [INFO] >> [INFO] Spark Project Parent POM .......................... SUCCESS >> [2.537s] >> [INFO] Spark Project Core ................................ FAILURE >> [25.917s] >> [INFO] Spark Project Bagel ............................... SKIPPED >> [INFO] Spark Project GraphX .............................. SKIPPED >> [INFO] Spark Project ML Library .......................... SKIPPED >> [INFO] Spark Project Streaming ........................... SKIPPED >> [INFO] Spark Project Tools ............................... SKIPPED >> [INFO] Spark Project REPL ................................ SKIPPED >> [INFO] Spark Project Assembly ............................ SKIPPED >> [INFO] Spark Project External Twitter .................... SKIPPED >> [INFO] Spark Project External Kafka ...................... SKIPPED >> [INFO] Spark Project External Flume ...................... SKIPPED >> [INFO] Spark Project External ZeroMQ ..................... SKIPPED >> [INFO] Spark Project External MQTT ....................... SKIPPED >> [INFO] Spark Project Examples ............................ SKIPPED >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD FAILURE >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 30.002s >> [INFO] Finished at: Fri Feb 13 11:21:36 PST 2015 >> [INFO] Final Memory: 49M/1226M >> [INFO] >> ------------------------------------------------------------------------ >> [WARNING] The requested profile "hadoop-2.4" could not be activated >> because >> it does not exist. >> [ERROR] Failed to execute goal >> net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) >> on >> project spark-core_2.10: Execution scala-compile-first of goal >> net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed >> -> >> [Help 1] >> >> >> >> On Friday, February 13, 2015 11:16 AM, Sean Owen <so...@cloudera.com> >> wrote: >> >> >> If you just need standalone mode, you don't need "-Pyarn". There is no >> "-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set >> -Dhadoop.version=2.6.0. That should be it. >> >> If that still doesn't work, define "doesn't succeed". >> >> On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert >> <rgra...@yahoo.com.invalid> wrote: >>> Hi guys, >>> >>> Probably a dummy question. Do you know how to compile Spark 0.9 to easily >>> integrate with HDFS 2.6.0 ? >>> >>> I was trying >>> sbt/sbt -Pyarn -Phadoop-2.6 assembly >>> or >>> mvn -Dhadoop.version=2.6.0 -DskipTests clean package >>> >>> but none of these approaches succeeded. >>> >>> Thanks, >>> Robert >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
