Thanks Sean for your prompt response.
I was trying to compile as following:
mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package
but I got a bunch of errors(see below). Hadoop-2.6.0 compiled correctly, and
all hadoop jars are in .m2 repository.
Do you have any idea what might happens ?
Robert
[WARNING] Class com.google.protobuf.Parser not found - continuing with a stub.
[ERROR] error while loading RpcResponseHeaderProto, class file
'/home/rgrandl/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar(org/apache/hadoop/ipc/protobuf/RpcHeaderProtos$RpcResponseHeaderProto.class)'
is broken
(class java.lang.NullPointerException/null)
[WARNING] one warning found
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [2.537s]
[INFO] Spark Project Core ................................ FAILURE [25.917s]
[INFO] Spark Project Bagel ............................... SKIPPED
[INFO] Spark Project GraphX .............................. SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Streaming ........................... SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project Assembly ............................ SKIPPED
[INFO] Spark Project External Twitter .................... SKIPPED
[INFO] Spark Project External Kafka ...................... SKIPPED
[INFO] Spark Project External Flume ...................... SKIPPED
[INFO] Spark Project External ZeroMQ ..................... SKIPPED
[INFO] Spark Project External MQTT ....................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.002s
[INFO] Finished at: Fri Feb 13 11:21:36 PST 2015
[INFO] Final Memory: 49M/1226M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop-2.4" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.1.5:compile (scala-compile-first) on
project spark-core_2.10: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.1.5:compile failed. CompileFailed ->
[Help 1]
On Friday, February 13, 2015 11:16 AM, Sean Owen <[email protected]>
wrote:
If you just need standalone mode, you don't need "-Pyarn". There is no
"-Phadoop-2.6"; you should use "-Phadoop-2.4" for 2.4+. Yes, set
-Dhadoop.version=2.6.0. That should be it.
If that still doesn't work, define "doesn't succeed".
On Fri, Feb 13, 2015 at 7:13 PM, Grandl Robert
<[email protected]> wrote:
> Hi guys,
>
> Probably a dummy question. Do you know how to compile Spark 0.9 to easily
> integrate with HDFS 2.6.0 ?
>
> I was trying
> sbt/sbt -Pyarn -Phadoop-2.6 assembly
> or
> mvn -Dhadoop.version=2.6.0 -DskipTests clean package
>
> but none of these approaches succeeded.
>
> Thanks,
> Robert
