build spark for cdh5
does anyone have the right maven invocation for cdh5 with yarn? i tried: $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn -DskipTests clean package $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn test it builds and passes tests just fine, but when i deploy on cluster and i try to run SparkPi i get: Caused by: java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportRequestProto overrides final method getUnknownFields.()Lcom/google/p\ rotobuf/UnknownFieldSet; so clearly i am doing something wrong. something with protobuf 2.4 versus 2.5 i do not want to use the cloudera version of spark for cdh 5 (it includes the wrong akka version for me) thanks
Re: build spark for cdh5
thanks! my bad On Wed, Feb 18, 2015 at 2:00 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi Koert, You should be using -Phadoop-2.3 instead of -Phadoop2.3. -Sandy On Wed, Feb 18, 2015 at 10:51 AM, Koert Kuipers ko...@tresata.com wrote: does anyone have the right maven invocation for cdh5 with yarn? i tried: $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn -DskipTests clean package $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn test it builds and passes tests just fine, but when i deploy on cluster and i try to run SparkPi i get: Caused by: java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportRequestProto overrides final method getUnknownFields.()Lcom/google/p\ rotobuf/UnknownFieldSet; so clearly i am doing something wrong. something with protobuf 2.4 versus 2.5 i do not want to use the cloudera version of spark for cdh 5 (it includes the wrong akka version for me) thanks
Re: build spark for cdh5
Hi Koert, You should be using -Phadoop-2.3 instead of -Phadoop2.3. -Sandy On Wed, Feb 18, 2015 at 10:51 AM, Koert Kuipers ko...@tresata.com wrote: does anyone have the right maven invocation for cdh5 with yarn? i tried: $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn -DskipTests clean package $ mvn -Phadoop2.3 -Dhadoop.version=2.5.0-cdh5.2.3 -Pyarn test it builds and passes tests just fine, but when i deploy on cluster and i try to run SparkPi i get: Caused by: java.lang.VerifyError: class org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportRequestProto overrides final method getUnknownFields.()Lcom/google/p\ rotobuf/UnknownFieldSet; so clearly i am doing something wrong. something with protobuf 2.4 versus 2.5 i do not want to use the cloudera version of spark for cdh 5 (it includes the wrong akka version for me) thanks
Re: Build Spark Against CDH5
After successfully building the official 0.9.0 release I attempted to build off of the github code again and was successfully able to do so. Not really sure what happened, but it works now. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Build-Spark-Against-CDH5-tp2129p2165.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Build Spark Against CDH5
Just as a second note, I am able to build the source in the official 0.9.0 release (http://d3kbcqa49mib13.cloudfront.net/spark-0.9.0-incubating-bin-hadoop2.tgz). -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Build-Spark-Against-CDH5-tp2129p2130.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
