Ah, this is because of a YARN API update in CDH 4.5.0 (as well as Apache Hadoop 2.2). You’ll need to wait for Spark 0.8.1 to compile against that. There is a release candidate posted on our Apache mailing list: http://spark.incubator.apache.org/mailing-lists.html.
Matei On Dec 16, 2013, at 4:51 PM, Debasish Das <[email protected]> wrote: > Hi Patrick, > > With the following configs: > > export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > export SPARK_YARN=true > > Inside the project yarn, the errors are as follows: > > [warn] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:59: > Treating numbers with a leading zero as octal is deprecated. > [warn] val STAGING_DIR_PERMISSION: FsPermission = > FsPermission.createImmutable(0700:Short) > [warn] > ^ > [warn] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:62: > Treating numbers with a leading zero as octal is deprecated. > [warn] val APP_FILE_PERMISSION: FsPermission = > FsPermission.createImmutable(0644:Short) > [warn] > ^ > [error] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:36: > object AMResponse is not a member of package > org.apache.hadoop.yarn.api.records > [error] import org.apache.hadoop.yarn.api.records.{AMResponse, > ApplicationAttemptId} > [error] ^ > [error] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:105: > value getAMResponse is not a member of > org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse > [error] val amResp = > allocateWorkerResources(workersToRequest).getAMResponse > [error] ^ > [warn] two warnings found > [error] two errors found > [error] (yarn/compile:compile) Compilation failed > [error] Total time: 15 s, completed Dec 16, 2013 7:47:03 PM > > Note that I can run the code against cdh4.5.0 mr1 client but we need the YARN > jar for deployment. > > Thanks. > Deb > > On Friday, December 13, 2013 11:03:32 AM UTC-8, Patrick Wendell wrote: > What errors are you getting in this case? Is it the same errors as before or > something else? > > > On Thu, Dec 12, 2013 at 11:54 PM, Debasish Das <[email protected]> wrote: > Thanks TD. sbt clean helped. > > With these configs I could get the jar file and it runs fine on the > standalone spark cluster: > > export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 > export SPARK_YARN=false > > If I try to generate the deployment jar for YARN with the following configs, > I am getting errors. > > export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > export SPARK_YARN=true > > Thanks. > Deb > > > On Thursday, December 12, 2013 10:39:05 PM UTC-8, TD wrote: > Can you try doing a "sbt clean" before building? I have seen this error once > and a clean build helped. > > > On Thu, Dec 12, 2013 at 10:37 PM, Debasish Das <[email protected]> wrote: > Hi, > > I could compile Spark with CDH 4.2.0 but when I tried to access hdfs it > failed. > > I looked for the old post on Spark user group and found that Spark should be > compiled with the exact hadoop client version of the cluster. > > Our cluster is at CDH 4.5.0. I put the following configs for the compilation > on the master branch: > > export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > export SPARK_YARN=true > > I also tried to see if I can build against the client only > > export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 > export SPARK_YARN=false > > I am getting 43 compilation errors from spark-streaming project. > > I have attached few msgs. > > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:51: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] dstream.filter((x => f(x).booleanValue())) > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:54: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] def cache(): JavaPairDStream[K, V] = dstream.cache() > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] def persist(): JavaPairDStream[K, V] = dstream.persist() > [error] ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] def persist(storageLevel: StorageLevel): JavaPairDStream[K, V] = > dstream.persist(storageLevel) > [error] > ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:66: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] def repartition(numPartitions: Int): JavaPairDStream[K, V] = > dstream.repartition(numPartitions) > [error] > ^ > [error] > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:83: > type mismatch; > [error] found : org.apache.spark.streaming.DStream[(K, V)] > [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > [error] Note: implicit method fromPairDStream is not applicable here because > it comes after the application point and it lacks an explicit result type > [error] dstream.window(windowDuration) > [error] ^ > > Note that the project compiled fine with CDH 4.2.0 but I could not access our > HDFS data. > > Thanks. > Deb > > > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Spark Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
