Thanks Matei. We will wait for the release candidate of Spark 0.8.1 and whether it could be run against latest CDH/HDP YARN.
On Monday, December 16, 2013 10:02:41 PM UTC-8, Matei Zaharia wrote: > > Ah, this is because of a YARN API update in CDH 4.5.0 (as well as Apache > Hadoop 2.2). You’ll need to wait for Spark 0.8.1 to compile against that. > There is a release candidate posted on our Apache mailing list: > http://spark.incubator.apache.org/mailing-lists.html<http://www.google.com/url?q=http%3A%2F%2Fspark.incubator.apache.org%2Fmailing-lists.html&sa=D&sntz=1&usg=AFQjCNEHrWNSNaUVApa1nquVgxPLGinJSQ> > . > > Matei > > On Dec 16, 2013, at 4:51 PM, Debasish Das <[email protected]<javascript:>> > wrote: > > Hi Patrick, > > With the following configs: > > export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > export SPARK_YARN=true > > Inside the project yarn, the errors are as follows: > > [warn] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:59: > > Treating numbers with a leading zero as octal is deprecated. > [warn] val STAGING_DIR_PERMISSION: FsPermission = > FsPermission.createImmutable(0700:Short) > [warn] > ^ > [warn] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:62: > > Treating numbers with a leading zero as octal is deprecated. > [warn] val APP_FILE_PERMISSION: FsPermission = > FsPermission.createImmutable(0644:Short) > [warn] > ^ > [error] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:36: > > object AMResponse is not a member of package > org.apache.hadoop.yarn.api.records > [error] import org.apache.hadoop.yarn.api.records.{AMResponse, > ApplicationAttemptId} > [error] ^ > [error] > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:105: > > value getAMResponse is not a member of > org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse > [error] val amResp = > allocateWorkerResources(workersToRequest).getAMResponse > [error] ^ > [warn] two warnings found > [error] two errors found > [error] (yarn/compile:compile) Compilation failed > [error] Total time: 15 s, completed Dec 16, 2013 7:47:03 PM > > Note that I can run the code against cdh4.5.0 mr1 client but we need the > YARN jar for deployment. > > Thanks. > Deb > > On Friday, December 13, 2013 11:03:32 AM UTC-8, Patrick Wendell wrote: >> >> What errors are you getting in this case? Is it the same errors as before >> or something else? >> >> >> On Thu, Dec 12, 2013 at 11:54 PM, Debasish Das <[email protected]>wrote: >> >>> Thanks TD. sbt clean helped. >>> >>> With these configs I could get the jar file and it runs fine on the >>> standalone spark cluster: >>> >>> export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 >>> export SPARK_YARN=false >>> >>> If I try to generate the deployment jar for YARN with the following >>> configs, I am getting errors. >>> >>> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 >>> export SPARK_YARN=true >>> >>> Thanks. >>> Deb >>> >>> >>> On Thursday, December 12, 2013 10:39:05 PM UTC-8, TD wrote: >>> >>>> Can you try doing a "sbt clean" before building? I have seen this error >>>> once and a clean build helped. >>>> >>>> >>>> On Thu, Dec 12, 2013 at 10:37 PM, Debasish Das <[email protected]>wrote: >>>> >>>>> Hi, >>>>> >>>>> I could compile Spark with CDH 4.2.0 but when I tried to access hdfs >>>>> it failed. >>>>> >>>>> I looked for the old post on Spark user group and found that Spark >>>>> should be compiled with the exact hadoop client version of the cluster. >>>>> >>>>> Our cluster is at CDH 4.5.0. I put the following configs for the >>>>> compilation on the master branch: >>>>> >>>>> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 >>>>> export SPARK_YARN=true >>>>> >>>>> I also tried to see if I can build against the client only >>>>> >>>>> export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 >>>>> export SPARK_YARN=false >>>>> >>>>> I am getting 43 compilation errors from spark-streaming project. >>>>> >>>>> I have attached few msgs. >>>>> >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:51: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] dstream.filter((x => f(x).booleanValue())) >>>>> [error] ^ >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:54: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] def cache(): JavaPairDStream[K, V] = dstream.cache() >>>>> [error] ^ >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:57: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] def persist(): JavaPairDStream[K, V] = dstream.persist() >>>>> [error] ^ >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:60: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] def persist(storageLevel: StorageLevel): JavaPairDStream[K, >>>>> V] = dstream.persist(storageLevel) >>>>> [error] >>>>> ^ >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:66: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] def repartition(numPartitions: Int): JavaPairDStream[K, V] = >>>>> dstream.repartition(numPartitions) >>>>> [error] >>>>> ^ >>>>> [error] /home/debasish/sag_spark/streaming/src/main/scala/org/ >>>>> apache/spark/streaming/api/java/JavaPairDStream.scala:83: type >>>>> mismatch; >>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] >>>>> [error] required: org.apache.spark.streaming. >>>>> api.java.JavaPairDStream[K,V] >>>>> [error] Note: implicit method fromPairDStream is not applicable here >>>>> because it comes after the application point and it lacks an explicit >>>>> result type >>>>> [error] dstream.window(windowDuration) >>>>> [error] ^ >>>>> >>>>> Note that the project compiled fine with CDH 4.2.0 but I could not >>>>> access our HDFS data. >>>>> >>>>> Thanks. >>>>> Deb >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Spark Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Spark Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Spark Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
