Hi thanks for the answer, I'm using this command to compile : SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true ./sbt/sbt assembly
When I do that, it runs for about 3-5 minutes and then after a long "packaging" phase, it simply says it failed. 2013/12/23 Patrick Wendell <[email protected]> > Hey Kevin, > > Could you give us the exact command that you are using to compile? > It's possible the YARN API changed in CDH 4.5 and our heuristics don't > detect it correctly. > > On Mon, Dec 23, 2013 at 6:43 AM, Kevin Moulart <[email protected]> > wrote: > > I just tried to compile version 0.8.1 against CDH-4.5.0 and it failed > just > > the same. > > > > Le mardi 17 décembre 2013 20:56:24 UTC+1, Debasish Das a écrit : > >> > >> Thanks Matei. > >> > >> We will wait for the release candidate of Spark 0.8.1 and whether it > could > >> be run against latest CDH/HDP YARN. > >> > >> On Monday, December 16, 2013 10:02:41 PM UTC-8, Matei Zaharia wrote: > >>> > >>> Ah, this is because of a YARN API update in CDH 4.5.0 (as well as > Apache > >>> Hadoop 2.2). You’ll need to wait for Spark 0.8.1 to compile against > that. > >>> There is a release candidate posted on our Apache mailing list: > >>> http://spark.incubator.apache.org/mailing-lists.html. > >>> > >>> Matei > >>> > >>> On Dec 16, 2013, at 4:51 PM, Debasish Das <[email protected]> > wrote: > >>> > >>> Hi Patrick, > >>> > >>> With the following configs: > >>> > >>> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > >>> export SPARK_YARN=true > >>> > >>> Inside the project yarn, the errors are as follows: > >>> > >>> [warn] > >>> > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:59: > >>> Treating numbers with a leading zero as octal is deprecated. > >>> [warn] val STAGING_DIR_PERMISSION: FsPermission = > >>> FsPermission.createImmutable(0700:Short) > >>> [warn] > >>> ^ > >>> [warn] > >>> > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:62: > >>> Treating numbers with a leading zero as octal is deprecated. > >>> [warn] val APP_FILE_PERMISSION: FsPermission = > >>> FsPermission.createImmutable(0644:Short) > >>> [warn] > >>> ^ > >>> [error] > >>> > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:36: > >>> object AMResponse is not a member of package > >>> org.apache.hadoop.yarn.api.records > >>> [error] import org.apache.hadoop.yarn.api.records.{AMResponse, > >>> ApplicationAttemptId} > >>> [error] ^ > >>> [error] > >>> > /home/debasish/sag_spark/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:105: > >>> value getAMResponse is not a member of > >>> org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse > >>> [error] val amResp = > >>> allocateWorkerResources(workersToRequest).getAMResponse > >>> [error] ^ > >>> [warn] two warnings found > >>> [error] two errors found > >>> [error] (yarn/compile:compile) Compilation failed > >>> [error] Total time: 15 s, completed Dec 16, 2013 7:47:03 PM > >>> > >>> Note that I can run the code against cdh4.5.0 mr1 client but we need > the > >>> YARN jar for deployment. > >>> > >>> Thanks. > >>> Deb > >>> > >>> On Friday, December 13, 2013 11:03:32 AM UTC-8, Patrick Wendell wrote: > >>>> > >>>> What errors are you getting in this case? Is it the same errors as > >>>> before or something else? > >>>> > >>>> > >>>> On Thu, Dec 12, 2013 at 11:54 PM, Debasish Das <[email protected]> > >>>> wrote: > >>>>> > >>>>> Thanks TD. sbt clean helped. > >>>>> > >>>>> With these configs I could get the jar file and it runs fine on the > >>>>> standalone spark cluster: > >>>>> > >>>>> export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 > >>>>> export SPARK_YARN=false > >>>>> > >>>>> If I try to generate the deployment jar for YARN with the following > >>>>> configs, I am getting errors. > >>>>> > >>>>> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > >>>>> export SPARK_YARN=true > >>>>> > >>>>> Thanks. > >>>>> Deb > >>>>> > >>>>> > >>>>> On Thursday, December 12, 2013 10:39:05 PM UTC-8, TD wrote: > >>>>>> > >>>>>> Can you try doing a "sbt clean" before building? I have seen this > >>>>>> error once and a clean build helped. > >>>>>> > >>>>>> > >>>>>> On Thu, Dec 12, 2013 at 10:37 PM, Debasish Das < > [email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I could compile Spark with CDH 4.2.0 but when I tried to access > hdfs > >>>>>>> it failed. > >>>>>>> > >>>>>>> I looked for the old post on Spark user group and found that Spark > >>>>>>> should be compiled with the exact hadoop client version of the > cluster. > >>>>>>> > >>>>>>> Our cluster is at CDH 4.5.0. I put the following configs for the > >>>>>>> compilation on the master branch: > >>>>>>> > >>>>>>> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 > >>>>>>> export SPARK_YARN=true > >>>>>>> > >>>>>>> I also tried to see if I can build against the client only > >>>>>>> > >>>>>>> export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 > >>>>>>> export SPARK_YARN=false > >>>>>>> > >>>>>>> I am getting 43 compilation errors from spark-streaming project. > >>>>>>> > >>>>>>> I have attached few msgs. > >>>>>>> > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:51: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] dstream.filter((x => f(x).booleanValue())) > >>>>>>> [error] ^ > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:54: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] def cache(): JavaPairDStream[K, V] = dstream.cache() > >>>>>>> [error] ^ > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] def persist(): JavaPairDStream[K, V] = dstream.persist() > >>>>>>> [error] ^ > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] def persist(storageLevel: StorageLevel): > JavaPairDStream[K, > >>>>>>> V] = dstream.persist(storageLevel) > >>>>>>> [error] > >>>>>>> ^ > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:66: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] def repartition(numPartitions: Int): JavaPairDStream[K, > V] > >>>>>>> = dstream.repartition(numPartitions) > >>>>>>> [error] > >>>>>>> ^ > >>>>>>> [error] > >>>>>>> > /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:83: > >>>>>>> type mismatch; > >>>>>>> [error] found : org.apache.spark.streaming.DStream[(K, V)] > >>>>>>> [error] required: > >>>>>>> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] > >>>>>>> [error] Note: implicit method fromPairDStream is not applicable > here > >>>>>>> because it comes after the application point and it lacks an > explicit result > >>>>>>> type > >>>>>>> [error] dstream.window(windowDuration) > >>>>>>> [error] ^ > >>>>>>> > >>>>>>> Note that the project compiled fine with CDH 4.2.0 but I could not > >>>>>>> access our HDFS data. > >>>>>>> > >>>>>>> Thanks. > >>>>>>> Deb > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Spark Users" group. > >>>>>>> To unsubscribe from this group and stop receiving emails from it, > >>>>>>> send an email to [email protected]. > >>>>>>> > >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. > >>>>>> > >>>>>> > >>>>> > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Spark Users" group. > >>>>> To unsubscribe from this group and stop receiving emails from it, > send > >>>>> an email to [email protected]. > >>>>> For more options, visit https://groups.google.com/groups/opt_out. > >>>> > >>>> > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Spark Users" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to [email protected]. > >>> For more options, visit https://groups.google.com/groups/opt_out. > >>> > >>> > > -- > > You received this message because you are subscribed to the Google Groups > > "Spark Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Spark Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/spark-users/T1soH67C5M4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- Kévin Moulart GSM France : +33 7 81 06 10 10 GSM Belgique : +32 473 85 23 85 Téléphone fixe : +32 2 771 88 45 -- You received this message because you are subscribed to the Google Groups "Spark Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
