sorry meant to say: i compiled against cdh 4.3.0 and been using it with 4.5.0 without trouble. i believe all cdh 4.x versions are wire compatible
On Tue, Dec 17, 2013 at 3:26 PM, Koert Kuipers <[email protected]> wrote: > i compiled against cdh 4.3.0 and been using it with 4.5.0 without using. i > believe all cdh 4.x are wire compatible > > > On Fri, Dec 13, 2013 at 1:37 AM, Debasish Das <[email protected]>wrote: > >> Hi, >> >> I could compile Spark with CDH 4.2.0 but when I tried to access hdfs it >> failed. >> >> I looked for the old post on Spark user group and found that Spark should >> be compiled with the exact hadoop client version of the cluster. >> >> Our cluster is at CDH 4.5.0. I put the following configs for the >> compilation on the master branch: >> >> export SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 >> export SPARK_YARN=true >> >> I also tried to see if I can build against the client only >> >> export SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.5.0 >> export SPARK_YARN=false >> >> I am getting 43 compilation errors from spark-streaming project. >> >> I have attached few msgs. >> >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:51: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] dstream.filter((x => f(x).booleanValue())) >> [error] ^ >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:54: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] def cache(): JavaPairDStream[K, V] = dstream.cache() >> [error] ^ >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] def persist(): JavaPairDStream[K, V] = dstream.persist() >> [error] ^ >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] def persist(storageLevel: StorageLevel): JavaPairDStream[K, V] >> = dstream.persist(storageLevel) >> [error] >> ^ >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:66: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] def repartition(numPartitions: Int): JavaPairDStream[K, V] = >> dstream.repartition(numPartitions) >> [error] >> ^ >> [error] >> /home/debasish/sag_spark/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:83: >> type mismatch; >> [error] found : org.apache.spark.streaming.DStream[(K, V)] >> [error] required: >> org.apache.spark.streaming.api.java.JavaPairDStream[K,V] >> [error] Note: implicit method fromPairDStream is not applicable here >> because it comes after the application point and it lacks an explicit >> result type >> [error] dstream.window(windowDuration) >> [error] ^ >> >> Note that the project compiled fine with CDH 4.2.0 but I could not access >> our HDFS data. >> >> Thanks. >> Deb >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Spark Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "Spark Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
