James Z.M. Gao created SPARK-2507: ------------------------------------- Summary: Compile error of streaming project with 2.0.0-cdh4.6.0 Key: SPARK-2507 URL: https://issues.apache.org/jira/browse/SPARK-2507 Project: Spark Issue Type: Bug Components: Streaming Affects Versions: 1.0.0, 0.9.1, 0.9.0 Environment: RedHat 5.3 2.0.0-cdh4.6.0 enable yarn java version "1.6.0_45" Reporter: James Z.M. Gao Priority: Minor
Hi, When compiling with {quote} ./make-distribution.sh --hadoop 2.0.0-cdh4.6.0 --with-yarn --tgz {quote} I have the following errors on streaming java api: {quote} Version is 0.9.0-incubating Making spark-0.9.0-incubating-hadoop_2.0.0-cdh4.6.0-bin.tar.gz Hadoop version set to 2.0.0-cdh4.6.0 YARN enabled [info] Loading project definition from /root/spark-source/project/project [info] Loading project definition from /root/spark-source/project [info] Set current project to root (in build file:/root/spark-source/) [info] Compiling 1 Scala source to /root/spark-source/streaming/target/scala-2.10/classes... [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57: type mismatch; [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)] [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type [error] dstream.filter((x => f(x).booleanValue())) [error] ^ [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60: type mismatch; [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)] [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type [error] def cache(): JavaPairDStream[K, V] = dstream.cache() [error] ^ [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:63: type mismatch; [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)] [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V] [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type [error] def persist(): JavaPairDStream[K, V] = dstream.persist() [error] ^ ...... [error] /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:669: type mismatch; [error] found : org.apache.spark.streaming.dstream.DStream[(K, (com.google.common.base.Optional[V], W))] [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V], W)] [error] Note: implicit method fromPairDStream is not applicable here because it comes after the application point and it lacks an explicit result type [error] joinResult.mapValues{case (v, w) => (JavaUtils.optionToOptional(v), w)} [error] ^ [error] 44 errors found [error] (streaming/compile:compile) Compilation failed {quote} Here is a simple PR fix this problem: https://github.com/apache/spark/pull/153 -- This message was sent by Atlassian JIRA (v6.2#6252)