[
https://issues.apache.org/jira/browse/SPARK-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das resolved SPARK-2507.
----------------------------------
Resolution: Fixed
> Compile error of streaming project with 2.0.0-cdh4.6.0
> ------------------------------------------------------
>
> Key: SPARK-2507
> URL: https://issues.apache.org/jira/browse/SPARK-2507
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 0.9.0, 0.9.1, 1.0.0, 1.0.1, 1.0.2
> Environment: RedHat 5.3
> 2.0.0-cdh4.6.0
> enable yarn
> java version "1.6.0_45"
> Reporter: James Z.M. Gao
> Priority: Minor
>
> Hi,
> When compiling with
> {quote}
> ./make-distribution.sh --hadoop 2.0.0-cdh4.6.0 --with-yarn --tgz
> {quote}
> I have the following errors on streaming java api:
> {quote}
> Version is 0.9.0-incubating
> Making spark-0.9.0-incubating-hadoop_2.0.0-cdh4.6.0-bin.tar.gz
> Hadoop version set to 2.0.0-cdh4.6.0
> YARN enabled
> [info] Loading project definition from /root/spark-source/project/project
> [info] Loading project definition from /root/spark-source/project
> [info] Set current project to root (in build file:/root/spark-source/)
> [info] Compiling 1 Scala source to
> /root/spark-source/streaming/target/scala-2.10/classes...
> [error]
> /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:57:
> type mismatch;
> [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
> [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
> [error] Note: implicit method fromPairDStream is not applicable here because
> it comes after the application point and it lacks an explicit result type
> [error] dstream.filter((x => f(x).booleanValue()))
> [error] ^
> [error]
> /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:60:
> type mismatch;
> [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
> [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
> [error] Note: implicit method fromPairDStream is not applicable here because
> it comes after the application point and it lacks an explicit result type
> [error] def cache(): JavaPairDStream[K, V] = dstream.cache()
> [error] ^
> [error]
> /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:63:
> type mismatch;
> [error] found : org.apache.spark.streaming.dstream.DStream[(K, V)]
> [error] required: org.apache.spark.streaming.api.java.JavaPairDStream[K,V]
> [error] Note: implicit method fromPairDStream is not applicable here because
> it comes after the application point and it lacks an explicit result type
> [error] def persist(): JavaPairDStream[K, V] = dstream.persist()
> [error] ^
> ......
> [error]
> /root/spark-source/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:669:
> type mismatch;
> [error] found : org.apache.spark.streaming.dstream.DStream[(K,
> (com.google.common.base.Optional[V], W))]
> [error] required:
> org.apache.spark.streaming.api.java.JavaPairDStream[K,(com.google.common.base.Optional[V],
> W)]
> [error] Note: implicit method fromPairDStream is not applicable here because
> it comes after the application point and it lacks an explicit result type
> [error] joinResult.mapValues{case (v, w) =>
> (JavaUtils.optionToOptional(v), w)}
> [error] ^
> [error] 44 errors found
> [error] (streaming/compile:compile) Compilation failed
> {quote}
> Here is a simple PR fix this problem: https://github.com/apache/spark/pull/153
--
This message was sent by Atlassian JIRA
(v6.2#6252)
