Repository: spark Updated Branches: refs/heads/branch-1.1 aec217a58 -> 5b4bc84ed
[SPARK-1022][Streaming][HOTFIX] Fixed zookeeper dependency of Kafka https://github.com/apache/spark/pull/1751 caused maven builds to fail. ``` ~/Apache/spark(branch-1.1|â) ⤠mvn -U -DskipTests clean install . . . [error] Apache/spark/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala:36: object NIOServerCnxnFactory is not a member of package org.apache.zookeeper.server [error] import org.apache.zookeeper.server.NIOServerCnxnFactory [error] ^ [error] Apache/spark/external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala:199: not found: type NIOServerCnxnFactory [error] val factory = new NIOServerCnxnFactory() [error] ^ [error] two errors found [error] Compile failed at Aug 5, 2014 1:42:36 PM [0.503s] ``` The problem is how SBT and Maven resolves multiple versions of the same library, which in this case, is Zookeeper. Observing and comparing the dependency trees from Maven and SBT showed this. Spark depends on ZK 3.4.5 whereas Apache Kafka transitively depends on upon ZK 3.3.4. SBT decides to evict 3.3.4 and use the higher version 3.4.5. But Maven decides to stick to the closest (in the tree) dependent version of 3.3.4. And 3.3.4 does not have NIOServerCnxnFactory. The solution in this patch excludes zookeeper from the apache-kafka dependency in streaming-kafka module so that it just inherits zookeeper from Spark core. Author: Tathagata Das <tathagata.das1...@gmail.com> Closes #1797 from tdas/kafka-zk-fix and squashes the following commits: 94b3931 [Tathagata Das] Fixed zookeeper dependency of Kafka (cherry picked from commit ee7f30856bf3f7b9a4f1d3641b6bc2cc4e842b0e) Signed-off-by: Patrick Wendell <pwend...@gmail.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5b4bc84e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5b4bc84e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5b4bc84e Branch: refs/heads/branch-1.1 Commit: 5b4bc84ed48211ac3801399804725f95e0350b54 Parents: aec217a Author: Tathagata Das <tathagata.das1...@gmail.com> Authored: Tue Aug 5 23:41:34 2014 -0700 Committer: Patrick Wendell <pwend...@gmail.com> Committed: Tue Aug 5 23:42:20 2014 -0700 ---------------------------------------------------------------------- external/kafka/pom.xml | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/5b4bc84e/external/kafka/pom.xml ---------------------------------------------------------------------- diff --git a/external/kafka/pom.xml b/external/kafka/pom.xml index 2aee999..4e2275a 100644 --- a/external/kafka/pom.xml +++ b/external/kafka/pom.xml @@ -68,6 +68,10 @@ <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> </exclusion> + <exclusion> + <groupId>org.apache.zookeeper</groupId> + <artifactId>zookeeper</artifactId> + </exclusion> </exclusions> </dependency> <dependency> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org