Kafka version dependency in Spark 1.2
Hi, Is there any plan to bump the Kafka version dependency in Spark 1.2 from 0.8.0 to 0.8.1.1? Current dependency is still on Kafka 0.8.0 https://github.com/apache/spark/blob/branch-1.2/external/kafka/pom.xml Thanks Bhaskie
Re: Kafka version dependency in Spark 1.2
Just curious, what are the pros and cons of this? Can the 0.8.1.1 client still talk to 0.8.0 versions of Kafka, or do you need it to match your Kafka version exactly? Matei On Nov 10, 2014, at 9:48 AM, Bhaskar Dutta bhas...@gmail.com wrote: Hi, Is there any plan to bump the Kafka version dependency in Spark 1.2 from 0.8.0 to 0.8.1.1? Current dependency is still on Kafka 0.8.0 https://github.com/apache/spark/blob/branch-1.2/external/kafka/pom.xml https://github.com/apache/spark/blob/branch-1.2/external/kafka/pom.xml Thanks Bhaskie
Re: Kafka version dependency in Spark 1.2
Can the 0.8.1.1 client still talk to 0.8.0 versions of Kafka Yes it can. 0.8.1 is fully compatible with 0.8. It is buried on this page: http://kafka.apache.org/documentation.html In addition to the pom version bump SPARK-2492 would bring the kafka streaming receiver (which was originally based on kafka 0.7) in line with kafka 0.8: https://issues.apache.org/jira/browse/SPARK-2492 https://github.com/apache/spark/pull/1420 I will soon test that PR on a spark+yarn cluster Cheers, Sean On Nov 10, 2014, at 11:58 AM, Matei Zaharia matei.zaha...@gmail.commailto:matei.zaha...@gmail.com wrote: Just curious, what are the pros and cons of this? Can the 0.8.1.1 client still talk to 0.8.0 versions of Kafka, or do you need it to match your Kafka version exactly? Matei On Nov 10, 2014, at 9:48 AM, Bhaskar Dutta bhas...@gmail.commailto:bhas...@gmail.com wrote: Hi, Is there any plan to bump the Kafka version dependency in Spark 1.2 from 0.8.0 to 0.8.1.1? Current dependency is still on Kafka 0.8.0 https://github.com/apache/spark/blob/branch-1.2/external/kafka/pom.xml Thanks Bhaskie
Re: Kafka version dependency in Spark 1.2
Version 0.8.2-beta is published. I'd consider waiting on this, it has quite a few nice changes coming. https://archive.apache.org/dist/kafka/0.8.2-beta/RELEASE_NOTES.html I started the 0.8.1.1 upgrade in a branch a few weeks ago but abandoned it because I wasn't sure if there was interest beyond myself. - Helena @helenaedelson -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-version-dependency-in-Spark-1-2-tp18503p18510.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
