[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274910#comment-15274910 ]
Mark Grover commented on SPARK-12177: ------------------------------------- I spent some time earlier today looking at the latest Kafka 0.10 RC. Thanks Cody, looks like you poked at it too. As Cody found out, the new kafka consumer API is in flux. I found and filed a PR (KAFKA-3669) for an incompatibility in KafkaConfig, which is now fixed in Kafka 0.10. But, there's more - KAFKA-3633 is of note - which discusses fixing a compatibility break between 0.9 and 0.10 and remains unresolved. So, at this point, I can say, that there's no point in committing the PR associated with this JIRA (https://github.com/apache/spark/pull/11863), until at least Kafka 0.10.0 is released and we have a good sense that Kafka 0.11.0 is not going to break compatibility with 0.10.0. Otherwise, we have to bear the burden of adding and maintaining complexity to build against multiple versions of Kafka, something Storm folks are already suffering from, having their KafkaSpout now using the new Kafka consumer API from 0.9. Given the timing of the Kafka 0.10.0 release and Spark 2 release, this JIRA likely wouldn't get resolved in 2.0. > Update KafkaDStreams to new Kafka 0.9 Consumer API > -------------------------------------------------- > > Key: SPARK-12177 > URL: https://issues.apache.org/jira/browse/SPARK-12177 > Project: Spark > Issue Type: Improvement > Components: Streaming > Affects Versions: 1.6.0 > Reporter: Nikita Tarasenko > Labels: consumer, kafka > > Kafka 0.9 already released and it introduce new consumer API that not > compatible with old one. So, I added new consumer api. I made separate > classes in package org.apache.spark.streaming.kafka.v09 with changed API. I > didn't remove old classes for more backward compatibility. User will not need > to change his old spark applications when he uprgade to new Spark version. > Please rewiew my changes -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org