[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089750#comment-15089750
]
Mark Grover commented on SPARK-12177:
-------------------------------------
Thanks for working on this, Nikita. I'd like to help out. Here are a few things
of feedback that I have:
1. I tried rebasing what you have in your current branch to upstream master
(yours still seems to be based off of pre-1.6.0 code) but mostly because of
some commits related to import ordering that happened on spark trunk relatively
recently, I found it easier to migrate/copy the code for kafka-v09 and make the
minor changes to examples and root pom instead of doing a 'git rebase'.
2. I also noticed that the v09DirectKafkaWordCount example is pulling at least
the ConsumerConfig class from Kafka 0.8.2.1. This because the examples pom
contains both kafka 0.8.2.1 and 0.9.0 dependencies and somewhat arbitrarily
puts the 0.8.2.1 ahead. Since the ConsumerConfig class is available in both
under the same namespace, we end up pulling 0.8.2.1. We should fix that.
In general, I may have a few more changes/fixes that I'd like to contribute to
your pull request. Would it be possible for us to collaborate? What's the best
way to do so? Reopening the pull request and me adding to it? Or, just me
issuing pull requests to [your
branch|https://github.com/nikit-os/spark/tree/kafka-09-consumer-api]? Thanks!
> Update KafkaDStreams to new Kafka 0.9 Consumer API
> --------------------------------------------------
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.6.0
> Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
