Pranav Nakhe created SPARK-18779:
------------------------------------
Summary: Upgrade Spark Streaming integration for Kafka 0.10 to use
Kafka client library at 0.10.1
Key: SPARK-18779
URL: https://issues.apache.org/jira/browse/SPARK-18779
Project: Spark
Issue Type: Improvement
Components: DStreams
Affects Versions: 2.0.2
Reporter: Pranav Nakhe
The Spark Streaming integration for Kafka 0.10 currently pulls the kafka client
10.0.1 as a maven dependency. There is a newer kafka client available 10.1.0.
Would it be possible to have Spark Kafka integration use kafka client 10.1.0.
The reason we need the later Kafka client is because the later one (10.1.0) has
introduced a new API 'offsetsForTimes' for the KafkaConsumer which look up the
offsets for the given partitions by timestamp. We are using that to find the
offset that needs to be fed to the createDirectStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
