Cody, thanks for the response.
So Kafka direct stream actually has consumer on both the driver and executor? Can you please provide more details? Thank you very much! ________________________________ From: Cody Koeninger [mailto:c...@koeninger.org] Sent: 2016年10月12日 20:10 To: Haopu Wang Cc: user@spark.apache.org Subject: Re: Kafka integration: get existing Kafka messages? its set to none for the executors, because otherwise they wont do exactly what the driver told them to do. you should be able to set up the driver consumer to determine batches however you want, though. On Wednesday, October 12, 2016, Haopu Wang <hw...@qilinsoft.com> wrote: Hi, I want to read the existing Kafka messages and then subscribe new stream messages. But I find "auto.offset.reset" property is always set to "none" in KafkaUtils. Does that mean I cannot specify "earliest" property value when create direct stream? Thank you!
