Re: Kafka Consumer Pre Fetch Messages + Async commits

2017-08-29 Thread Julia Wistance
th repeat messages in > the event of failure in any case, so the only difference sync commit > would make would be (possibly) slower run time. > > On Sat, Aug 26, 2017 at 1:07 AM, Julia Wistance > <julia.wista...@gmail.com> wrote: > > Hi Experts, > > > > A quest

Kafka Consumer Pre Fetch Messages + Async commits

2017-08-26 Thread Julia Wistance
Hi Experts, A question on what could potentially happen with Spark Streaming 2.2.0 + Kafka. LocationStrategies says that "new Kafka consumer API will pre-fetch messages into buffers.". If we store offsets in Kafka, currently we can only use a async commits. So, 1 - Could it happen that we commit