1. No, prefetched message offsets aren't exposed. 2. No, I'm not aware of any plans for sync commit, and I'm not sure that makes sense. You have to be able to deal with repeat messages in the event of failure in any case, so the only difference sync commit would make would be (possibly) slower run time.
On Sat, Aug 26, 2017 at 1:07 AM, Julia Wistance <julia.wista...@gmail.com> wrote: > Hi Experts, > > A question on what could potentially happen with Spark Streaming 2.2.0 + > Kafka. LocationStrategies says that "new Kafka consumer API will pre-fetch > messages into buffers.". > If we store offsets in Kafka, currently we can only use a async commits. > > So, > 1 - Could it happen that we commit offsets that we havent processed yet but > the kafka consumers has prefetched > 2 - Are there plans to support a sync commit? Although we can go for an > alternate store of commits like HBase / Zookeeper, MySQL etc the code would > wait till the offsets are stored in either of these systems. It would make > sense that Spark / Kafka also adds a sync commit option? > > Appreciate the reply. > JW > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
