Hi Experts, A question on what could potentially happen with Spark Streaming 2.2.0 + Kafka. LocationStrategies says that "new Kafka consumer API will pre-fetch messages into buffers.". If we store offsets in Kafka, currently we can only use a async commits.
So, 1 - Could it happen that we commit offsets that we havent processed yet but the kafka consumers has prefetched 2 - Are there plans to support a sync commit? Although we can go for an alternate store of commits like HBase / Zookeeper, MySQL etc the code would wait till the offsets are stored in either of these systems. It would make sense that Spark / Kafka also adds a sync commit option? Appreciate the reply. JW
