When using manual Kafka offset commit in Spark streaming job and application fails to process current batch without committing offset in executor, is it expected behavior that next batch will be processed and offset will be moved to next batch regardless of application failure to commit? It seems so based on glance at the code. If so, is it expected that job termination upon failure to process batch and commit offset should resume from last committed offset?
I’m asking since until now I didn’t have to deal with Spark streaming from Kafka where assumption was “successfully processed at-least-once”. Stopping Kafka processing or streaming on any application failure may seem rather extreme but it is what it is. Thank you, Andre
